There used to be quite a bit of interest in Prolog circles in automatic cycle detection. Of course the comparative absence of side effects in Prolog makes it a *lot* easier. The major surviving Prolog systems don't bother, in part because keeping accurate track of stack depth conflicts with tail recursion optimisation. The Clean and Haskell compilers I have access to basically let you set a limit on stack size (if you want to) and when the hardware/OS tell you it has run out, kill your program.
Interlisp-D on the Xerox Dandelions and such (1108, 1109, 1185, 1186) had a hard limit of 64k cells in the stack. It wasn't that hard to run out, but somehow most Interlisp programmers never noticed that there was a limit.
Preferences are entirely the wrong way to go because they have to be set by the wrong person, who lacks the knowledge about how the limit might be set. Something like aBlock valueWithStackLimit: aNumber would let the programmer override the limit at the point where s/he has some idea what limit will be needed.
Even that's a bit dodgy. The main thing is that it's ok to blast away the thread that gets too big, but not to crash the entire Smalltalk system.