investigating a bit further.. actually #markAndTrace: is the only sender of self lastPointerOf: oop recordWeakRoot: true.
the another sender of it is #startObj, inlined into markAndTrace: itself, but with false argument. So, it means that only "roots" are affected by this bug, an entry points where VM using this method to start tracing objects graph. The bug uncovers itself only in cases like with Nautilus, which having too deep (but could be finite) stack, with references to weak objects on it, because #markAndTraceStackPage: using #markAndTrace: for all oops on stack, which contributes to weak roots table.
I suspecting that markAndTraceStackPage: don't needs to record roots, because stack page , despite it is not a regular object oh heap actually acts as a root, and won't be GCed, because it is already reachable: (self assert: (stackPages isFree: thePage) not.)
which means that recording every weak object found on stack as weak root probably an overkill (you don't do it for regular ones ,isnt?). this is easy to modify, by adding extra keyword to #markAndTrace: passing whether it should record the incoming oop as weak root or not, instead of always assume "true" Because indeed, the weak roots table size is calculated by WeakRootTableSize := RootTableSize + RemapBufferSize + 100.
but it doesn't takes into account an arbitrary depth of stack :)
Here the code to crash VM (attached):
It crashes my VM stably at depth ~ 1450
CrashMe new crashMe: 1450
(as i said before , you need only a half of weak roots table to get things overflowing... WeakRootTableSize 2625