So the idea is the FFI callouts to execute in a separate child-process.
That child-process has no access to the VM's memory so a memory violation in the C-callout could not crash the VM.
Obviously there will be some performance penalty, but the question is to what degree. There are two reasons to use an external library via FFI.
1. Speed
2. Functionality
Where its more about functionality than speed (e.g. git, libusb, libsodium, pdfium) application developers newly programming against an unfamiliar C library may be willing to trade speed for safety. Perhaps its used part-time like the Assert-VM during development, and production uses the standard higher performance FFI.
The idea of executing FFI callouts in a child-process arose while reading about Linux's clone() function that the parent process can allocate memory for the stack of the child process.
The child-process might be a simple event loop waiting on a semaphore.
My understanding of the FFI callout mechanism is that stack frame is constructed in the form expected by the function being invoked. With SafeFFI, when "fficallout" semaphore is being waited on, the child stack is static, so maybe the VM-parent-process could arrange the stack in the child-process such that sem_wait() returns not to line 005 but instead executes the required FFI-callout function. The "fficallout" semaphore is signalled from the Image once the stack frame has been constructed.
001 main()
002 { expose_child_function_addresses_to_parent_process();
003 while(true)
004 { sem_wait(&fficallout);
// Smalltalk image reconstructs stack frame here
005 printf("Dummy statement. Never gets here");
006 }
007 }
008
009 demo_redirect()
010 { printf("SafeFFI demo success");
011 }
So how feasible would something like that be?