Is there a way to permanently unload a Squeak plugin without recompiling 
the VM?

I've tried

Smalltalk unloadModule: 'APluginName'.

but as soon as I call a method that uses the plugin, it loads back up. 
I'm trying to get a feel for speed without the usual plugins.