The larger the VM binary is, the slower it will be on today's CPUs.
how come?
It's because of the cache hierarchy. You don't have control over what will be where in a binary, or what sizes and levels of cache a CPU has. The smaller the binary is, the higher the chance that the part of the VM which needs to be used right now is in the CPU cache. I wrote about this a few years ago. I found that building all non-essential plugins as external gives ~4-5% better performance (using the Interpreter).
sounds interesting, you have a link on that?