There are many books and papers about VMs.
For example, the book about BCPL by Richards and Whitby-Strevens
describes the OCODE vm for BCPL.  Ultimately it all goes back to
Dijkstra's Algol 60 compiler, one of the first Algol 60 compilers.
(I defy anyone to read Dijkstra's paper about that compiler, turn
to the Java VM book, and not say "do these guys owe Dijkstra big
time or what?".  Aren't we lucky that people weren't into software
IP back then?)  That Algol compiler inspired the B5500 and B6700
hardware.  As Burroughs discovered with their machines, you can hold
the top of stack in a register cache (a comparatively large one in
the B7700) and get much of the benefit of registers while still having
the compact code of a stack machine.  B6700 code was notably more
compact than PDP-11 (register machine) code, despite the B6700 being
a 48-bit machine.

Smalltalk was implemented at Xerox on Xerox microprogrammed machines
(Alto, Dorado, Dolphin &c).  The Alto was originally programmed in BCPL.
I would be surprised if the instruction set was enormously different
from OCODE.  The Xerox "stable" included MESA (there was a PhD thesis
on optimising the MESA instruction set), Smalltalk, Interlisp (there's
at least one paper about the Interlisp architecture), and I don't know
what else.  The Xerox machine I used had 32MB of virtual memory with
512 byte pages; the more compact your code is, the less work it is to
bring it into memory and the less often you have to do it anyway.  All
the Xerox systems I know about were stack-based VMs.  The Lisp Machines
(LMI) and Symbolics implementations of Lisp were also stack-based.

I don't remember what the P4 VM was, but the UCSD P-System VM was a
stack-based one.  (I've still got a manual somewhere that lists all the
order codes.)

I can't remember whether it was HP or IBM, but I read a paper recently
about a Java VM implementation by one of them which used a top-of-stack
register cache.  It's conceptually quite simple, and since most expressions
are simple, it can eliminate a lot of memory traffic.  Back to the 60s...