A lot of documentation on capability-based security, which these systems often base on, can be found at www.erights.org, which describes the E language. IMHO, it's a must read if you're interested in these topics.
I've done some work on building sandboxes *inside* a Sqeak image, based on capabilities. I got the basic system working, but never got a practical system going. Specifically, I had a very good sandbox, but didn't write wrappers for very many useful facilities. I did get a bouncing atoms morph working once, albeit very slowly.
Unfortunately, Sqeak has shifted and my stuff doesn't file in any longer. I keep meaning to get it working again, but it's been over a year now, and there never seems to be enough time. Since there is suddenly so much interest on this topic on the list, I'll post the writeup and the changesets just so people can see them, even though it's not immediately useful right now.
http://minnow.cc.gatech.edu/squeak/2074
The writeup, at least, may give people some ideas. It seems to me that, once you latch on to the basic idea that an object is a privilage, all other problems can be solved directly, without much imagination. Perhaps others will agree.
-Lex