Stephen writes:

>A while back (in 2.2), I wrote some primitives that allowed me to allocate
and free external blocks of memory (using standard malloc and free).  I was
doing that in an attempt to create a generic mechanism for calling out to
external functions (using any of several well known calling
conventions)...which I was doing to implement a framework for abstracting OS
platform specifics.  I too needed to guarantee the location of the memory
wouldn't change underneath me.  I then created a nice Smalltalk class to
wrapper the blocks of external memory.  It was very elegant to have this
ability to manage external memory using Smalltalk objects.  The only thing
lacking was a finalization capability in 2.2 (so memory was never freed, but
I was waiting for others to add the finalization).

That would, indeed, be a useful thing.

> Getting the memory from the operating system is likely a lot safer and
easier that trying to "steal" it from Squeak heaps.  My guess is fiddling
with Squeak heaps would tricky and dangerous.

Right -- I don't suggest messing with the heaps.  Rather, if you are writing 
some code that needs memory for, say a stringbuffer or the like, you can 
accomplish this result by asking interpreterProxy to instantiate a String or a 
ByteArray for you, and access it from C using the C array access facilities.  
Alternatively, the ByteArray can be created by the calling routine and passed 
either as a parameter or an instance variable.  The downside of this is that 
the block of memory, though it will survive garbage collections since it is 
pointed to by the Smalltalk references, will be moved in memory from time to 
time, so every C procedure must access the block of memory through the 
parameter or instance variable, and not by hard addresses, and hard addresses 
should not be stored in the array itself.

This works just great in conjunction with the pluggable primitives where the 
smalltalk/C language interface is effectively seamless (I'm writing a short 
article on the subject now, and have put a brief piece on pluggables up on the 
Swiki), particularly for things like returning and manipulating strings and 
the like, but it won't suit all purposes.

> I stopped work on this stuff once I heard that pluggable primitives were
being added (sounds like I had a similar experience to Chris Reuter).  Has
anyone tried the pluggable primitives?  How well does it work?  Is good
support for marshaling objects to and from external data types?

PP's work GREAT!  Contrary to the suggestion earlier, I find them 
straightforward to work with and fun to code.  The problem, of course, is that 
they are almost undocumented as a feature -- I intend to help remedy that 
problem.  In answer to your question, its relatively straightforward to get 
access to objects in a pluggable, to the extent permitted by the 
interpreterProxy facility.

In short, you can get and manipulate an object reference; you can get and 
manipulate instance variables from the object; and you can get and manipulate 
the array elements at a byte, word or other level.  There is support for 
conversion between some basic types, and that's about it -- but its plenty for 
you to build whatever else you might need.

Additionally, you can, of course, manipulate the calling stack (to handle 
parameters and return a value), and obtain access to some basic Smalltalk 
services -- such as to instantiate objects to which you have a pointer (some 
basic objects already are provided).

Facilities are provided to make the access to arrays, for example, ABSOLUTELY 
SEAMLESS (including the fact that Smalltalk arrays start at 1, and C arrays 
start at 0), so that you can code almost all of what you need in Smalltalk and 
have it translate correctly to C.  In short, you take your object "handle", 
ask the interpreterProxy to give you the pointer to the object in array form 
(its a void *) to C, and a CArrayAccess (or something like that), coerce the 
sucker to whatever you need, and reach the array either in C or in Smalltalk 
without another thought.  If you need to send that data structure back to 
smalltalk, use the object "handle" and you're done!

However, all of this presupposes that you are writing the primitive while 
aware you are dealing with a smalltalk interpreter.  Consider the application 
with which I am presently playing around -- interfacing GNU's rx (regular 
expression) library to squeak:

No problem with respect to the wrapper or the interface: it was 
straightforward to build an rxPlugin and an interface to the main Posix 
functions, and the code I have even compiles and searches literal regular 
expressions correctly and with great speed.  The problem is that when the 
library calls malloc (as most third party libraries are wont to do), things go 
crunch in the night.  We need to be able to provide such a plugin with memory 
of the "traditional C heap" variety, and not access to floating blocks as 
described above.

Please tell me more about your primitive.  Does using malloc and free from 
within Smalltalk interfere with the Smalltalk memory management -- if not, its 
a straightforward solution and (I might suggest) a basic and essential one 
that the interpreterProxy arguably ought to provide to the PrimitivePlugin 
world to avoid perpetual reinventing of the wheel.

PP's are PERFECT for speeding up hunks of Smalltalk code; and they are great 
for providing interfaces to native system resources.  It would be nice to be 
able to build plugIns to generally portable open source libraries such as rx.  
However, to do so, we need to provide some kind of memory management facility 
to those libraries.

What do you think?