Dear VM devs,
I'm trying to allocate a Bitmap of around 16MB size in a plugin via `interpreterProxy instantiateClass: interpreterProxy classBitmap indexableSize: size`. However, I always get NULL back. I already tried wrapping the primitive call in `retryWithGC:until:`, with the same effect. Running a normal `Bitmap new: 16000000` from a workspace works flawlessly.
I'm on a Ubuntu 16.04 64bit, running a squeak 32bit built from trunk.
Is there anything to watch out for? More precisely, I'm trying to copy the contents of an image buffer generated from a pdf via libpoppler at 144dpi to a new bitmap in squeak. The primitive and how I invoke it can be seen here: https://pastebin.com/9NCdnx2J (line 50 is the call in question)
Thanks in advance for any pointers! Tom
On Fri, Jun 23, 2017 at 10:35 AM, Tom Beckmann tomjonabc@gmail.com wrote:
Dear VM devs,
I'm trying to allocate a Bitmap of around 16MB size in a plugin via `interpreterProxy instantiateClass: interpreterProxy classBitmap indexableSize: size`. However, I always get NULL back. I already tried wrapping the primitive call in `retryWithGC:until:`, with the same effect. Running a normal `Bitmap new: 16000000` from a workspace works flawlessly.
I'm on a Ubuntu 16.04 64bit, running a squeak 32bit built from trunk.
Is there anything to watch out for?
Yes. Object allocation is better handled in the image, because you don't want to deal with garbage collection strategies in a primitive. The typical way is to pass a pre-allocated array into the primitive which fills it.
- Bert -
Hi Tom,
On Fri, Jun 23, 2017 at 1:35 AM, Tom Beckmann tomjonabc@gmail.com wrote:
Dear VM devs,
I'm trying to allocate a Bitmap of around 16MB size in a plugin via `interpreterProxy instantiateClass: interpreterProxy classBitmap indexableSize: size`. However, I always get NULL back. I already tried wrapping the primitive call in `retryWithGC:until:`, with the same effect. Running a normal `Bitmap new: 16000000` from a workspace works flawlessly.
I'm on a Ubuntu 16.04 64bit, running a squeak 32bit built from trunk.
Is there anything to watch out for? More precisely, I'm trying to copy the contents of an image buffer generated from a pdf via libpoppler at 144dpi to a new bitmap in squeak. The primitive and how I invoke it can be seen here: https://pastebin.com/9NCdnx2J (line 50 is the call in question)
Thanks in advance for any pointers!
I want to endorse what Bert said; allocating in the image is a much better approach. But I did want to explain what you're seeing, assuming you're using Spur (Squeak 5.x, Pharo 6).
The Spur old space heap is segmented. It grows in increments defined by vm parameter 25 (Smalltalk vmParameterAt:[put:]), and shrinks whenever free memory is above parameter 24 and GC empties one or more segments. It will not grow memory beyond parameter 67, if that parameter is non-zero:
24 memory threshold above which to shrink object memory (read-write) 25 memory headroom when growing object memory (read-write) 67 the maximum allowed size of old space in bytes, 0 implies no internal limit (Spur only).
Parmeter 25 defaults to 16Mb (grow in increments of 16Mb) and 24 to 32 Mb (only shrink while 32Mb or more of empty segments exist).
Currently memory will /only/ grow in these circumstances - under image control via growMemoryByAtLeast: - when scavenging tenures sufficient objects to old space that old space must grow - when a fullGC fails to ensure there is at least vm parameter 25's worth of free space - when an allObjects or allInstances primitive needs more room to answer its result
The current allocator does /not/ try and grow memory in the allocation routines (which are accessed in plugins via instantiateClass:indexableSize: and in the image via new: basicNew: et al), failing instead. This gives the image the chance to mediate GC and heap growth (see e.g. Behavior>>handleFailing[Failing]BasicNew:).
Perhaps instantiateClass:indexableSize: /should/ allow growth but I prefer it remaining the same. You would then rewrite your primitive to - when it fails to allocate memory via instantiateClass:indexableSize: fail with interpreterProxy primitiveFailFor: PrimErrNoMemory - have the Smalltalk primitive failure code check the primitive error code (see senders of primitive:module:error:) and if the failure code is #'insufficient object memory' grow memory by the necessary amount and retry the primitive (a la handleFailingBasicNew:)
To the general audience, I think information like the above is key to being able to understand and exploit the system effectively, but where should it reside? Clearly "in my head" is not satisfactory. It belongs somewhere in the image, but it needs to be somewhere where people can find it and/or will look. Suggestions for an "architectural information" documentation section gratefully received. Object class>>whatIsAPrimitive might perhaps work or perhaps be overloaded.
_,,,^..^,,,_ best, Eliot
On 23-06-2017, at 9:45 AM, Eliot Miranda eliot.miranda@gmail.com wrote:
To the general audience, I think information like the above is key to being able to understand and exploit the system effectively, but where should it reside? Clearly "in my head" is not satisfactory. It belongs somewhere in the image, but it needs to be somewhere where people can find it and/or will look. Suggestions for an "architectural information" documentation section gratefully received. Object class>>whatIsAPrimitive might perhaps work or perhaps be overloaded.
A brief explanation in the sources as a comment for the class and/or some relevant methods *plus* a link to a swiki page with the full details. The briefest explanation might well be as little as “this is a quite complex issue; see swiki.squeak.org/85643505485"
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful random insult:- A .22 caliber intellect in a .357 Magnum world.
On 6/23/2017 2:01 PM, tim Rowledge wrote:
On 23-06-2017, at 9:45 AM, Eliot Mirandaeliot.miranda@gmail.com wrote:
To the general audience, I think information like the above is key to being able to understand and exploit the system effectively, but where should it reside? Clearly "in my head" is not satisfactory. It belongs somewhere in the image, but it needs to be somewhere where people can find it and/or will look. Suggestions for an "architectural information" documentation section gratefully received. Object class>>whatIsAPrimitive might perhaps work or perhaps be overloaded.
A brief explanation in the sources as a comment for the class and/or some relevant methods *plus* a link to a swiki page with the full details. The briefest explanation might well be as little as “this is a quite complex issue; see swiki.squeak.org/85643505485"
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful random insult:- A .22 caliber intellect in a .357 Magnum world.
But this information is for Smalltalk developers, not just for VM developers. And urls might eventually break.
I think the full information on how to better use the VM belong in some file in VM sources, maybe one or several .md files. And Smalltalk systems using the VMs might chose to duplicate it in the image. I'd surely do it for Cuis.
This might be too much for Object class>>whatIsAPrimitive. I think some methods in a 'documentation' category in SystemDictionary would be better. Or one (or several) new classes, just for this. Better if we can share them in all the distributions: Squeak, Pharo, Newspeak and Cuis.
Thanks,
On Fri, Jun 23, 2017 at 09:45:45AM -0700, Eliot Miranda wrote:
Hi Tom,
On Fri, Jun 23, 2017 at 1:35 AM, Tom Beckmann tomjonabc@gmail.com wrote:
Dear VM devs,
I'm trying to allocate a Bitmap of around 16MB size in a plugin via `interpreterProxy instantiateClass: interpreterProxy classBitmap indexableSize: size`. However, I always get NULL back. I already tried wrapping the primitive call in `retryWithGC:until:`, with the same effect. Running a normal `Bitmap new: 16000000` from a workspace works flawlessly.
I'm on a Ubuntu 16.04 64bit, running a squeak 32bit built from trunk.
Is there anything to watch out for? More precisely, I'm trying to copy the contents of an image buffer generated from a pdf via libpoppler at 144dpi to a new bitmap in squeak. The primitive and how I invoke it can be seen here: https://pastebin.com/9NCdnx2J (line 50 is the call in question)
Thanks in advance for any pointers!
I want to endorse what Bert said; allocating in the image is a much better approach. But I did want to explain what you're seeing, assuming you're using Spur (Squeak 5.x, Pharo 6).
I also want to endorse Bert's advice here. But by way of practical advice if you did really do need to debug your primitive:
Compile the VM with debugging enabled and compiler optimization disabled. Run the VM under a debugger (gdb or whatever). Put a breakpoint at the primitive function, and step through it to see where things are going wrong.
If you are dealing with a primitive that is called many times from the image, then just make a second copy of the primitive and give it a different name (so if you are working on #primitiveFoo, then make another #primitiveFoo2). You can then call that second copy from the image, and debug it when you hit the breakpoint in gdb.
You mentioned that you were working with "trunk" VM sources. If that refers to the interpreter VM, then in the makefile example at http://squeakvm.org/cgi-bin/viewvc.cgi/squeak/trunk/platforms/unix/cmake/Mak... you can use this to compile with debugging and no optimization:
CFLAGS_PARAM="--CFLAGS='-O0 -g'"
In Spur/Cog the equivalent is in the mvm scripts. Either way, the idea is to compile your plugin with debugging symbols enabled, and with all of the compiler optimizations turned off. Once you have that, you can poke around in the dgb debugger and see what is going wrong. Nine times out of ten it will turn out to be something you never would have guessed :-)
HTH, Dave
vm-dev@lists.squeakfoundation.org