Stephen Pair wrote:

...
In fact, the profiling results that you have for the 30 second test are
probably not valid at all for the 5 hour case.  They show that the bulk
of your time is being spent in low level (and probably optimized) code.
...

=========

Well, like I said, in stepping through your exmaple in the debugger, the commit
process took only seconds to get to the very last part, which is where the
remaining 4.99 hours were spent.  This is what exposed to me the very
inefficient Squeak operation that I didn't know about; namely removing
LargeInteger keys from a Dictionary.  Here is a piece of generic code that
demonstrates the issue, and is very similar to the Magma code I had profiled
earlier:

| d x testSize |
testSize _ 2001.  "you committed 2001 objects"
d _ Dictionary new.
x _  281474976711656.  "base oid"
1 to: testSize do: [ :e | d at: x+e put: (x+e) printString ].
[ (1 to: testSize) asArray shuffled do:
	[ :e |
	d removeKey: x + e ] ] timeToRun

I didn't wait for this one to finish, but I expect you'll see something very
close to five hours.

Now, change the initial value of x to 0 instead of the LargeInteger above
(281474976711656).  On my machine, it runs in 138 milliseconds.  All I can say
about that is, "Wow!"  Andreas Raab also demonstrated this huge improvement in
his demonstration which used SmallIntegers instead of large ones.

So I've decided to have this be my "fix" for now.  

When I said I don't expect any performance issues in Magma, what I meant was,
when writing the code, I was constantly paying attention to make sure only did
operations I knew to be efficient in Smalltalk.  I didn't expect removing
LargeInteger keys to be so expensive or I wouldn't have used it.

Thankfully, I believe my solution is simple.  I'm going to change my oid map so
that new and user objects are below 32-bits.  It'll limit a Magma repository to
32 million objects (excluding integers, floats and characters) but hey, I don't
think Squeak can currently write past just a few gigs anyway.

Thank you for exposing this problem to me, I'll let you know when I've posted
the fix (probably later this week)..


__________________________________________________
Do You Yahoo!?
Yahoo! Health - Feel better, live better
http://health.yahoo.com