memory

[Chris Muller]

> 2. Traversing a Reader 'hogs' memory on the server
> --------------------------------------------------
> This is our biggest problem. If I have a script which iterates over a
> reader, doing something with the tree reachable each object and then
> stubbing it out at the bottom of the loop, the server image's memory
> footprint just keeps growing. The client image's memory is stable.
> 
> e.g.
> 
> 	| reader |
> 	reader := (session root things read: #name).
> 	1 to: reader size do: [ :i | | thing |
> 		thing := reader at: i.
> 		"do other things with thing here..."
> 		"materialize a lot of objects but no transactions"
> 		session stubOut: thing.
> 		session finalizeOids.
> 		Smalltalk garbageCollect.
> 	].

I did find a memory "leak" that occurs for commits, I'll explain.  We must now
re-serialize commit-packages to support disaster-recovery, I was not
"finalizingOids" of the serializer that takes care of this work.  The attached
change-set corrects this.  (Note, in this case, you'll see I was actually able
to use #resetOidManager, which is quicker for clean-up than #finalizeOids, but
you not usable for a MagmaSessions serializer..).

I was not able to reproduce the case above only enumerating a reader to cause
the server to leak memory..  Let me share my test scripts and, if it is still
happening for you, perhaps you can modify the scripts such that they recreate
the problem and then send them back to me..

** Script 1 - Create a huge MagmaSet of UUID's, indexed on #asString36.

| path s set |
path _ 'c:\temp\bigger'.
MagmaRepositoryController create: path root: 
	(MagmaSet equivalenceIndex:
		((MaSearchStringIndexDefinition attribute: #asString36)
			keySize: 192 ;
			yourself)).
s _ MagmaSession openLocal: path.
s connectAs: 'cmm'.
s begin.
set _ s root.
1 to: 1000000 do:
	[ : n |
	set add: UUID new.
	n \\ 100 = 0 
		ifTrue: 
			[ s 
				commitAndBegin ;
				finalizeOids ] ].
s commit.
s

NOTE, in the interests of time, I broke this after it had added 493000+ UUID's
to run the next scripts..


** Script 2 - Start the server **

path _ 'c:\temp\bigger'.
MagmaServerConsole new
	open: path ;
	processOn: 1010


** Script 3 - Enumerate (separate image) **

| path s set reader |
path _ 'c:\temp\bigger'.
s _ MagmaSession hostAddress: #(127 0 0 1) asByteArray port: 1010.
s connectAs: 'chris'.
set _ s root.
	reader := (s root read: #asString36).
	1 to: reader size do: [ :i | | thing |
		thing := reader at: i.
		"do other things with thing here..."
		"materialize a lot of objects but no transactions"
i\\1000=0 ifTrue: [ Transcript show: '.' ].
	].
s


I let this run but the server hovered at 30MB constant for several minutes
before I interrupted it.

My guess is you probably had the commit memory-leak affecting you.  The
attached change-set seems to fix it.  I hope it fixes your memory-leak issue..

 - Chris
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MaRecoveryManager-endApply.st.gz
Type: application/x-gzip
Size: 326 bytes
Desc: 2076244045-MaRecoveryManager-endApply.st.gz
Url : http://lists.squeakfoundation.org/pipermail/magma/attachments/20050726/8251e8d8/MaRecoveryManager-endApply.st.bin