Thanks Levente, but. . . no, wrong guess , sorry :-)

Instead of memory intensive I should have said keeping memory about 95/99 % used all the time until hang.

When squeak hangs it is about 60, 80, 90 or 100Mo , not always the same, but not that much because it is a small test database (less than 500 mo in mssql) and in most databases a lot of columns have redundant values - ie bad selectivity. I've been thinking of something wrong with FFI/ODBC, but another test I just made without it have the same problem (initially with Cog but I reported it in my squeak image) : I load a 1Gb bulk export file of the table of another database into one big dictionary. With squeak It hangs in few minutes at about 60 mo , in Cog it finishes in 30 minutes at about 220 mo.

I'm trying to compile the vm (ok for now) and run it under debug but gdb is not ok at this time, I have problems with my mingw toolchain (MSVC didn't want to compile perhaps due to the express version I have), I 'm not sure it will be useful and I would prefer a "smalltalk" way to search.

I don't understand why everything seems to be ok without MessageTally ? It seems abnormal to me - am I wrong? - to see only few processes in the debug call stack ? Another symptom that makes me think processes are terminated is that in the debug output console a lot of event buffer overflow messages are displayed when I move the mouse , I suppose because the UI process has vanished and does not peek them. I tryed to intercept process termination but it didn't work, I will make another try with dump to a file it could work if they are "cleanly" terminated

Thank you BTW for your response, any ideas are very welcome

Alain

"Levente Uzonyi" leves@elte.hu a écrit dans le message de news: Pine.LNX.4.64.1010221455550.13297@login03.caesar.elte.hu...

On Thu, 21 Oct 2010, Alain_Rastoul wrote:

...

Hi Everybody, I have a standard squeak 4.1 with seaside 3.0 preloaded under Windows XP (seaside 3.0 rc OneClick Image) with FFI, ODBC packages installed and a very memory intensive application who load data from a MSSQL server and hash it into dictionaries. It is running in a process started at systemBackgroundPriority, and is launched with a MessageTally spyAllOn:. After running a few minutes it hangs up at full CPU, and I can't see what is going on, everything is hung (no Ctrl., no mouse) When I open the debug console and dump processes I see only few processes (MessageTally>spyEvery:on:, handleTimerEvent, userInterruptWatcher, lowSpaceWatcher) but no UI process and no app process. I tryed to put halts in code but it doesn't change anything. I noticed that it occurs when the memory usage is about 99% (I see it with a MemoryUsageMorph ticking), but not allways at the same time, it can load 10k, 20k or 30k rows before hanging. If I remove the MessageTally, it ran several times without hanging. I don't see any direct relation between MessageTally and hanging the VM (allocations / GC ?) I will try to generate and compile the vm with MSVC if I can (I can't test my app under linux)

What is the way go with this kind of problem ? Any help, tip for debugging, advice, idea, remark (...) are welcome. Thanks in advance

I guess you're running out of memory. What's the memory usage of Squeak when the image hangs?

Levente

...

Regards, Alain

Ok, if you don't have

Smalltalk setGCBiasToGrowGCLimit: "some magic number in megabytes" Smalltalk setGCBiasToGrow: 1.

Then you can enter into a situation (unlikely) where the amount of free space is just less than enough to make the image grow, but enough to make the software do a full compaction. Think of it as a grey zone you approach if you can pass thru it promptly the image will grow, if your movement is just not correct then then you wander in the grey zone for days... The setGCBiasToGrow: alter that behavior so we pound thru the grey zone until we've allocated setGCBiasToGrowGCLimit: where we must do a full compaction (expensive).

However what's more likely is that you actually run out of memory, this is a I believe on windows 512MB? Other platforms may cap by default at 512MB or 1GB depending on the VM and it's default settings. For non-windows platforms you can use the instructions "found somewhere?" to alter the default setting.

What happens in this case is when you approach the final bits of free memory via the hard upper cap the VM will enter a endless cycle of doing GC work and finds just enough free memory measured in kilobytes to to make another GC cycle attempt. If you are lucky this pattern will fail by the failure to allocated a needed method context and the VM will crash.

I can only suppose that MessageTally requires memory to run, is it statically bound? Or will it's memory needs grow forever?

On 2010-10-22, at 10:30 AM, Alain_Rastoul wrote:

Another precision, I have removed all gc tuning values I used before having this problem, so the vm has all the default values for gc, I don't know if it is gc or memory related.

-- =========================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ===========================================================================

Hi, In case it could help someone else, I finally found that setting vmParamterAt: 24 to 256 MB seems to have resolved my problem

"parameter 24 : memory threshold above which shrinking object memory (rw) originally 8*1024*1024" SmalltalkImage current vmParameterAt: 24 put: 256*1024*1024.

Alain

"Alain_Rastoul" alr.dev@free.fr a écrit dans le message de news: i9stcj$5j5$1@dough.gmane.org...

Hi John,

Thanks for the tips. My VM is a standard squeak VM (limited to 512 M, but this should be ok at least for the first test)

I tryed : Smalltalk setGCBiasToGrowGCLimit: 16*1024*1024. Smalltalk setGCBiasToGrow: 1.

and my image still hang. (tryed different numbers -biggers and smaller- without success)

I saw your posts about GC and GCMonitor and need to understand the GC logic to ivestigate. (unfortunatly I must stop right now :-( and can't go further today and tomorrow).

Thanks again

"John M McIntosh" johnmci@smalltalkconsulting.com a écrit dans le message de news: 2E4850AD-F676-484C-A508-09AC6E2D7F3F@smalltalkconsulting.com...

...