Vm-dev October 2010

vm-dev@lists.squeakfoundation.org

27 participants
60 discussions

What is meant by 'shrink' in the VM parameters?
by Ken Causey 22 Oct '10

22 Oct '10

I'm wondering what 'shrink' means in the following VM parameter comments: "24 memory threshold above which shrinking object memory (rw)" and "32 number of shrink memory requests (read-only)" My first thought is that a shrink represents Squeak release memory back to the operating system once it is no longer needed. As I've mentioned before we are trying to reduce the RAM footprint of the www.squeak.org process and are watching various related figures. We can see the value in VM parameter 32 increase, but while the value for parameter 3 (end of memory) goes down as you would expect, the operating system's view of the memory usage does not change. Ken

2 5

VM Maker: VMMaker-dtl.195.mcz
by squeak-dev-noreply＠lists.squeakfoundation.org 22 Oct '10

22 Oct '10

Dave Lewis uploaded a new version of VMMaker to project VM Maker: http://www.squeaksource.com/VMMaker/VMMaker-dtl.195.mcz ==================== Summary ==================== Name: VMMaker-dtl.195 Author: dtl Time: 21 October 2010, 9:20:38 am UUID: 7659f426-0e81-4dfe-9b75-507879a6f2e9 Ancestors: VMMaker-dtl.194 VMMaker 4.3.9 Adopt LargeIntegerPlugin fixes from oscog for #digitSubLarge:With: and #unsaveByteOf:at:. KernelTests-Numbers are green both before and after the change, but oscog updates are more recent and presumed to be correct. Include additional oscog CCodeGenerator updates for proper TReturnNode translation, required for the LargeIntegerPlugin updates. Method stamps for John M McIntosh were improperly mangled in the process of converting to pragmas. Fix these method stamps.

1 0

Re: [squeak-dev] What are we leaking in the www.squeak.org process?
by Ken Causey 21 Oct '10

21 Oct '10

Janko and I have been doing some fiddling around and monitoring today and I begin to develop a theory. First, meet some images: Image 1: SqueakMap (map.squeak.org). This is our oldest Squeak service image with the possible exception of wiki.squeak.org. This is a 3.8 image running on a 3.8 VM. SqueakMap saves it's data to a separate file and snapshots of the image are only made when source code changes occur, aka manually. Image 2: SqueakSource (source.squeak.org). This is a 3.11 image running on a 3.11 VM. This image saves it's data by snapshoting hourly. Image 3: www.squeak.org. This is a 3.9 VM that has been running for some time on the same 3.8 VM as SqueakMap but we just a short while ago decided to try it on the 3.11 VM SqueakSource is using and it is running fine so far. So for a few hours now I have been monitoring the memory usage and GC behavior of these three images. The short story is that SqueakMap is almost completely rock solid but that the SqueakSource image, despite my earlier claims to the contrary, appears to suffer a much reduced form of the problem we are observing on www.squeak.org. To reiterate: The problem is that the resident set size of the process appears to continually grow. The operating system will occasionally swap out some of this giving the impression that the RSS has dropped but I'm almost certain that the total cost (RAM+swap) of the process is the same and continuing to increase. Our available RAM and swap on the server is quite limited with all the various services we are running. What I'm observing is a discrepancy between what the VM claims is the end of memory and the RSS. When any of these images start this discrepancy is quite small, maybe a megabyte. But whenever a GC occurs the end of memory figure drops, but the RSS goes up. So from the images side it all appears copacetic: we cleaned up the garbage and we are more or less back at our base memory usage level. But as far as the operating system is concerned nothing has happened, in fact more memory has been claimed. So my theory is that somehow when GC occurs on the linux VM (3.8 VM and 3.11 VM) the freed up memory is not released to the operating system but it seems that new objects go in a different memory location from the operating system's perspective therefore using additional memory on top of what was used before the GC. I only really observe this on full GCs. When few if any full GCs occur, there is no real problem. But when you do things like snapshot the image hourly, this all adds up to several megabytes a day of lost RAM. This may not be quite right, I'm still observing and of course this theory is built out of black box observation and little else. Feel free to set me straight. Ken

1 0

Fwd: [Vm-dev] VM Maker: VMMaker-dtl.194.mcz
by Eliot Miranda 21 Oct '10

21 Oct '10

---------- Forwarded message ---------- From: Eliot Miranda <eliot.miranda(a)gmail.com> Date: Wed, Oct 20, 2010 at 5:01 PM Subject: Re: [Vm-dev] VM Maker: VMMaker-dtl.194.mcz To: "David T. Lewis" <lewis(a)mail.msen.com> On Wed, Oct 20, 2010 at 4:43 PM, David T. Lewis <lewis(a)mail.msen.com> wrote: > On Wed, Oct 20, 2010 at 10:02:31AM -0700, Eliot Miranda wrote: > > On Tue, Oct 19, 2010 at 10:55 PM, < > > squeak-dev-noreply(a)lists.squeakfoundation.org> wrote: > > > > > > > Added #asExpression to > TSendNode>>emitCCodeAsArgumentOn:level:generator: > > > needed for cpp code generation for expressions. > > > > > > Methods #generateInlineCPreprocessorDirective:on:indent: and > > > #generateCppDirective:on:indent: are redundant, get rid of one (or > both) of > > > them. Consider using #isDefined:inSmalltalk:comment:ifTrue:ifFalse: > instead > > > in Cog. > > > > > > > Ooh! I like it. Why didn't I think of this? Make it so. The > > cPreprocessorDirective: was my most egregious hack. But I would just use > > #ifDefined: aPreprocessorMacroName ifTrue: [] ifFalse: [] > > and then use cCode:inSmalltalk: within each arm as appropriate rather > than > > conflating the two. > > Hi Eliot, > > The cCode:inSmalltalk: part actually works pretty well and translates > nicely to C, see #isIntegerObject: for the most important example. > > The following are already implemented in VMMaker "trunk". The first two are > important, the third is unreferenced, and the fourth is both unreferenced > and redundant with the oscog implementation. > #isDefinedTrueExpression:inSmalltalk:comment:ifTrue:ifFalse: > #isDefined:inSmalltalk:comment:ifTrue:ifFalse: > #isDefined:inSmalltalk:comment:ifTrue: > #preprocessorExpression: > > The #cPreprocessorDirective: implementation is important for Cog, and > is now implemented in both VMMaker branches (but could be replaced > as noted above). > > So my suggestion would be to add these into oscog: > #isDefinedTrueExpression:inSmalltalk:comment:ifTrue:ifFalse: > #isDefined:inSmalltalk:comment:ifTrue:ifFalse: > #isDefined:inSmalltalk:comment:ifTrue: > > And then get rid of #preprocessorExpression: in favor of your > #cPreprocessorDirective: > > As a practical matter, I'm not sure the best way to include this in the > oscog branch. I can send a change set if that is helpful, or you can pick > updates from Montecello. Unless you advise otherwise, I don't think that > I should make any commits to the oscog branch, as that is likely to lead > to confusion. > A change set ifs fine. But I still don't quite understand why you need the inSmalltalk: argument. I'm assuming self ifDefined: aConstantName inSmalltalk: [smalltalk statements] comment: aComment ifTrue: [true statements] ifFalse: [false statements] then surely that's equivalent to self ifDefined: aConstantName comment: aComment ifTrue: [self cCode: [true statements] inSmalltalk: [smalltalk statements]] ifFalse: [self cCode: [false statements] inSmalltalk: [smalltalk statements]] which is much nicer IMO because it is orthogonal. But am I not understanding and instead the argument to inSmalltalk: is deciding whether isDefined: is true at execution time? If so, I get it, it makes sense, but I find the reuse of inSmalltalk: confusing (doh!). best Eliot > > Dave > >

3 2

VM Maker: VMMaker-dtl.194.mcz
by squeak-dev-noreply＠lists.squeakfoundation.org 21 Oct '10

21 Oct '10

Dave Lewis uploaded a new version of VMMaker to project VM Maker: http://www.squeaksource.com/VMMaker/VMMaker-dtl.194.mcz ==================== Summary ==================== Name: VMMaker-dtl.194 Author: dtl Time: 19 October 2010, 11:52:12 am UUID: 9dd94fba-0da6-49f7-820c-9af777558bb7 Ancestors: VMMaker-dtl.193 VMMaker 4.3.8 Adopt CCodeGenerator improvements from oscog, notably including proper C code generation for ifTrue:ifFalse: constructs as expressions, as well as methods used in Cog code generation. Produces equivalent C code for the traditional interpreter, modulo some cosmetic differences in the output (line breaks, spaces, etc). Notes: Added #asExpression to TSendNode>>emitCCodeAsArgumentOn:level:generator: needed for cpp code generation for expressions. Methods #generateInlineCPreprocessorDirective:on:indent: and #generateCppDirective:on:indent: are redundant, get rid of one (or both) of them. Consider using #isDefined:inSmalltalk:comment:ifTrue:ifFalse: instead in Cog.

3 2

Cog: Infinite recursion leads to crash
by Igor Stasenko 20 Oct '10

20 Oct '10

Hello, Eliot. By running some experimental stuff under Cog, Chris get into an infinite resursion, which, instead of showing 'low space' warning leads to VM crash. I am not sure what exactly caused recursion in his code (i hope this just a simple bug, not related to VM). What i found disturbing, that i was able to reproduce crash with simple code (see attachment). The only difference between crash report of mine: Smalltalk stack dump: 0x924088 M InfiniteRecursion>deeeep: 292975484: a(n) InfiniteRecursion 0x9240ac M InfiniteRecursion>deeeep: 292975484: a(n) InfiniteRecursion 0x91e6f8 M InfiniteRecursion>deeeep: 292975484: a(n) InfiniteRecursion 0x91e71c M InfiniteRecursion>deeeep: 292975484: a(n) InfiniteRecursion 0x91e740 M InfiniteRecursion>deeeep: 292975484: a(n) InfiniteRecursion 0x91e764 M InfiniteRecursion>deeeep: 292975484: a(n) InfiniteRecursion 0x91e788 M InfiniteRecursion>deeeep: 292975484: a(n) InfiniteRecursion .... and Chris, that in his case an objects, somehow went outside of a heap: Smalltalk stack dump: 0xbfa31668 M class()> 20181257720x784a27cc is not on the heap 0xbfa31690 M class( class)> 20268642800x78cf7e98 is not on the heap 0xbfa316a8 M class( class)> 20268642800x78cf7e98 is not on the heap 0xbfa316c8 M ()> 20366153080x7964488c is not on the heap 0xbfa316e0 M class> 20129812280x77fba7ec is not on the heap 0xbfa316f8 M > 20135071280x7803ae38 is not on the heap 0xbfa31720 M class( class)> 20268642800x78cf7e98 is not on the heap 0xbfa31738 M class( class)> 20268642800x78cf7e98 is not on the heap 0xbfa31758 M ()> 20366153080x7964488c is not on the heap 0xbfa31770 M class> 20129812280x77fba7ec is not on the heap 0xbfa31788 M > 20135071280x7803ae38 is not on the heap ... I was running under windoze, and were able to watch how vm consuming memory, and somewhere little above 512Mb, it crashing. It is bad, of course, when your own code going mad, but it should not be the reason for VM to go mad as well :) -- Best regards, Igor Stasenko AKA sig.

2 1

Re: [squeak-dev] What are we leaking in the www.squeak.org process?
by Ken Causey 20 Oct '10

20 Oct '10

To add a bit more data to this. I surveyed the rest of the VM Parameters. (1-25, above 25 I was getting primitive failures and I don't believe any of those are relevant, let me know if I am mistaken.) And I can't find any data there that explains the discrepancy reported between VM Parameter 3 (end of memory) and the resident set size of the process as reported by the operating system. Ignoring those entries that are clearly counts of GC iterations and GC runtimes and such all of the rest of the values that seem like they might be related to memory usage summed up does not come close the current discrepancy of about 70MB. I would really really appreciate some guidance here. Ken On Fri, 2010-10-15 at 15:05 -0500, Ken Causey wrote: > On Thu, 2010-10-14 at 19:00 -0700, Ken Causey wrote: > > > -------- Original Message -------- > > > Subject: Re: [Webteam] Re: [squeak-dev] What are we leaking in the > > > www.squeak.org process? > > > From: Igor Stasenko <siguctua(a)gmail.com> > > > Date: Thu, October 14, 2010 8:45 pm > > > To: The general-purpose Squeak developers list > > > <squeak-dev(a)lists.squeakfoundation.org> > > > > > > > > > On 15 October 2010 04:10, Ken Causey <ken(a)kencausey.com> wrote: > > > > Another update with more data conflicting somewhat with my last report: > > > > > > > > Shown by ps: 214560KB RSS > > > > Shown by vmParameters (2,3): 64.6MB/67.4MB > > > > > > > > A discrepancy of some 150MB. > > > > > > My understanding is, that Squeak VM cannot shrink a memory which were > > > commited (due to peak growth), > > > to give it back to OS. > > > > > > So, if at some peak, VM comitted 150 MB, and OS sees it as 150M, > > > while VM after GC actually using just 64M. > > > > No, I don't believe this is true. The RSS does go both down as well as > > up. I thought I made that clear in my original comments. > > > > Although I suppose it is possible that drops in the RSS figure are due > > to paging out to swap. I'm not sure how to really check that. > > > > Ken > > For example, another data point: > > Shown by ps: 167344KB RSS > Shown by vmParameters (2,3): 62.9MB/65.8MB > > The process has not been restarted but the RSS has dropped. > > Ken > > > > > > > > > > > > Ken > > > > > > > > On Thu, 2010-10-14 at 17:31 -0500, Ken Causey wrote: > > > >> An update as things may not be quite as I originally described them. It > > > >> seems the values reported by vmParameters 2 and 3 are not as constant as > > > >> I had been lead to believe. I noticed that just a short while after > > > >> sending the image the RSS had jumped up to 165800KB. I still had VNC > > > >> open on the image so I checked what is shown there and I see > > > >> 145.6MB/147.7MB so there is still a discrepancy, but not perhaps as wide > > > >> as I indicated. > > > >> > > > >> I also did not mention that in addition to restarting Swazoo every 15 > > > >> minutes there is other cleanup that is done once a day related to > > > >> statistics gathering that goes on. My understanding is that this > > > >> cleanup is meant to return things very close to the starting state. > > > >> Perhaps Janko can comment on that more fully. > > > >> > > > >> Ken > > > >> > > > >> On Thu, 2010-10-14 at 16:55 -0500, Ken Causey wrote: > > > >> > For some time now I've been bugging Janko about the memory usage of the > > > >> > process that serves http://www.squeak.org/ . I've gathered a bit more > > > >> > info today and I'd like to see if anyone here has any thoughts. > > > >> > > > > >> > First while there is some variation most of the Squeak processes that > > > >> > serve squeak.org sites (source.squeak.org, map.squeak.org, > > > >> > wiki.squeak.org) have a more or less constant and reasonable memory > > > >> > usage. Let me clarify that what I am talking about here is RSS > > > >> > (Resident Set Size) as reported by ps/top on Linux. Swiki as the oldest > > > >> > of the group is the clear winner at just under 30MB, squeakmap is around > > > >> > 47MB, and source.squeak.org is around 59MB. > > > >> > > > > >> > In contrast immediately after starting it www.squeak.org is using about > > > >> > 70MB and it grows from there. Now a bit more than an hour since the > > > >> > last restart it is already up to about 85MB. I have seen it over 300MB > > > >> > before, and it commonly uses 140-250MB. It doesn't simply grow and > > > >> > grow, it does shrink occasionally. > > > >> > > > > >> > First, there is a known issue with Swazoo that it does not close sockets > > > >> > correctly and Janko has addressed this by having Swazoo automatically > > > >> > restarted every 15 minutes. So using netstat for example I can watch > > > >> > the socket count for the process continually grow and then it drops to > > > >> > zero and this repeats. In contrast for all the other process (all > > > >> > running some variation of Kom/Comanche I believe) I'm hard pressed to > > > >> > even catch a single open socket at any time (not counting the listener > > > >> > that is). > > > >> > > > > >> > Secondly, since I've been complaining about this issue for a long time > > > >> > Janko has running a little morph that shows current memory usage. What > > > >> > this shows is the values of SmalltalkImage current vmParameterAt: 2 and > > > >> > 3. These numbers are quite stable. For example at this moment these > > > >> > are 68.3MB and 71.0MB respectively. Whereas ps shows the RSS at > > > >> > 85752KB. Where is the discrepancy here? > > > >> > > > > >> > My current theory is that the vmParameters are not reflecting incidental > > > >> > memory use by the VM and plugins or that we are simply misinterpreting > > > >> > the values. Is it possible that Swazoo is failing to release something > > > >> > that is causing the VM to use more memory than it should? > > > >> > > > > >> > Note that the same VM is used by all of these processes. It is > > > >> > > > > >> > 3.8a-1 #1 Sun May 1 19:46:46 EDT 2005 gcc 3.3.5 > > > >> > > > > >> > Yes, I know it's old, but 'if it ain't broke, don't fix it' has been the > > > >> > motto and with the possible exception of Swazoo/www.squeak.org it has > > > >> > been working wonderfully for a long time now. > > > >> > > > > >> > Your thoughts would be greatly appreciated, > > > >> > > > > >> > Ken > > > >> > > > >> _______________________________________________ > > > >> Webteam mailing list > > > >> Webteam(a)lists.squeakfoundation.org > > > >> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Igor Stasenko AKA sig. > > > > > > > > _______________________________________________ > Webteam mailing list > Webteam(a)lists.squeakfoundation.org > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam

1 0

Strawman proposal for m17n
by K. K. Subramaniam 16 Oct '10

16 Oct '10

Hi, I wanted to revisit old discussions in 2003/2005/2007 about getting Squeak VM to handle multilingual inputs in Indic context. Indic keyboard input can come through XIM in multiple languages regardless of the locale setting. LANG may be set to en_US.UTF-8 or en_IN.UTF-8. XIM input method engines are used to generate multilingual keystrokes so the app only sees UTF-8 encoded characters, not keys. The current design for passing multiple encodings into the image will not work for m17n. Currently, the logic for keycode and the keychar (i.e. character typed, possibly composed of multiple keycodes) are splattered across the VMs and images. Tying input encoding to locales complicates Indic support. Composition is platform-specific and is best handled in VM. It looks to me that the complications are due to multiplexing of two keyboard input streams here - control codes (buttons) and text codes (Characters). Buttons are used to fire operations while Characters go into text streams. Button codes need to deal with modifiers and codes < 127 but not with m17n, AFAIK. Characters codes don't need modifiers but need to deal with m17n issues. Here is my proposal to move forward without affecting existing deployments: 1. Map key codes into button codes (e.g. OK, Cancel, Cut, Copy, ...) in the VM itself and pass only button codes into the image. I like Lex Spoon's proposal to start with X11 encodings for buttons (keysymdef.h). One of the button codes can be reserved for a soft keyboard. New images can start using these codes and be ready to run on handhelds and tablets too. 2. Henceforth all VMs will encode all char inputs in utf8 except for non-en Latin1 locales. For these locales, latin1 will be used by default and utf8 if -compositioninput is used. The new VM will pass a dummy button on startup to signal input encoding, or we could introduce a new primitive to signal this state. New images can use it to unify clipboard and input interpreters. Old images can be patched to work with new VMs. I am not fully aware of past history in code page issues for latin1 locales, so this is just a strawman. Please do point out gaps in it. Thanks .. Subbu

1 1

VM Maker: SlangBrowser-dtl.4.mcz
by squeak-dev-noreply＠lists.squeakfoundation.org 16 Oct '10

16 Oct '10

Dave Lewis uploaded a new version of SlangBrowser to project VM Maker: http://www.squeaksource.com/VMMaker/SlangBrowser-dtl.4.mcz ==================== Summary ==================== Name: SlangBrowser-dtl.4 Author: dtl Time: 16 October 2010, 11:17:05 am UUID: 28c89fcd-a3dc-45f9-b71e-adcbaa08cd52 Ancestors: SlangBrowser-dtl.3 Clear code generator cache on notification of class-side changes, not just instance side changes. Use ObjectMemory>>initializeConstants to reflect recent updates in VMMaker 4.2.3 (VMMaker-dtl.177) and later.

1 0

libc6 requirements for Linux VM
by K. K. Subramaniam 16 Oct '10

16 Oct '10

Hi, This is continuation of a discussion at etoys-dev http://lists.squeakland.org/pipermail/etoys-dev/2010-October/005949.html Essentially, 4.x Linux VMs may fail with an error: Could not load library: /lib/libc.so.6: version `GLIBC_2.7' not found This is due to their dependency on libc6-2.7 or later. This version shipped in Ubuntu 8.04+ and kernels 2.6.24+. Machines with 256MB RAM running perfectly well on Ubuntu 7.10 became terribly slow when upgraded to 8.04 (google for "ubuntu hardy slow"). Linux kernel degraded poorly when faced with memory pressure caused by increased working set of 8.04. People with 256MB RAM systems (e.g. schools) could not upgrade to distros with libc6 v2.7+ without upgrading their hardware. XOs have 256MB RAMs and so do virtual machines that run in qemu or kvm. So the increased requirement may affect these deployments. Has anyone faced this issue in Squeak v4.1 on production machines? Subbu

2 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Vm-dev October 2010