On Tue, Mar 12, 2002 at 09:00:59AM -0500, Kevin Fisher wrote:

Whoops!

When I file-in this changeset under Windows, it crashes the VM!

I'm using the 'latest' 3.2gamma image (with the most recent changesets), using the latest compiled VM for Windows from Andreas' site. Other details.. this is on a Win2K box...the image I'm attempting to file this in to is running a ComSwiki. Your pre-compiled DLL for Windows is also present in the squeak run directory when I file the changeset in.

Well, like I was saying, no warranty on that DLL. I built in on my Win98 box, but I don't have Win2K or anything else to test with, so you may be stuck building one.

I also have tested this OSProcess release only on Win98 and Linux, so it's entirely possible that something does not work right on other Windows flavors.

Dave

On Tue, Mar 12, 2002 at 07:39:02PM -0500, PhiHo Hoang wrote:

One way to work around this for now, don't put that dll in the path until after you filed in 'OSProcessV3-0-dtl.cs'. With this clue, maybe David can tell why. I can install it on my WindowsXP machine.

Hi PhiHo, So, does OSProcess work on your WindowXP machine when you add the DLL after the change set is loaded?

I still cannot run HUGS, 'stdOut' is not implemented for Windows ?

The console looks cool. I don't have time right now to go through your writing yet and have not tried anything except HUGS.

That's right, stdOut is not yet implemented for Windows. This will be easy to add, but the hard part is that we need to have non-blocking read and write to Windows anonymous pipes, otherwise the Squeak VM will lock up when doing command pipelines. I have some ideas on how to do this, but it might take some time to implement, so I went ahead and released the Windows OSProcess without this.

You can see the problem for yourself if you create a new OSPipe and read from it. If there is no data in the pipe, your VM will lock up and you will need to do <ctl><alt><del>. This would not be good if you were trying to read and write to an external program (HUGS) and there was no data to read.

Dave

On Wed, Mar 13, 2002 at 02:07:43AM -0500, PhiHo Hoang wrote:

Hi David,

I could file in all the change sets and tried out the 'sqsh' builtin commands.

I could even do a '3 + 4' ;-) but 'sqsh' couldn't start 'notepad.exe'.

But if I save the image and restart it (with dll in the path now), Squeak seems to hang (but you can still access the menu to show version...)

It looks like the precompiled DLL which I sent to the list is no good, at least for Win2K and WinXP. I guess for now folks will need to compile their own Win32OSProcessPlugin.dll.

If anyone has the plugin working on Win2K, please speak up. I don't have a Win2K system, so I don't know if the problem is in the plugin itself, or just something about the way I compiled it on my older Win98 system.

Dave

On Fri, Mar 15, 2002 at 03:51:43AM -0500, PhiHo Hoang wrote:

David,

You may want to look into 'primitiveGetEnvironmentStrings' which calls 'stringFromCString(p)' and crashed the GC system.

The call stack shows 'stringFromCString', 'instantiateClassindexableSize', 'sufficientSpaceAfterGC', 'incrementalGC', 'fullGC'. The problem is in this loop:

Thanks, this is a big help.

Probably I have a bug in Win32OSProcessPlugin>>primitiveGetEnvironmentStrings that only shows up on newer flavors of Windows (I am using Win98).

I'll look into it this weekend.

Dave

David T. Lewis wrote:

It looks like ObjectMemory>>pushRemappableOop: is using a stack which

has been allocated with > 25 elements, and it has no error checking to prevent stack overflows.

p.s. Just comment out the call to primitiveGetEnvironmentStrings in WindowsOSProcessAccessor>>primGetEnvironmentStrings if you want to get OSProcess working. There will probably be some things that don't

work right in

CommandShell if you do this, but at least your VM should stop crashing

;)

Instead, I pumped up the 'remapBuffer[26]' to 'remapBuffer[128]' then everything worked like a charm.

Maybe someone should revise the VM to ensure that there will be no overflows.

My only concern was that how were the numbers arrived at for things like:

int atCache[65]; int (*compilerHooks[16])(); int externalPrimitiveTable[4097]; int methodCache[4097]; int remapBuffer[26]; int rootTable[2501]; int semaphoresToSignalA[501]; int semaphoresToSignalB[501];

Cheers,

PhiHo.

-----Original Message----- From: squeak-dev-admin@lists.squeakfoundation.org [mailto:squeak-dev-admin@lists.squeakfoundation.org] On Behalf Of David T. Lewis Sent: Saturday, March 16, 2002 8:30 AM To: squeak-dev@lists.squeakfoundation.org Subject: Re: [ENH][GOODIE] OSProcess V3.0 (OSProcess for Windows)

On Fri, Mar 15, 2002 at 03:51:43AM -0500, PhiHo Hoang wrote:

David,

You may want to look into 'primitiveGetEnvironmentStrings' which

calls 'stringFromCString(p)' and crashed the GC system.

The call stack shows 'stringFromCString', 'instantiateClassindexableSize', 'sufficientSpaceAfterGC', 'incrementalGC', 'fullGC'. The problem is in this loop:

Very interesting. How large is the environment on your Windows (Win2k?) system? Are there more that 25 environment variables defined?

Here is what I think is happening:

Win32OSProcessPlugin>>primitiveGetEnvironmentStrings reads your environment block. For each entry, it creates a Smalltalk String and pushes it onto a special stack of "non-remappable" objects with ObjectMemory>>pushRemappableOop:.

It looks like ObjectMemory>>pushRemappableOop: is using a stack which has been allocated with 25 elements, and it has no error checking to prevent stack overflows.

My guess is that you have more than 25 strings in your environment, and that my code is stepping past the end of the stack and blowing up the VM as a result.

The problem would not show up on my Win98 computer because I have only a few environment variables defined. It does not show up on Unix systems because I used a different method of getting the environment in UnixOSProcessPlugin.

I'll put out some patches if a day or so. Thanks again for the debugging help.

Dave

p.s. Just comment out the call to primitiveGetEnvironmentStrings in WindowsOSProcessAccessor>>primGetEnvironmentStrings if you want to get OSProcess working. There will probably be some things that don't work right in CommandShell if you do this, but at least your VM should stop crashing ;)

On Sat, Mar 16, 2002 at 04:07:39PM -0500, PhiHo Hoang wrote:

David T. Lewis wrote:

...

It looks like ObjectMemory>>pushRemappableOop: is using a stack which

has been allocated with > 25 elements, and it has no error checking to prevent stack overflows.

...

p.s. Just comment out the call to primitiveGetEnvironmentStrings in WindowsOSProcessAccessor>>primGetEnvironmentStrings if you want to get OSProcess working. There will probably be some things that don't

work right in

...

CommandShell if you do this, but at least your VM should stop crashing

;)

Instead, I pumped up the 'remapBuffer[26]' to 'remapBuffer[128]' then everything worked like a charm.

Maybe someone should revise the VM to ensure that there will be no overflows.

Good, I justed posted a VM fix for this. Can you check to see if my proposed fix messes up your MOP project? If so, maybe my fix is not the best way to do it.

Dave

On Sat, 16 Mar 2002, PhiHo Hoang wrote:

My only concern was that how were the numbers arrived at for things like:

int atCache[65];

AFAIK, This is an optimization; it doesn't affect correctness. But, experimentation for different sizes might be fruitful. Anyone know how much the atCache affects the runtime? I may get a chance to look at it.

int (*compilerHooks[16])();
int externalPrimitiveTable[4097];

Don't know about these.

int methodCache[4097];

This is superseeded by the new methodcache.

In any case, this is an optimization; it does not impact corrrectness. But, experimentation for different sizes might be fruitful.

In the new methodcache, I arbitrarly chose 2048 slots (creating 8192 actual entries, and an array of size 65537 (256kb) to hold it), believing that no inner loop in morphic or elsewhere is going to be using more than 1000-2000 selector/class pairs at once.

int remapBuffer[26];

I know nothing about this.

int rootTable[2501];

With the root-table-overflow patch, a root table overflow costs an incrgc and a tenure operation, not a fullGC. Thus, this parameter has lost most of its importance. (For certain workloads, experimentation may show that this number should be reduced.)

In any case, barring pathological cases, AFAIK, this parameter does not impact correctness.

int semaphoresToSignalA[501];
int semaphoresToSignalB[501];

Dunno about these. Would these be a limit on the maximum number of open sockets?

Scott

On Thu, 28 Mar 2002, John M McIntosh wrote:

I'll just cherry pick the easy ones.

...

On Sat, 16 Mar 2002, PhiHo Hoang wrote:

...

My only concern was that how were the numbers arrived at for things like:

...

int atCache[65];

AFAIK, This is an optimization; it doesn't affect correctness. But, experimentation for different sizes might be fruitful. Anyone know how much the atCache affects the runtime? I may get a chance to look at it.

I presume here you're talking about the methodCache, not the atCache?

A year ago I looked at this and from a note I sent to Dan I'll quote: "I noted that in lookupInMethodCacheSelclass, the number of calls was about 50 million,and cache hits: 20M on probe1, 16M on probe2, 11M on probe3 and 1.8M not found

Thats roughtly what I saw.. The new methodcache superseeds this, with a much better overall hit rate, and a much better hit-rate for the first probe.

BTW I looked at atCache, but mmm 'since objects die young' found that over 70% of the at cache would be flushed at any given incremental GC event. (IE 70% of the time objects were in new space)."

Interesting, what about if someone changes GC parameters so that incrGC's happen a lot less often (by, say, making 100,000 allocs/incrgc). Then, doubling or quadrupling this might have a useful effect... Flushing even 32 (4x as many) entries is inconsequential compared to the methodCache which has 512, much less the cost of the incrGC.

I should study the design and consider increasing the size.

...

...

int semaphoresToSignalA[501];
int semaphoresToSignalB[501];

Dunno about these. Would these be a limit on the maximum number of open sockets?

This is my change a few years back.

on. So can you signal more than 500 interrupts in 1 to 3ms?

Ok.. Excellent.. I just saw a semaphore-related limit and wondered if it created a limit on total number of semaphores. It doesn't.. Thanks.

Scott

-- No DVD movie will ever enter the public domain, nor will any CD. The last CD and the last DVD will have moldered away decades before they leave copyright. This is not encouraging the creation of knowledge in the public domain.

On Thu, Mar 28, 2002 at 10:21:21PM -0800, Dan Ingalls wrote:

...

...

int remapBuffer[26];

I know nothing about this.

It doesn't help at all to make this bigger than necessary, but it's disastrous if it isn't big enough. It holds a stack (that is not checked for overflow!) of oops to handle code patterns like...

interpreterProxy pushRemappableOop: ffiRetClass. retOop _ self ffiCreateReturnOop: retVal. ffiRetClass _ interpreterProxy popRemappableOop.

In this case ffiCreateReturnOop could cause a GC, and thus invalidate the value of ffiRetClass. By pushing it into this buffer and then popping it back out, one is assured that if a GC does take place, an updated opp will be returned by the pop.

[In reviewing this code I think I'd figure some low-overhead way to check for overflow here. I don't think there are many places where this is time-critical and it would be an *awful* bug to track down].

I recently posted a possible fix for this under the title "[VM][BUG][FIX] pushRemappableOop: stack overflow", see http://swiki.gsug.org:8080/sqfixes/2272.html.

This is probably a reasonable fix, although to be honest I think that any primitive which overflows this stack is probably going to fail catastrophically anyway, so it might be just as well to stop the VM with error dump when this happens. In any case, from what I could tell, most existing code seems to use this stack to a depth no greater than about six (I don't remember exactly), and most primitives are just using it to a depth of one.

And yes, it was a nasty bug to track down. I had written a primitive which answers an arbitrarily sized array of strings (the collection of environment strings on a Windows OS), and which was building the array on the fly. If the array had more than 25 entries, bad things happened. I'm afraid that I have gotten so accustomed to Squeak doing the right thing with collections that I completely forgot to look at what kind of "collection" I was dealing with in the VM ;)

Dave

Hi Scott, Hi John, Hi Dan,

Sorry, I couldn't get back to you all sooner. I have been a bit busy putting a final touch on my 'double clickable FreeCell.st'. More details in another message, though.

Thanks for your enlightments. At John's suggestion and to give Scott a toy to play in his optimisation effort, I made it configurable from the initialisation file 'SqM.ini' the following parameters:

(lines with ';;' at the front are commented out).

[TuningParams] ;; Was 4097 Now Default: 8192 ;; _EXTERNAL_PRIMITIVE_TABLE_SIZE_ = 16384

;; Was 501 Now Default: 1024 ;; _SEMAPHORES_TO_SIGNAL_SIZE_ = 512

;; Was 4097 Now Default: 8192 _METHOD_CACHE_SIZE_ = 16384

;; Was 65 Now Default: 128 ;; _AT_CACHE_SIZE_ = 35

;; Was 26 Now Default: 64 ;; _REMAP_BUFFER_SIZE_ = 16

;; Was 2501 Now Default: 4096 ;; _ROOT_TABLE_SIZE_ = 4096

It was found that the VM will crash at different places as _AT_CACHE_SIZE_ is reduced.

Higher _METHOD_CACHE_SIZE_ and lower _AT_CACHE_SIZE_ tend to crash the VM.

This doesn't seem to enforce Dan's statement:

...

int atCache[65];

Well, I just thought about what it was doing and guessed at what it

would take.

You could try doubling it and halving it. I think it will provide most

of its

benefit with only one entry.

Did I misunderstand this statement ?

I will leave it as an exercise for those who will take a trip in the SqueakMobile to implement some primitives to modify these parameters to 'change the flat tires and clean the spark plugs' while zooming back to the future. ;-)

Cheers,

PhiHo.

-----Original Message----- From: Scott A Crosby [mailto:crosby@qwes.math.cmu.edu] Sent: Thursday, March 28, 2002 10:36 PM To: PhiHo Hoang Cc: Squeak List Subject: Fixed limits in the image. (was Re: ...)

On Sat, 16 Mar 2002, PhiHo Hoang wrote:

My only concern was that how were the numbers arrived at for things like:

int atCache[65];

AFAIK, This is an optimization; it doesn't affect correctness. But, experimentation for different sizes might be fruitful. Anyone know how much the atCache affects the runtime? I may get a chance to look at it.

int (*compilerHooks[16])();
int externalPrimitiveTable[4097];

Don't know about these.

int methodCache[4097];

This is superseeded by the new methodcache.

In any case, this is an optimization; it does not impact corrrectness. But, experimentation for different sizes might be fruitful.

In the new methodcache, I arbitrarly chose 2048 slots (creating 8192 actual entries, and an array of size 65537 (256kb) to hold it), believing that no inner loop in morphic or elsewhere is going to be using more than 1000-2000 selector/class pairs at once.

int remapBuffer[26];

I know nothing about this.

int rootTable[2501];

With the root-table-overflow patch, a root table overflow costs an incrgc and a tenure operation, not a fullGC. Thus, this parameter has lost most of its importance. (For certain workloads, experimentation may show that this number should be reduced.)

In any case, barring pathological cases, AFAIK, this parameter does not impact correctness.

int semaphoresToSignalA[501];
int semaphoresToSignalB[501];

Dunno about these. Would these be a limit on the maximum number of open sockets?

Scott

Hi Scott,

The reason is that the compiler takes these constants and inlines them

into various code in

the image. Thus, these can't be changed without retranslating the

interpreter.

The size of these arrays are now variables and the 'arrays' are dynamically allocated.

'Interp.c' was searched for these constants and replaced with variables, so I hope this approach is as good as retranslating the interpreter with new arrays size.

Unless, of course, these constants are used and hard coded into other thingies. Maybe that's the reason why it's so hard to modularise Squeak ?

While I was searching for these constants, I felt quite itchy when I saw arrays initialised from 1 instead of 0. But I know it's wise to leave it alone ;-)

So far, I have no troubles playing my FreeCell with this configurable VM, as long as I do not use a small array for atCache and a big array for methodCache. It also survives Macro Benchmarks, no crashing.

While I get hold of you, what's the status of the integration between your and Scott's enhancements to the VM ? Please keep up with the good work.

Thanks for your help, I appreciated it.

Cheers,

PhiHo.

-----Original Message----- From: Scott A Crosby [mailto:crosby@qwes.math.cmu.edu] Sent: Sunday, March 31, 2002 10:33 PM To: PhiHo Hoang Cc: 'Squeak List' Subject: RE: Fixed limits in the image. (was Re: ...)

On Sun, 31 Mar 2002, PhiHo Hoang wrote:

Hi Scott, Hi John, Hi Dan,

Thanks for your enlightments. At John's suggestion and to give

Scott

a toy to play in his optimisation effort, I made it configurable from the initialisation file 'SqM.ini' the following parameters:

It was found that the VM will crash at different places as _AT_CACHE_SIZE_ is reduced.

Higher _METHOD_CACHE_SIZE_ and lower _AT_CACHE_SIZE_ tend to crash the

VM.

The reason is that the compiler takes these constants and inlines them into various code in the image. Thus, these can't be changed without retranslating the interpreter.

I responded to your post because I thought it wise to elaborate on what I knew and see if anyone else knew anything.

Scott

Hi Scott, Hi John, Hi Dan,

Sorry, I couldn't get back to you all sooner. I have been a bit busy putting a final touch on my 'double clickable FreeCell.st'. More details in another message, though.

Thanks for your enlightments. At John's suggestion and to give Scott a toy to play in his optimisation effort, I made it configurable from the initialisation file 'SqM.ini' the following parameters:

(lines with ';;' at the front are commented out).

[TuningParams] ;; Was 4097 Now Default: 8192 ;; _EXTERNAL_PRIMITIVE_TABLE_SIZE_ = 16384

Well I was thinking more of changes to SystemDictionary>>vmParameterAt:put:

So you can change the values on the fly, versus having to recompile the VM each time you want to change the size of these arrays. Just malloc a new array and take what action is required to move things over from the old array if needbe.

Also remembering highwater marks that you could access via SystemDictionary>>vmParameterAt: might be a useful thing.