Hi Folks -
Since we're nearing release status I was going over the VM requirements for Croquet to see if we could just go with a bunch of standard VMs or whether we'd need some customized ones. Looking over some recent VMMaker versions, it turns out that there are actually a few important bits and pieces missing from the current VMMaker release which we will want to use in Croquet.
Most importantly, we need the following issues resolved:
1) GC improvements: As discussed in April last year (see http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse?list=vm-dev&cmd...) we need the weak array fixes and the GC measurements/tweaks that John did. Those actually turn out to be important for us, if only for analysis of current behavior.
Questions: * What's the general status of those changes? The way I understand the discussion we had back in april, is that John did the adjustments to 64bit and then ... what? * @Tim: Does a VMMaker repository exist by now? Like I was saying back then, I am not alltogether happy with the current situation of new releases just popping up on SqueakMap and with no way of seeing what's in there and how things have developed.
2) Floating point accuracy: As discussed in november last year (see http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse?list=vm-dev&cmd...) we need consistent floating point behavior across platforms. I'm actually in the process or rolling out a bunch of tests for it which will be based on fdlibm as described in the above message. One issue that is critical here is that I am planning to checksum (SHA1) the VMs while we are running Croquet to see if the VM is "compliant" (if not we'd run a whole suite of tests to verify the behavior).
Questions: * On Mac and Unix, do we implement getSystemAttribute: -1 to return the full path to the executable? * On Mac and Unix, can one read the executable while running Squeak? I need this to checksum the VM proper.
3) Some Socket fixes: David Reed made some changes for Sockets which turn out to be critical in particular on Macs and Unix systems. I'm still in the process of bundling up these changes but they are fairly straightforward so I don't see many issues with them.
Besides the above three, I have one additional question: What is the general status of the 64bit work right now? Do we consider this complete or work in progress or experimental? Can we rule out any ill effects for the 32bit VMs?
Thanks and Cheers, - Andreas
On 16-Mar-06, at 2:10 PM, Andreas Raab wrote:
Hi Folks -
Since we're nearing release status I was going over the VM requirements for Croquet to see if we could just go with a bunch of standard VMs or whether we'd need some customized ones. Looking over some recent VMMaker versions, it turns out that there are actually a few important bits and pieces missing from the current VMMaker release which we will want to use in Croquet.
Most importantly, we need the following issues resolved:
- GC improvements: As discussed in April last year
(see http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse? list=vm-dev&cmd=showmsg&msgnum=184) we need the weak array fixes and the GC measurements/tweaks that John did. Those actually turn out to be important for us, if only for analysis of current behavior.
Questions:
- What's the general status of those changes? The way I understand
the discussion we had back in april, is that John did the adjustments to 64bit and then ... what?
These were folded into the VMMaker in mid may 2005, waiting feedback from others.
- @Tim: Does a VMMaker repository exist by now? Like I was saying
back then, I am not alltogether happy with the current situation of new releases just popping up on SqueakMap and with no way of seeing what's in there and how things have developed.
- Floating point accuracy: As discussed in november last year
(see http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse? list=vm-dev&cmd=showmsg&msgnum=365) we need consistent floating point behavior across platforms. I'm actually in the process or rolling out a bunch of tests for it which will be based on fdlibm as described in the above message. One issue that is critical here is that I am planning to checksum (SHA1) the VMs while we are running Croquet to see if the VM is "compliant" (if not we'd run a whole suite of tests to verify the behavior).
I did load that floating point lib and it works on the mac vm, waiting a decision from others...
Questions:
- On Mac and Unix, do we implement getSystemAttribute: -1 to return
the full path to the executable?
- On Mac and Unix, can one read the executable while running
Squeak? I need this to checksum the VM proper.
Mmmm let me see, nope (-1) doesn't work, do you know if that works with unix?
You should be able to read the vm file, although you should consider at some point a vm could be marked as executable and not readable.
- Some Socket fixes: David Reed made some changes for Sockets
which turn out to be critical in particular on Macs and Unix systems. I'm still in the process of bundling up these changes but they are fairly straightforward so I don't see many issues with them.
Well then share.
Besides the above three, I have one additional question: What is the general status of the 64bit work right now? Do we consider this complete or work in progress or experimental? Can we rule out any ill effects for the 32bit VMs?
? Ask Ian or Dan
Thanks and Cheers,
- Andreas
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
Hi John -
Thanks for the information.
- GC improvements:
Questions:
- What's the general status of those changes? The way I understand the
discussion we had back in april, is that John did the adjustments to 64bit and then ... what?
These were folded into the VMMaker in mid may 2005, waiting feedback from others.
So is there a "current" VMMaker version that includes these? (I'm asking since when Josh sent the message lately you replied by pointing to a webpage with a few bits of loose code on there ;-)
Questions:
- On Mac and Unix, do we implement getSystemAttribute: -1 to return
the full path to the executable?
- On Mac and Unix, can one read the executable while running Squeak? I
need this to checksum the VM proper.
Mmmm let me see, nope (-1) doesn't work, do you know if that works with unix?
No, I don't know. The main thing is that I need a way of computing the checksum for the actual binary (not just a shell script which invoked the the binary somehow) and I would expect that getSystemAttribute: 0 gives us the shell script, no?
You should be able to read the vm file, although you should consider at some point a vm could be marked as executable and not readable.
An alternative would be to require a primitive that gives us a build UUID but I'm trying to avoid that.
Cheers, - Andreas
- GC improvements: As discussed in April last year
(see http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse? list=vm-dev&cmd=showmsg&msgnum=184) we need the weak array fixes and the GC measurements/tweaks that John did. Those actually turn out to be important for us, if only for analysis of current behavior.
Those changes were incorporated LAST APRIL. If you had bothered to actually build a VM with any vmmaker since then you'd have found them in as agreed. The latest vmmaker can be found on SM, along with several earlier versions.
- @Tim: Does a VMMaker repository exist by now? Like I was saying
back then, I am not alltogether happy with the current situation of new releases just popping up on SqueakMap and with no way of seeing what's in there and how things have developed.
No, there is no vmmaker repository.
- Floating point accuracy: As discussed in november last year
(see http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse? list=vm-dev&cmd=showmsg&msgnum=365)
Yes, and .... what? I have no record of anything going beyond a vague suggestion. How would this have anything to affect vmmaker? You want to use it, you change your makefile settings.
Questions:
- On Mac and Unix, do we implement getSystemAttribute: -1 to return
the full path to the executable?
See http://minnow.cc.gatech.edu/squeak/314 This is a spec that was agreed to about 8 years ago. If any platform is non-compliant it should be brought into compliance asap.
- Some Socket fixes: David Reed made some changes for Sockets
which turn out to be critical in particular on Macs and Unix systems. I'm still in the process of bundling up these changes but they are fairly straightforward so I don't see many issues with them.
Submit them in the usual manner and I'll look at them sometime.
Besides the above three, I have one additional question: What is the general status of the 64bit work right now? Do we consider this complete or work in progress or experimental? Can we rule out any ill effects for the 32bit VMs?
As far as I'm concerned the 64 bit stuff is pathetically sulking in a corner due to almost nobody having paid it any attention. I have done everything reasoanble for vmmaker but there are a largish number of places where there are 'flag: #dan' type comments and there are places where there are comments about some change needing work someday. The only VMs I know of that have been built and released with the 64bit changes incorporated have been on RISC OS and OSX so to the best of my knowledge there is no information on whether they are any use on windoze or *nix. I was considering backing out all the changes since it seems that nobody cares even fractionally enough to finish what they started.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: ED: Eject Disk
On 16-Mar-06, at 5:13 PM, tim Rowledge wrote:
Questions:
- On Mac and Unix, do we implement getSystemAttribute: -1 to
return the full path to the executable?
See http://minnow.cc.gatech.edu/squeak/314 This is a spec that was agreed to about 8 years ago. If any platform is non-compliant it should be brought into compliance asap.
Sigh The mac vm has always returned the path to the vm, and not the name of the executable, note the comment in getSystemAttribute: However I'll consider changing that.
Smalltalk getSystemAttribute: -1
does nothing on the mac or unix unless you've passed in a parameter on the cmd line.
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
Hi Tim -
- GC improvements: As discussed in April last year
(see http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse?list=vm-dev&cmd...)
we need the weak array fixes and the GC measurements/tweaks that John did. Those actually turn out to be important for us, if only for analysis of current behavior.
Those changes were incorporated LAST APRIL. If you had bothered to actually build a VM with any vmmaker since then you'd have found them in as agreed. The latest vmmaker can be found on SM, along with several earlier versions.
No reason to get upset. I was just asking for a status update. I checked with 3.8b4 as reference version since all the others have big fat disclaimers on them about "use at your own peril etc." and 3.8b4 (btw, SqueakMap says it's from september '05) does not contain these changes. They first show up in 3.8b5-64 (from december '05). And talking about versions what *is* the latest stable version? SqueakMap lists 3.8b5-64B as the latest; is there any version that is either newer or considered "non-beta" or both?
- @Tim: Does a VMMaker repository exist by now? Like I was saying back
then, I am not alltogether happy with the current situation of new releases just popping up on SqueakMap and with no way of seeing what's in there and how things have developed.
No, there is no vmmaker repository.
Okay, can we agree on one? Say, source.squeakfoundation.org?
- Floating point accuracy: As discussed in november last year
(see http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse?list=vm-dev&cmd...)
Yes, and .... what? I have no record of anything going beyond a vague suggestion. How would this have anything to affect vmmaker? You want to use it, you change your makefile settings.
I didn't say it would affect VMMaker. I am trying to find out whether it is possible on various platforms to checksum the VM since we have very specific requirements (like float behavior) for Croquet. Also, in the post back then I was asking the question about what people would think about building entire VMs using fdlibm. That's still on the table (but the lack of comments back then makes me think that I'll probably opt for a MathPlugin instead of fixing the VMs internally).
Questions:
- On Mac and Unix, do we implement getSystemAttribute: -1 to return
the full path to the executable?
See http://minnow.cc.gatech.edu/squeak/314 This is a spec that was agreed to about 8 years ago. If any platform is non-compliant it should be brought into compliance asap.
It's actually not clear to me what is "compliant" in this case. For one thing, is reporting the name of a shell script that starts Squeak compliant for system attribute 0? (I would think so) For another, is the name of the VM part of the "negative set" of attributes (yes for Windows but mostly for historical reasons since system attribute 0 used to be reported in 8.3 naming style)?
- Some Socket fixes: David Reed made some changes for Sockets which
turn out to be critical in particular on Macs and Unix systems. I'm still in the process of bundling up these changes but they are fairly straightforward so I don't see many issues with them.
Submit them in the usual manner and I'll look at them sometime.
Will do, thanks.
As far as I'm concerned the 64 bit stuff is pathetically sulking in a corner due to almost nobody having paid it any attention. I have done everything reasoanble for vmmaker but there are a largish number of places where there are 'flag: #dan' type comments and there are places where there are comments about some change needing work someday. The only VMs I know of that have been built and released with the 64bit changes incorporated have been on RISC OS and OSX so to the best of my knowledge there is no information on whether they are any use on windoze or *nix. I was considering backing out all the changes since it seems that nobody cares even fractionally enough to finish what they started.
Well, as always, I guess the lack of an actual user base is problematic here. And my main problem is that I don't know much about the actual changes that happened (I know them on the large scale, yes, but not in detail) and since these things went back and forth I am just not certain what the current status of them is.
On the other hand, if John has been using those changes for OSX VMs and not gotten a whole lot of horrible complaints I think I'm willing to take that risk.
Cheers, - Andreas
On 16-Mar-06, at 5:59 PM, Andreas Raab wrote:
Questions:
- On Mac and Unix, do we implement getSystemAttribute: -1 to
return the full path to the executable?
See http://minnow.cc.gatech.edu/squeak/314 This is a spec that was agreed to about 8 years ago. If any platform is non-compliant it should be brought into compliance asap.
It's actually not clear to me what is "compliant" in this case. For one thing, is reporting the name of a shell script that starts Squeak compliant for system attribute 0? (I would think so) For another, is the name of the VM part of the "negative set" of attributes (yes for Windows but mostly for historical reasons since system attribute 0 used to be reported in 8.3 naming style)?
I'm changing the mac vm to return the full vm executable path name for system attribute 0 as of 3.8.11b3
Well, as always, I guess the lack of an actual user base is problematic here. And my main problem is that I don't know much about the actual changes that happened (I know them on the large scale, yes, but not in detail) and since these things went back and forth I am just not certain what the current status of them is.
On the other hand, if John has been using those changes for OSX VMs and not gotten a whole lot of horrible complaints I think I'm willing to take that risk.
No complaints.
Oh and btw if you use sqGnu.h and think that the register overrides for the interp method actually improves things in gcc 4.0, then you should think again.... Also do some benchmarking and cross checking, perhaps GCC 4.x is smarter now.
Cheers,
- Andreas
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On Thu, Mar 16, 2006 at 02:10:08PM -0800, Andreas Raab wrote:
Questions:
- On Mac and Unix, do we implement getSystemAttribute: -1 to return the
full path to the executable?
Did you mean system attribute 0, which corresponds to argv[0]? But no, this would not necessarily return "the" full path to the executable, nor does it necessarily return *any* full path to the executable. It's just the name that some arbitrary shell program used to locate an executable file after having consulted with the $PATH and following various symbolic links and so forth. Once the executable file was located, the shell did a fork() and exec(), after which all was forgotten. As far as I know there is no reliable way to determine the actual identity of the file system artifact from which the VM was loaded.
- On Mac and Unix, can one read the executable while running Squeak? I
need this to checksum the VM proper.
Yes, you can do what the shell would have done with argv[0], and you can find the executable file that most probably might have been used to provide the bytes that got loaded to produce the VM that is currently running, and you can read that file once you have located it. But this sounds like a rather shakey foundation for a security system.
By way of illustrating the general awfulness of this approach, the following works on my system to get a file stream on the image file:
(FileStream readOnlyFileNamed: (OSProcess accessor realpath: (ShellSyntax new findExecutablePathFor: (Smalltalk getSystemAttribute: 0) inDirectoryPath: nil)))
Seriously, this seems likely to attract crackers who might be entertained by the challenge of breaking the "security", and it is almost certain to be a source of annoyance to ordinary folks who will not understand why their system does not work after it was installed on whatever whacky network setup has been provided in their school system.
Besides the above three, I have one additional question: What is the general status of the 64bit work right now? Do we consider this complete or work in progress or experimental? Can we rule out any ill effects for the 32bit VMs?
In my experience, the 64 bit changes have no negative impact for a 32 bit VM. In general, they represent a tidying up of the system, and are all for the good. There is a a small amount of added complexity for address decoding, but it is quite localized in the source code. The compiler warnings resulting from the 64 bit changes tend to be either harmless, or to identify things that really do need to be cleaned up.
Clearly the 64 bit changes are a work in progress, and I am not in a position to say if they will be "finished". But the work to date is fundamentally sound, and I cannot think of any reason that it should not move forward. FWIW, I'll help where I can.
Dave
Hi Dave -
Did you mean system attribute 0, which corresponds to argv[0]? But no, this would not necessarily return "the" full path to the executable, nor does it necessarily return *any* full path to the executable. It's just the name that some arbitrary shell program used to locate an executable file after having consulted with the $PATH and following various symbolic links and so forth. Once the executable file was located, the shell did a fork() and exec(), after which all was forgotten. As far as I know there is no reliable way to determine the actual identity of the file system artifact from which the VM was loaded.
That's too bad, but I guess it cannot be helped. Windows has the ability to tell you the name of the binary that it loaded (be that an .exe or a .dll) and that's what I was really hoping for.
- On Mac and Unix, can one read the executable while running Squeak? I
need this to checksum the VM proper.
Yes, you can do what the shell would have done with argv[0], and you can find the executable file that most probably might have been used to provide the bytes that got loaded to produce the VM that is currently running, and you can read that file once you have located it. But this sounds like a rather shakey foundation for a security system.
It's not for security. It's really so that we know whether a VM has ever been put to the test or not. In other words when the image starts, it would check whether it has been started with an "unknown" VM and if so, runs its VM tests and reports to the user if there are any issues. The thing is that some of these tests take some time and rather than running them anew upon each start of the image I'd like to be able to checksum the VM and run them only when we detect that the VM has changed.
Cheers, - Andreas
On 3/17/06, Andreas Raab andreas.raab@gmx.de wrote:
The thing is that some of these tests take some time and rather than running them anew upon each start of the image I'd like to be able to checksum the VM and run them only when we detect that the VM has changed.
Apart from the problems with reproducing the shell path lookup (doable and there's bound to float code for that on the Net), the real killer here, as was remarked, is that admins may mark the executable as execute only. Without the read bit set on the file, you can't do the checksumming. It smells like a hack, too - it'd be much cleaner just to have a UUID generated on each build, that would *exactly* represent what you are looking for and would be relatively simple to implement on all platforms (except for wxSqueak, probably, it using MSVC to be built but that's something that needs to be solved as well :-))
Am 17.03.2006 um 05:35 schrieb Andreas Raab:
Hi Dave -
Did you mean system attribute 0, which corresponds to argv[0]? But no, this would not necessarily return "the" full path to the executable, nor does it necessarily return *any* full path to the executable. It's just the name that some arbitrary shell program used to locate an executable file after having consulted with the $PATH and following various symbolic links and so forth. Once the executable file was located, the shell did a fork() and exec(), after which all was forgotten. As far as I know there is no reliable way to determine the actual identity of the file system artifact from which the VM was loaded.
That's too bad, but I guess it cannot be helped. Windows has the ability to tell you the name of the binary that it loaded (be that an .exe or a .dll) and that's what I was really hoping for.
Well, under most circumstances it *will* be the path to the executable. Like, if you run Croquet using the shell script that is in the Croquet distribution, which we could adapt to provide the whole path (or maybe it already does, haven't actually looked into it for a while).
- On Mac and Unix, can one read the executable while running
Squeak? I need this to checksum the VM proper.
Yes, you can do what the shell would have done with argv[0], and you can find the executable file that most probably might have been used to provide the bytes that got loaded to produce the VM that is currently running, and you can read that file once you have located it. But this sounds like a rather shakey foundation for a security system.
It's not for security. It's really so that we know whether a VM has ever been put to the test or not. In other words when the image starts, it would check whether it has been started with an "unknown" VM and if so, runs its VM tests and reports to the user if there are any issues. The thing is that some of these tests take some time and rather than running them anew upon each start of the image I'd like to be able to checksum the VM and run them only when we detect that the VM has changed.
Right. So if a user chooses some non-canonical way to run the VM that screws up system attribute 0, he just will have to wait longer.
OTOH, if you're building a new VM anyways using the new math lib, what's the problem with adding a primitive for checking? There are several ways in which the checksumming by the image could fail, or be slow. Like, the VM might be executable but not readable. Or it might sit on a slow network drive. And, are you sure check-summing the VM is sufficient, and you do not need to verify some external modules, too? There are people having all modules external.
- Bert -
On Fri, Mar 17, 2006 at 11:07:31AM +0100, Bert Freudenberg wrote:
Am 17.03.2006 um 05:35 schrieb Andreas Raab:
It's not for security. It's really so that we know whether a VM has ever been put to the test or not. In other words when the image starts, it would check whether it has been started with an "unknown" VM and if so, runs its VM tests and reports to the user if there are any issues. The thing is that some of these tests take some time and rather than running them anew upon each start of the image I'd like to be able to checksum the VM and run them only when we detect that the VM has changed.
Right. So if a user chooses some non-canonical way to run the VM that screws up system attribute 0, he just will have to wait longer.
OTOH, if you're building a new VM anyways using the new math lib, what's the problem with adding a primitive for checking? There are
Yes, having the VM identify itself through a primitive sounds like a simpler and more reliable approach.
Dave
Besides the above three, I have one additional question: What is the general status of the 64bit work right now? Do we consider this complete or work in progress or experimental? Can we rule out any ill effects for the 32bit VMs?
I extent the question:
- Do the 64bit changes solve the problems with more than 2G of RAM?
Cheers,
-- Diego
vm-dev@lists.squeakfoundation.org
-
Andreas Raab -
Bert Freudenberg -
Cees De Groot -
David T. Lewis -
Diego Gomez Deck -
John M McIntosh -
tim Rowledge