Dear fellow Squeakers,
I continued my quest to create a current clean 4.6 trunk imge and still run into problems.
I am on OS X 10.9.4. I use Eliot's latest Cog.app and the 4.5 release image (Squeak4.5-13680.image) from the FTP server. See here [1] for the bash commands I use to set up my starting point.
Then I execute the following script in a workspace:
MCMcmUpdater defaultUpdateURL: 'http://source.squeak.org/trunk'; updateFromServer.
During processing of update-eem.287.mcm a popup menu appears saying that the Squeak4.5-13680.changes file does not exist. This is incorrect because the file exists. When I choose Debug I get FileDoesNotExistException. See the attached PNG and SqueakDebug.log.
I would be interested if others run into the same problem.
Cheers, Bernhard
[1] mkdir trunk cd trunk
curl -O http://www.mirandabanda.org/files/Cog/VM/VM.r3063/Cog.app-14.32.3063.tgz gunzip -c Cog.app-14.32.3063.tgz | tar xopf - && rm Cog.app-14.32.3063.tgz
curl -O ftp://ftp.squeak.org/4.5/SqueakV41.sources.zip unzip SqueakV41.sources.zip && rm SqueakV41.sources.zip
curl -O ftp://ftp.squeak.org/4.5/Squeak4.5-13680.zip unzip Squeak4.5-13680.zip && rm Squeak4.5-13680.zip
./Cog.app/Contents/MacOS/Squeak Squeak4.5-13680.image
2014-08-30 15:31 GMT+02:00 Bernhard Pieber bernhard@pieber.com:
Dear fellow Squeakers,
I continued my quest to create a current clean 4.6 trunk imge and still run into problems.
I am on OS X 10.9.4. I use Eliot's latest Cog.app and the 4.5 release image (Squeak4.5-13680.image) from the FTP server. See here [1] for the bash commands I use to set up my starting point.
Then I execute the following script in a workspace:
MCMcmUpdater defaultUpdateURL: 'http://source.squeak.org/trunk'; updateFromServer.
During processing of update-eem.287.mcm a popup menu appears saying that the Squeak4.5-13680.changes file does not exist. This is incorrect because the file exists. When I choose Debug I get FileDoesNotExistException. See the attached PNG and SqueakDebug.log.
I would be interested if others run into the same problem.
Cheers, Bernhard
It happens to me from time to time. I suspect that there are two many files opened before the reclamation facility has a chance to work. I suspect the read-only copy of change file to open those files.
I you open the debugger and restart from the right place (?), the update correctly resumes (but it's not very automated...).
[1] mkdir trunk cd trunk
curl -O http://www.mirandabanda.org/files/Cog/VM/VM.r3063/Cog.app-14.32.3063.tgz gunzip -c Cog.app-14.32.3063.tgz | tar xopf - && rm Cog.app-14.32.3063.tgz
curl -O ftp://ftp.squeak.org/4.5/SqueakV41.sources.zip unzip SqueakV41.sources.zip && rm SqueakV41.sources.zip
curl -O ftp://ftp.squeak.org/4.5/Squeak4.5-13680.zip unzip Squeak4.5-13680.zip && rm Squeak4.5-13680.zip
./Cog.app/Contents/MacOS/Squeak Squeak4.5-13680.image
Hi Nicolas,
Thanks for your answer! See below.
Am 31.08.2014 um 09:50 schrieb Nicolas Cellier nicolas.cellier.aka.nice@gmail.com:
2014-08-30 15:31 GMT+02:00 Bernhard Pieber bernhard@pieber.com:
Dear fellow Squeakers,
I continued my quest to create a current clean 4.6 trunk imge and still run into problems.
I am on OS X 10.9.4. I use Eliot's latest Cog.app and the 4.5 release image (Squeak4.5-13680.image) from the FTP server. See here [1] for the bash commands I use to set up my starting point.
Then I execute the following script in a workspace:
MCMcmUpdater defaultUpdateURL: 'http://source.squeak.org/trunk'; updateFromServer.
During processing of update-eem.287.mcm a popup menu appears saying that the Squeak4.5-13680.changes file does not exist. This is incorrect because the file exists. When I choose Debug I get FileDoesNotExistException. See the attached PNG and SqueakDebug.log.
I would be interested if others run into the same problem.
It happens to me from time to time.
It's a consolation to know that I am not the only one seeing this. With the process I follow I have it every time, i.e. I can reliably reproduce it.
I suspect that there are two many files opened before the reclamation facility has a chance to work. I suspect the read-only copy of change file to open those files.
I am not sure I understand your last sentence. Could you please elaborate?
I you open the debugger and restart from the right place (?), the update correctly resumes (but it's not very automated...).
Right. Even if I don't debug but just choose the existing changes file in the popup menu the trunk update resumes and finishes.
However, there are several other problems: 1. The Environments package is dirty and has two ancestors (nice.47, cmm.51). 2. There are many corrupt files in the package-cache. 3. I get a strange variant of the "Error: Bits size mismatch" in CompiledMethodTrailer>>#encodeUsingZip when I execute the following script to find out the corrupt Monticello files from a Squeak workspace:
MCCacheRepository default allFileNames select: [:each | [MCCacheRepository default versionInfoFromFileNamed: each. false] on: Error do: [true]].
See the attached SqueakDebug.log for details. Again, I would be very interested if others can reproduce that error.
All this makes me wonder if I can safely use that image?
Cheers, Bernhard
Hi Bernard,
Thanks for your answer! See below.
Am 31.08.2014 um 09:50 schrieb Nicolas Cellier nicolas.cellier.aka.nice@gmail.com:
2014-08-30 15:31 GMT+02:00 Bernhard Pieber bernhard@pieber.com:
Dear fellow Squeakers,
I continued my quest to create a current clean 4.6 trunk imge and still run into problems.
I am on OS X 10.9.4. I use Eliot's latest Cog.app and the 4.5 release image (Squeak4.5-13680.image) from the FTP server. See here [1] for the bash commands I use to set up my starting point.
Then I execute the following script in a workspace:
MCMcmUpdater defaultUpdateURL: 'http://source.squeak.org/trunk'; updateFromServer.
During processing of update-eem.287.mcm a popup menu appears saying that the Squeak4.5-13680.changes file does not exist. This is incorrect because the file exists. When I choose Debug I get FileDoesNotExistException. See the attached PNG and SqueakDebug.log.
I would be interested if others run into the same problem.
It happens to me from time to time.
It's a consolation to know that I am not the only one seeing this. With the process I follow I have it every time, i.e. I can reliably reproduce it.
This is really strange Bernnard. But one thing to remember is that the 13680 image has a corruption. Search the mailing list for subject: "(Environment named: #Smalltalk) trap in 4.5." That thread explains the corruption related to having String-key's in an IdentityDictionary..
Could you try updating your 13680 image from the 'squeak45' repository FIRST, to ensure you get my fix for that BEFORE you change to trunk. This will take you to 13687. THEN you can change your updateUrl to trunk and update from there. Does that help on your end at all?
I just tried it both ways but didn't experience the issues you did whether I updated from squeak45 first or not. No dirty packages, and your Error catch select: produced no errors but returned an empty collection for me.
But I'm going from Linux and using the 2776 VM which ships with 4.5. I think that is the most reliable VM right now. I'm curious if you reproduce my steps above whether it reproduces my success?
Let's not give up on this! It sounds like there's a problem lurking, possibly only in Mac..??
I suspect that there are two many files opened before the reclamation facility has a chance to work. I suspect the read-only copy of change file to open those files.
I am not sure I understand your last sentence. Could you please elaborate?
I you open the debugger and restart from the right place (?), the update correctly resumes (but it's not very automated...).
Right. Even if I don't debug but just choose the existing changes file in the popup menu the trunk update resumes and finishes.
However, there are several other problems:
- The Environments package is dirty and has two ancestors (nice.47, cmm.51).
- There are many corrupt files in the package-cache.
- I get a strange variant of the "Error: Bits size mismatch" in CompiledMethodTrailer>>#encodeUsingZip when I execute the following script to find out the corrupt Monticello files from a Squeak workspace:
MCCacheRepository default allFileNames select: [:each | [MCCacheRepository default versionInfoFromFileNamed: each. false] on: Error do: [true]].
See the attached SqueakDebug.log for details. Again, I would be very interested if others can reproduce that error.
All this makes me wonder if I can safely use that image?
Cheers, Bernhard
Hi Chris,
On Aug 31, 2014, at 3:48 PM, Chris Muller asqueaker@gmail.com wrote:
Hi Bernard,
Thanks for your answer! See below.
Am 31.08.2014 um 09:50 schrieb Nicolas Cellier nicolas.cellier.aka.nice@gmail.com:
2014-08-30 15:31 GMT+02:00 Bernhard Pieber bernhard@pieber.com:
Dear fellow Squeakers,
I continued my quest to create a current clean 4.6 trunk imge and still run into problems.
I am on OS X 10.9.4. I use Eliot's latest Cog.app and the 4.5 release image (Squeak4.5-13680.image) from the FTP server. See here [1] for the bash commands I use to set up my starting point.
Then I execute the following script in a workspace:
MCMcmUpdater defaultUpdateURL: 'http://source.squeak.org/trunk'; updateFromServer.
During processing of update-eem.287.mcm a popup menu appears saying that the Squeak4.5-13680.changes file does not exist. This is incorrect because the file exists. When I choose Debug I get FileDoesNotExistException. See the attached PNG and SqueakDebug.log.
I would be interested if others run into the same problem.
It happens to me from time to time.
It's a consolation to know that I am not the only one seeing this. With the process I follow I have it every time, i.e. I can reliably reproduce it.
This is really strange Bernnard. But one thing to remember is that the 13680 image has a corruption. Search the mailing list for subject: "(Environment named: #Smalltalk) trap in 4.5." That thread explains the corruption related to having String-key's in an IdentityDictionary..
Could you try updating your 13680 image from the 'squeak45' repository FIRST, to ensure you get my fix for that BEFORE you change to trunk. This will take you to 13687. THEN you can change your updateUrl to trunk and update from there. Does that help on your end at all?
I just tried it both ways but didn't experience the issues you did whether I updated from squeak45 first or not. No dirty packages, and your Error catch select: produced no errors but returned an empty collection for me.
But I'm going from Linux and using the 2776 VM which ships with 4.5. I think that is the most reliable VM right now. I'm curious if you reproduce my steps above whether it reproduces my success?
Please NO, NO and thrice NO. Each VM I post on my site is posted because it fixes some bug or achieves some performance improvement or adds functionality. 2776 is old. It contains bugs fixed by subsequent releases. Unless you have regression treats showing failures with later VMs you should use the latest VM available.
Note that there is significant regression testing of Newspeak VMs at Cadence.
Releasing old VMs just makes everyone's life harder. I have to explain that it's not the latest, and they have to upload it. So /please/ use the latest VM available.
Obviously Spur VMs are buggy but they're young and hence less stable, but that's not the case for the Cog VMs.
Let's not give up on this! It sounds like there's a problem lurking, possibly only in Mac..??
I suspect that there are two many files opened before the reclamation facility has a chance to work. I suspect the read-only copy of change file to open those files.
I am not sure I understand your last sentence. Could you please elaborate?
I you open the debugger and restart from the right place (?), the update correctly resumes (but it's not very automated...).
Right. Even if I don't debug but just choose the existing changes file in the popup menu the trunk update resumes and finishes.
However, there are several other problems:
- The Environments package is dirty and has two ancestors (nice.47, cmm.51).
- There are many corrupt files in the package-cache.
- I get a strange variant of the "Error: Bits size mismatch" in CompiledMethodTrailer>>#encodeUsingZip when I execute the following script to find out the corrupt Monticello files from a Squeak workspace:
MCCacheRepository default allFileNames select: [:each | [MCCacheRepository default versionInfoFromFileNamed: each. false] on: Error do: [true]].
See the attached SqueakDebug.log for details. Again, I would be very interested if others can reproduce that error.
All this makes me wonder if I can safely use that image?
Cheers, Bernhard
Hi Bernard,
Thanks for your answer! See below.
Am 31.08.2014 um 09:50 schrieb Nicolas Cellier nicolas.cellier.aka.nice@gmail.com:
2014-08-30 15:31 GMT+02:00 Bernhard Pieber bernhard@pieber.com:
Dear fellow Squeakers,
I continued my quest to create a current clean 4.6 trunk imge and still run into problems.
I am on OS X 10.9.4. I use Eliot's latest Cog.app and the 4.5 release image (Squeak4.5-13680.image) from the FTP server. See here [1] for the bash commands I use to set up my starting point.
Then I execute the following script in a workspace:
MCMcmUpdater defaultUpdateURL: 'http://source.squeak.org/trunk'; updateFromServer.
During processing of update-eem.287.mcm a popup menu appears saying that the Squeak4.5-13680.changes file does not exist. This is incorrect because the file exists. When I choose Debug I get FileDoesNotExistException. See the attached PNG and SqueakDebug.log.
I would be interested if others run into the same problem.
It happens to me from time to time.
It's a consolation to know that I am not the only one seeing this. With the process I follow I have it every time, i.e. I can reliably reproduce it.
This is really strange Bernnard. But one thing to remember is that the 13680 image has a corruption. Search the mailing list for subject: "(Environment named: #Smalltalk) trap in 4.5." That thread explains the corruption related to having String-key's in an IdentityDictionary..
Could you try updating your 13680 image from the 'squeak45' repository FIRST, to ensure you get my fix for that BEFORE you change to trunk. This will take you to 13687. THEN you can change your updateUrl to trunk and update from there. Does that help on your end at all?
I just tried it both ways but didn't experience the issues you did whether I updated from squeak45 first or not. No dirty packages, and your Error catch select: produced no errors but returned an empty collection for me.
But I'm going from Linux and using the 2776 VM which ships with 4.5. I think that is the most reliable VM right now. I'm curious if you reproduce my steps above whether it reproduces my success?
Please NO, NO and thrice NO. Each VM I post on my site is posted because it fixes some bug or achieves some performance improvement or adds functionality. 2776 is old. It contains bugs fixed by subsequent releases. Unless you have regression treats showing failures with later VMs you should use the latest VM available.
Sheesh Eliot. I assume your outburst is by my statement that "I think 2776 is the most reliable right now". If so, fine, I stand corrected but 1) I said "think" which implies a degree of uncertainty, and 2) we're not talking about Cog right now, we're simply trying to establish consisten reproducibility of Bernard's issue, of which the VM is a major component.
In this case, the failure occurred when Bernard used the latest VM on Mac, but didn't when I used 2776 on Linux.
I also wanted to test it with the actual VM that was released WITH Squeak-4.5, so I can assess whether the problem Bernard is having could potentially affect new Squeak users using the All-In-One. Is that okay?
Note that there is significant regression testing of Newspeak VMs at Cadence.
Okay, so the difference in reproducibility is probably not the VM, still, I hope you won't mind if I MENTION what VM I used. I'll try to be careful in the future about making any statements about relative quality to newer versions.
Releasing old VMs just makes everyone's life harder. I have to explain that it's not the latest, and they have to upload it. So /please/ use the latest VM available.
Who is releasing old VM's? Not I.
As for using the latest, please understand I have a lot on my plate right now. Like when a new motherboard BIOS is released, I don't immediately shut everything down to update it unless there's a good reason to. I can always go forward if problems develop, but going backward is not easy.
Cheers, Chris
Hi Chris,
my message was in no way an outburst. Merely a plea that folks not release old VMs. I misread your paragraph about the 2776 VM, and saw it as a proposal to release using the 2776 VM. Apologies. I'm still jet lagged...
On Mon, Sep 1, 2014 at 12:04 PM, Chris Muller ma.chris.m@gmail.com wrote:
Hi Bernard,
Thanks for your answer! See below.
Am 31.08.2014 um 09:50 schrieb Nicolas Cellier <
nicolas.cellier.aka.nice@gmail.com>:
2014-08-30 15:31 GMT+02:00 Bernhard Pieber bernhard@pieber.com:
Dear fellow Squeakers,
I continued my quest to create a current clean 4.6 trunk imge and
still run into problems.
I am on OS X 10.9.4. I use Eliot's latest Cog.app and the 4.5
release image (Squeak4.5-13680.image) from the FTP server. See here [1] for the bash commands I use to set up my starting point.
Then I execute the following script in a workspace:
MCMcmUpdater defaultUpdateURL: 'http://source.squeak.org/trunk'; updateFromServer.
During processing of update-eem.287.mcm a popup menu appears saying
that the Squeak4.5-13680.changes file does not exist. This is incorrect because the file exists. When I choose Debug I get FileDoesNotExistException. See the attached PNG and SqueakDebug.log.
I would be interested if others run into the same problem.
It happens to me from time to time.
It's a consolation to know that I am not the only one seeing this.
With the process I follow I have it every time, i.e. I can reliably reproduce it.
This is really strange Bernnard. But one thing to remember is that the 13680 image has a corruption. Search the mailing list for subject: "(Environment named: #Smalltalk) trap in 4.5." That thread explains the corruption related to having String-key's in an IdentityDictionary..
Could you try updating your 13680 image from the 'squeak45' repository FIRST, to ensure you get my fix for that BEFORE you change to trunk. This will take you to 13687. THEN you can change your updateUrl to trunk and update from there. Does that help on your end at all?
I just tried it both ways but didn't experience the issues you did whether I updated from squeak45 first or not. No dirty packages, and your Error catch select: produced no errors but returned an empty collection for me.
But I'm going from Linux and using the 2776 VM which ships with 4.5. I think that is the most reliable VM right now. I'm curious if you reproduce my steps above whether it reproduces my success?
Please NO, NO and thrice NO. Each VM I post on my site is posted
because it fixes some bug or achieves some performance improvement or adds functionality. 2776 is old. It contains bugs fixed by subsequent releases. Unless you have regression treats showing failures with later VMs you should use the latest VM available.
Sheesh Eliot. I assume your outburst is by my statement that "I think 2776 is the most reliable right now". If so, fine, I stand corrected but 1) I said "think" which implies a degree of uncertainty, and 2) we're not talking about Cog right now, we're simply trying to establish consisten reproducibility of Bernard's issue, of which the VM is a major component.
In this case, the failure occurred when Bernard used the latest VM on Mac, but didn't when I used 2776 on Linux.
I also wanted to test it with the actual VM that was released WITH Squeak-4.5, so I can assess whether the problem Bernard is having could potentially affect new Squeak users using the All-In-One. Is that okay?
Note that there is significant regression testing of Newspeak VMs at
Cadence.
Okay, so the difference in reproducibility is probably not the VM, still, I hope you won't mind if I MENTION what VM I used. I'll try to be careful in the future about making any statements about relative quality to newer versions.
Releasing old VMs just makes everyone's life harder. I have to explain
that it's not the latest, and they have to upload it. So /please/ use the latest VM available.
Who is releasing old VM's? Not I.
As for using the latest, please understand I have a lot on my plate right now. Like when a new motherboard BIOS is released, I don't immediately shut everything down to update it unless there's a good reason to. I can always go forward if problems develop, but going backward is not easy.
Cheers, Chris
On 31.08.2014, at 09:50, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com wrote:
2014-08-30 15:31 GMT+02:00 Bernhard Pieber bernhard@pieber.com:
During processing of update-eem.287.mcm a popup menu appears saying that the Squeak4.5-13680.changes file does not exist. This is incorrect because the file exists. When I choose Debug I get FileDoesNotExistException. See the attached PNG and SqueakDebug.log.
It happens to me from time to time. I suspect that there are two many files opened before the reclamation facility has a chance to work. I suspect the read-only copy of change file to open those files.
Yep. I noticed that on my SqueakJS VM (which does not yet support weak refs/finalization), in a Squeak 4.5 image the sources and changes files are opened many many times but never closed (I'm refcounting the handles).
I suspect a strategically placed #close after we're done with the read-only copy would solve this problem for good.
Even with finalization support it's a good idea to actually close files when you're done, because it's unpredictable when the finalizer will actually run.
Another thought is that given the abundance of memory these days, we might cache both sources and changes in main memory (which would also speed up full-text searches).
- Bert -
Hi Bert,
On Sep 1, 2014, at 4:31 AM, Bert Freudenberg bert@freudenbergs.de wrote:
On 31.08.2014, at 09:50, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com wrote:
2014-08-30 15:31 GMT+02:00 Bernhard Pieber bernhard@pieber.com:
During processing of update-eem.287.mcm a popup menu appears saying that the Squeak4.5-13680.changes file does not exist. This is incorrect because the file exists. When I choose Debug I get FileDoesNotExistException. See the attached PNG and SqueakDebug.log.
It happens to me from time to time. I suspect that there are two many files opened before the reclamation facility has a chance to work. I suspect the read-only copy of change file to open those files.
Yep. I noticed that on my SqueakJS VM (which does not yet support weak refs/finalization), in a Squeak 4.5 image the sources and changes files are opened many many times but never closed (I'm refcounting the handles).
I suspect a strategically placed #close after we're done with the read-only copy would solve this problem for good.
Even with finalization support it's a good idea to actually close files when you're done, because it's unpredictable when the finalizer will actually run.
Another thought is that given the abundance of memory these days, we might cache both sources and changes in main memory (which would also speed up full-text searches).
Pharo is planning to eliminate them altogether which is more coherent than caching them. But IMO the solution is easy, maintain a *single* read-only copy of the sources and changes files in SourceFilesArray (or whatever the class is called; I'm on my phone) and read source through them instead of reopen ing the damn things all the time. Then the file's own buffers will provide done caching. Annoying that I write this code in 2008 for newspeak but we still rely on the mad "run the GC to finalize files when open fails" approach.
- Bert -
Eliot phone
On 01.09.2014, at 15:56, Eliot Miranda eliot.miranda@gmail.com wrote:
Another thought is that given the abundance of memory these days, we might cache both sources and changes in main memory (which would also speed up full-text searches).
Pharo is planning to eliminate them altogether which is more coherent than caching them.
That's another discussion, but it might be a step towards that.
But IMO the solution is easy, maintain a *single* read-only copy of the sources and changes files in SourceFilesArray (or whatever the class is called; I'm on my phone) and read source through them instead of reopen ing the damn things all the time.
Except that Squeak files do not maintain an independent file position pointer, so reading from different positions in the same file is not thread-safe. That's why the file is opened again.
Then the file's own buffers will provide done caching. Annoying that I write this code in 2008 for newspeak but we still rely on the mad "run the GC to finalize files when open fails" approach.
Well, you writing this for newspeak does not immediately benefit Squeak. But if you point us to the code maybe someone can port it to Squeak?
- Bert -
But IMO the solution is easy, maintain a *single* read-only copy of the sources and changes files in SourceFilesArray (or whatever the class is called; I'm on my phone) and read source through them instead of reopen ing the damn things all the time.
Except that Squeak files do not maintain an independent file position pointer, so reading from different positions in the same file is not thread-safe. That's why the file is opened again.
It's a side effect of using the somewhat ridiculous minimal C lib interface that is so totally non-thread safe it's offensive. Yes, it's simple for basic porting but the result is decidedly suboptimal in the long run.
When making the RISC OS translation layer to allow the vm to fake out the multiple openings of files I had to cache file pointers and translate and mess around with permissions and.. yeuch. Using direct calls to 'proper' file apis that take a file reference, position, etc makes life much easier. It's at least a decade past time we replaced the file interface. I'm not in a position to check anything in the image for the next week or two so I may be misremembering but didn't Colin P write something better recently? Yes, I suspect that making the transition would be interesting.
/tim {insert witticism here}
Hi Bert,
On Mon, Sep 1, 2014 at 7:13 AM, Bert Freudenberg bert@freudenbergs.de wrote:
On 01.09.2014, at 15:56, Eliot Miranda eliot.miranda@gmail.com wrote:
Another thought is that given the abundance of memory these days, we
might cache both sources and changes in main memory (which would also speed up full-text searches).
Pharo is planning to eliminate them altogether which is more coherent
than caching them.
That's another discussion, but it might be a step towards that.
But IMO the solution is easy, maintain a *single* read-only copy of the
sources and changes files in SourceFilesArray (or whatever the class is called; I'm on my phone) and read source through them instead of reopen ing the damn things all the time.
Except that Squeak files do not maintain an independent file position pointer, so reading from different positions in the same file is not thread-safe. That's why the file is opened again.
Can you explain more. I don't understand. As I see it every file instance has its own FILE structure so I don't understand how this can be so. Files are derived from FilePlugin's primitiveFileOpen function. That is implemented in terms of fileOpenNamesizewritesecure, which allocates a ByteArray to hold state (so no two Smalltalk files share state) and then uses sqFileOpen to open the underlying file and fill in the state. The C library implementation of sqFileOpen in platforms/Cross/plugins/FilePlugin/sqFilePluginBasicPrims.c uses fopen et al, again creating unique state. So I don't see how having a writable Smalltalk file and a separate read-only Smalltalk file on the sources and changes can result in other than two separate independent file pointers.
There *is* an issue with the structure of file access in the debugger. If one were to step through execution of accessing source from e.g. the read-only sources file then the very act of fetching sources for the methods being displayed would confuse the state in the file one was observing through the debugger. But that's hardly a new situation (one can get into a similar situation with the current setup), and the debugger could ease the situation by wrapping source access with something that reset the file's buffer etc.
Then the file's own buffers will provide done caching. Annoying that I
write this code in 2008 for newspeak but we still rely on the mad "run the GC to finalize files when open fails" approach.
Well, you writing this for newspeak does not immediately benefit Squeak. But if you point us to the code maybe someone can port it to Squeak?
I already did a year or two ago and it got shot down on the debugger grounds I reiterated above.
- Bert -
At least part the problem is that the fopen/fread etc API is dangerous unless well implemented at the lowest level. Intermixed uses of fpos and fread can easily mangle your data unless the lower level code really carefully tracks things. Our API providing fpos at the image level is decidedly dangerous, adding a second layer of possible problems.
/tim {insert witticism here}
Hi Tim,
On Tue, Sep 2, 2014 at 10:38 AM, tim Rowledge tim@rowledge.org wrote:
At least part the problem is that the fopen/fread etc API is dangerous unless well implemented at the lowest level. Intermixed uses of fpos and fread can easily mangle your data unless the lower level code really carefully tracks things. Our API providing fpos at the image level is decidedly dangerous, adding a second layer of possible problems.
forgive me, but this strikes me as FUD. We're talking about adding a read-only copy, not mixing many writable files. I can't see that the API presents any hazard to the proposed usage.
/tim {insert witticism here}
forgive me, but this strikes me as FUD. We're talking about adding a read-only copy, not mixing many writable files. I can't see that the API presents any hazard to the proposed usage.
It might be, but although I can't point to any concrete case right now I think we've had cases suggesting that one or more OS makes a mess of keeping the two separate. There's some ugly code in the vanilla file plugin (which might not be used code anymore, come to think of it) where wild guesses are made as to whether an fpos is being done in a way that implies a need to do some fudge to force a buffer flush, for example. I'm inclined to se that as evidence of a worrying API.
But we all know that I worry about things that don't always need to be worried about. /tim {insert witticism here}
On 02.09.2014, at 17:03, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Bert,
On Mon, Sep 1, 2014 at 7:13 AM, Bert Freudenberg bert@freudenbergs.de wrote: On 01.09.2014, at 15:56, Eliot Miranda eliot.miranda@gmail.com wrote:
Another thought is that given the abundance of memory these days, we might cache both sources and changes in main memory (which would also speed up full-text searches).
Pharo is planning to eliminate them altogether which is more coherent than caching them.
That's another discussion, but it might be a step towards that.
But IMO the solution is easy, maintain a *single* read-only copy of the sources and changes files in SourceFilesArray (or whatever the class is called; I'm on my phone) and read source through them instead of reopen ing the damn things all the time.
Except that Squeak files do not maintain an independent file position pointer, so reading from different positions in the same file is not thread-safe. That's why the file is opened again.
Can you explain more. I don't understand.
I think we're in violent agreement ;) This was a comment on the image-side file handling, not about the VM. (Although we might need better file prims. Tim appears to have ideas)
As I see it every file instance has its own FILE structure so I don't understand how this can be so. Files are derived from FilePlugin's primitiveFileOpen function. That is implemented in terms of fileOpenNamesizewritesecure, which allocates a ByteArray to hold state (so no two Smalltalk files share state) and then uses sqFileOpen to open the underlying file and fill in the state. The C library implementation of sqFileOpen in platforms/Cross/plugins/FilePlugin/sqFilePluginBasicPrims.c uses fopen et al, again creating unique state.
Precisely. Every Squeak file-open also opens an OS file via fopen(). The number of files a process can open is limited. That is why we run out of file handles unless the files gets closed properly.
So I don't see how having a writable Smalltalk file and a separate read-only Smalltalk file on the sources and changes can result in other than two separate independent file pointers.
What I was getting at is that it would be much better to actually open the file only once (or twice, once for writing and once for read-only) and then maintain a file pointer in the image independently of the OS's file position. That is how we could share a single read-only file for many readers. The problem is that the OS file also has a file position, and the file positions we maintain in the image would easily get out of sync with that.
There *is* an issue with the structure of file access in the debugger. If one were to step through execution of accessing source from e.g. the read-only sources file then the very act of fetching sources for the methods being displayed would confuse the state in the file one was observing through the debugger. But that's hardly a new situation (one can get into a similar situation with the current setup), and the debugger could ease the situation by wrapping source access with something that reset the file's buffer etc.
Then the file's own buffers will provide done caching. Annoying that I write this code in 2008 for newspeak but we still rely on the mad "run the GC to finalize files when open fails" approach.
Well, you writing this for newspeak does not immediately benefit Squeak. But if you point us to the code maybe someone can port it to Squeak?
I already did a year or two ago and it got shot down on the debugger grounds I reiterated above.
Well, maybe it's time to revisit, then.
- Bert -
Hi Bert,
On Tue, Sep 2, 2014 at 11:23 AM, Bert Freudenberg bert@freudenbergs.de wrote:
On 02.09.2014, at 17:03, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Bert,
On Mon, Sep 1, 2014 at 7:13 AM, Bert Freudenberg bert@freudenbergs.de wrote:
On 01.09.2014, at 15:56, Eliot Miranda eliot.miranda@gmail.com wrote:
Another thought is that given the abundance of memory these days, we
might cache both sources and changes in main memory (which would also speed up full-text searches).
Pharo is planning to eliminate them altogether which is more coherent
than caching them.
That's another discussion, but it might be a step towards that.
But IMO the solution is easy, maintain a *single* read-only copy of
the sources and changes files in SourceFilesArray (or whatever the class is called; I'm on my phone) and read source through them instead of reopen ing the damn things all the time.
Except that Squeak files do not maintain an independent file position pointer, so reading from different positions in the same file is not thread-safe. That's why the file is opened again.
Can you explain more. I don't understand.
I think we're in violent agreement ;) This was a comment on the image-side file handling, not about the VM. (Although we might need better file prims. Tim appears to have ideas)
As I see it every file instance has its own FILE structure so I don't understand how this can be so. Files are derived from FilePlugin's primitiveFileOpen function. That is implemented in terms of fileOpenNamesizewritesecure, which allocates a ByteArray to hold state (so no two Smalltalk files share state) and then uses sqFileOpen to open the underlying file and fill in the state. The C library implementation of sqFileOpen in platforms/Cross/plugins/FilePlugin/sqFilePluginBasicPrims.c uses fopen et al, again creating unique state.
Precisely. Every Squeak file-open also opens an OS file via fopen(). The number of files a process can open is limited. That is why we run out of file handles unless the files gets closed properly.
So I don't see how having a writable Smalltalk file and a separate read-only Smalltalk file on the sources and changes can result in other than two separate independent file pointers.
What I was getting at is that it would be much better to actually open the file only once (or twice, once for writing and once for read-only) and then maintain a file pointer in the image independently of the OS's file position. That is how we could share a single read-only file for many readers.
Right, now we are in agreement. That's what I like. 1 read-only copy, one writeable copy for the changes file, and presumably a single read-only file for the sources file.
The problem is that the OS file also has a file position, and the file positions we maintain in the image would easily get out of sync with that.
Potentially, but don't do that. If the image wants to update the file position it must also use primSetPosition:to: to keep the OS file pointer up-to-date. There's a thread-safety issue, but that's to be kept orthogonal for now (the current situation also has such issues).
There *is* an issue with the structure of file access in the debugger. If
one were to step through execution of accessing source from e.g. the read-only sources file then the very act of fetching sources for the methods being displayed would confuse the state in the file one was observing through the debugger. But that's hardly a new situation (one can get into a similar situation with the current setup), and the debugger could ease the situation by wrapping source access with something that reset the file's buffer etc.
Then the file's own buffers will provide done caching. Annoying that I
write this code in 2008 for newspeak but we still rely on the mad "run the GC to finalize files when open fails" approach.
Well, you writing this for newspeak does not immediately benefit Squeak. But if you point us to the code maybe someone can port it to Squeak?
I already did a year or two ago and it got shot down on the debugger grounds I reiterated above.
Well, maybe it's time to revisit, then.
Will do.
- Bert -
squeak-dev@lists.squeakfoundation.org