As Dave mentioned, the immediate priority is simply ensuring continued
operation of squeaksource.com. He is heading up the transition to a new server this week whether that's box3, box4 (Gandi) or a server hosted by Robert's group. If we cannot get it running on a new image and/or VM immediately, we'll just have to deal with the crashes/lockups until we can.
While I think Squeak should host its own services, unfortunately what we
have right now is unacceptably dangerous. It appears someone has hacked our SqueakSource implementation to bypass its original design for persistence to use "image-based" persistence. Image-based persistence, in itself, would be fine, except for the fact that our Linux-based SS system scripts carelessly KILL the SS process by PID as a means of opening the image under VNC -- potentially when the image could be in a half-saved state!
Which SqS are you referring to? Fabrizio once send Andreas and Me a (then current) image of the squeaksource.com (SS.C) and IIRC, it just had VNC running, so you probably are referring to source.squeak.org (S.S.O)?
Sorry, I also do not understand what you mean by the original design
being “hacked” for image-based persistence.
Yes, I was referring to SSO, which is running a slightly newer/forked version of SqueakSource and using the save-the-image persistence.
SSO is also running on an old version of Squeak and old VM. Whatever is good for the SS upgrade will be good for SSO and vice-versa.
Such a misfortune could wipe out the entire community if we were unable to recover from a backup. Therefore, I fully support your idea to move to SqueakSource3 running on GemStone.
But there is one more option. Last month, Tim Rowledge expressed
interest in having a more-capable SqueakSource that could support what Magma-based MC repositories provide today:
http://lists.squeakfoundation.org/pipermail/squeak-dev/2013-July/172243.html
The advantage to this approach, besides having a working server example
based on Squeak, is that all MCDefinitions across all Versions of each SSProject would be canonicalized and indexed inside the repository, so that the 'browse mc versions' and 'browse mc origin' functions which are afforded by Magma-based MCRepository's would be available on the trunk repository for everyone.
I had the Idea (but not yet enough pressure) to provide similar features for SS3.
McModel is self-contained. Besides substituting an RcDictionary for the MagmaPreallocatedDictionary, you might be able to lift it straight from Magma. It's indexing is sufficient to provide the minimum MCRepository subclass-responsibility API, plus the aforementioned (#historyOf: aMCDefinition) and (#originOf: aMCDefinition) functions, all without MC even knowing it exists.
I've already begun the work for this, but if there is no interest in
this, I will simply bow out and support the idea that we go with the GemStone solution.
SS3 already cracks open MCZs for determining diffs, browsing code, finding parents, reading version informations. It would be easy to just cache this information and provide links inbetween. Remember, however, that both, a Magma-Based solution or a SS3-feature would have to respect acls in the sense, that you shouldn't be able to browse into a non-readable project by means of 'browse mc origin'.
Yes, I'd planned for each SSProject to have its own McModel instance, so access is allowed only through the existing Project level ACL.
We have ODBMS' we should use them to exceed what we had before with only the filesystem.
- Chris
PS -- I attached McModel.st.gz in case you're interested.
Am 26.08.2013 um 21:59 schrieb Chris Muller ma.chris.m@gmail.com:
As Dave mentioned, the immediate priority is simply ensuring continued
operation of squeaksource.com. He is heading up the transition to a new server this week whether that's box3, box4 (Gandi) or a server hosted by Robert's group. If we cannot get it running on a new image and/or VM immediately, we'll just have to deal with the crashes/lockups until we can.
While I think Squeak should host its own services, unfortunately what we
have right now is unacceptably dangerous. It appears someone has hacked our SqueakSource implementation to bypass its original design for persistence to use "image-based" persistence. Image-based persistence, in itself, would be fine, except for the fact that our Linux-based SS system scripts carelessly KILL the SS process by PID as a means of opening the image under VNC -- potentially when the image could be in a half-saved state!
Which SqS are you referring to? Fabrizio once send Andreas and Me a (then current) image of the squeaksource.com (SS.C) and IIRC, it just had VNC running, so you probably are referring to source.squeak.org (S.S.O)?
Sorry, I also do not understand what you mean by the original design
being “hacked” for image-based persistence.
Yes, I was referring to SSO, which is running a slightly newer/forked version of SqueakSource and using the save-the-image persistence.
What do you exactly mean by save-the-image-persistence? I ask, as I carefully hand-merged/back-ported/cross-ported features from SSO _and_ SSC to SS3 prior to its alpha release.
SSO is also running on an old version of Squeak and old VM. Whatever is good for the SS upgrade will be good for SSO and vice-versa.
Such a misfortune could wipe out the entire community if we were unable to recover from a backup. Therefore, I fully support your idea to move to SqueakSource3 running on GemStone.
But there is one more option. Last month, Tim Rowledge expressed
interest in having a more-capable SqueakSource that could support what Magma-based MC repositories provide today:
http://lists.squeakfoundation.org/pipermail/squeak-dev/2013-July/172243.html
The advantage to this approach, besides having a working server example
based on Squeak, is that all MCDefinitions across all Versions of each SSProject would be canonicalized and indexed inside the repository, so that the 'browse mc versions' and 'browse mc origin' functions which are afforded by Magma-based MCRepository's would be available on the trunk repository for everyone.
I had the Idea (but not yet enough pressure) to provide similar features for SS3.
McModel is self-contained. Besides substituting an RcDictionary for the MagmaPreallocatedDictionary, you might be able to lift it straight from Magma. It's indexing is sufficient to provide the minimum MCRepository subclass-responsibility API, plus the aforementioned (#historyOf: aMCDefinition) and (#originOf: aMCDefinition) functions, all without MC even knowing it exists.
I will have a look, thanks for sharing.
I've already begun the work for this, but if there is no interest in
this, I will simply bow out and support the idea that we go with the GemStone solution.
SS3 already cracks open MCZs for determining diffs, browsing code, finding parents, reading version informations. It would be easy to just cache this information and provide links inbetween. Remember, however, that both, a Magma-Based solution or a SS3-feature would have to respect acls in the sense, that you shouldn't be able to browse into a non-readable project by means of 'browse mc origin'.
Yes, I'd planned for each SSProject to have its own McModel instance, so access is allowed only through the existing Project level ACL.
sure, but do only acl-accessible versions get added to the McModel instance or every 'reachable' ancestor?
We have ODBMS' we should use them to exceed what we had before with only the filesystem.
:) Best -Tobias
On Mon, Aug 26, 2013 at 3:52 PM, Tobias Pape Das.Linux@gmx.de wrote:
Am 26.08.2013 um 21:59 schrieb Chris Muller ma.chris.m@gmail.com:
As Dave mentioned, the immediate priority is simply ensuring continued
operation of squeaksource.com. He is heading up the transition to a new server this week whether that's box3, box4 (Gandi) or a server hosted by Robert's group. If we cannot get it running on a new image and/or VM immediately, we'll just have to deal with the crashes/lockups until we can.
While I think Squeak should host its own services, unfortunately what we
have right now is unacceptably dangerous. It appears someone has hacked our SqueakSource implementation to bypass its original design for persistence to use "image-based" persistence. Image-based persistence, in itself, would be fine, except for the fact that our Linux-based SS system scripts carelessly KILL the SS process by PID as a means of opening the image under VNC -- potentially when the image could be in a half-saved state!
Which SqS are you referring to? Fabrizio once send Andreas and Me a (then current) image of the squeaksource.com (SS.C) and IIRC, it just had VNC running, so you probably are referring to source.squeak.org (S.S.O)?
Sorry, I also do not understand what you mean by the original design
being “hacked” for image-based persistence.
Yes, I was referring to SSO, which is running a slightly newer/forked version of SqueakSource and using the save-the-image persistence.
What do you exactly mean by save-the-image-persistence? I ask, as I carefully hand-merged/back-ported/cross-ported features from SSO _and_ SSC to SS3 prior to its alpha release.
As you know SSStorage encapsulates the behaviors needed for back-end storage support. loadMonticello:from: -- "Answer the binary contents of the original mcz file as a ByteArray." loadRepository -- "Answer the root object of this SqueakSource instance, an instance of SSRepository." saveMonticello:of:to: -- "Save the mcz file contents contained by aString." saveRepository: -- "Save the root SSRepository object into persistent storage." shutdown -- "shutdown hook"
The SqueakSource instance supporting SSO has no senders of #saveRepository: at all. However in the image is a workspace entitled: "How this image was created" with this script:
"Save the image periodically" guardian ifNotNil:[guardian terminate]. guardian := [ [true] whileTrue:[ "Synchronized with Morphic please" Smalltalk future snapshot: true andQuit: false. (Delay forSeconds: 60 minutes asSeconds) wait ]. ] fork.
I think the reason for doing this was performance, since SSFilesystem does its work in the background, I'm not sure..
Yes, I'd planned for each SSProject to have its own McModel instance, so access is allowed only through the existing Project level ACL.
sure, but do only acl-accessible versions get added to the McModel instance or every 'reachable' ancestor?
Only the acl-accessible versions. Just as each Project has its own directory when using a SSFilesystem, each Project has its own McModel when using a SSMagma. There are no cross-links to objects for other Projects in either case.
- Chris
Am 27.08.2013 um 00:08 schrieb Chris Muller ma.chris.m@gmail.com:
On Mon, Aug 26, 2013 at 3:52 PM, Tobias Pape Das.Linux@gmx.de wrote:
Am 26.08.2013 um 21:59 schrieb Chris Muller ma.chris.m@gmail.com:
As Dave mentioned, the immediate priority is simply ensuring continued
operation of squeaksource.com. He is heading up the transition to a new server this week whether that's box3, box4 (Gandi) or a server hosted by Robert's group. If we cannot get it running on a new image and/or VM immediately, we'll just have to deal with the crashes/lockups until we can.
While I think Squeak should host its own services, unfortunately what we
have right now is unacceptably dangerous. It appears someone has hacked our SqueakSource implementation to bypass its original design for persistence to use "image-based" persistence. Image-based persistence, in itself, would be fine, except for the fact that our Linux-based SS system scripts carelessly KILL the SS process by PID as a means of opening the image under VNC -- potentially when the image could be in a half-saved state!
Which SqS are you referring to? Fabrizio once send Andreas and Me a (then current) image of the squeaksource.com (SS.C) and IIRC, it just had VNC running, so you probably are referring to source.squeak.org (S.S.O)?
Sorry, I also do not understand what you mean by the original design
being “hacked” for image-based persistence.
Yes, I was referring to SSO, which is running a slightly newer/forked version of SqueakSource and using the save-the-image persistence.
What do you exactly mean by save-the-image-persistence? I ask, as I carefully hand-merged/back-ported/cross-ported features from SSO _and_ SSC to SS3 prior to its alpha release.
As you know SSStorage encapsulates the behaviors needed for back-end storage support. loadMonticello:from: -- "Answer the binary contents of the original mcz file as a ByteArray." loadRepository -- "Answer the root object of this SqueakSource instance, an instance of SSRepository." saveMonticello:of:to: -- "Save the mcz file contents contained by aString." saveRepository: -- "Save the root SSRepository object into persistent storage." shutdown -- "shutdown hook"
The SqueakSource instance supporting SSO has no senders of #saveRepository: at all. However in the image is a workspace entitled: "How this image was created" with this script:
"Save the image periodically" guardian ifNotNil:[guardian terminate]. guardian := [ [true] whileTrue:[ "Synchronized with Morphic please" Smalltalk future snapshot: true andQuit: false. (Delay forSeconds: 60 minutes asSeconds) wait ]. ] fork.
I think the reason for doing this was performance, since SSFilesystem does its work in the background, I'm not sure..
Ah I see. I should have known. #saveRepository: is gone/noop in SS3/GemStone too, for obvious reasons.
Yes, I'd planned for each SSProject to have its own McModel instance, so access is allowed only through the existing Project level ACL.
sure, but do only acl-accessible versions get added to the McModel instance or every 'reachable' ancestor?
Only the acl-accessible versions. Just as each Project has its own directory when using a SSFilesystem, each Project has its own McModel when using a SSMagma. There are no cross-links to objects for other Projects in either case.
But iirc, SqueakSource uses the in-image MCDefinition-cache, have you disabled that for Magma?
Best -Tobias
On Mon, 26 Aug 2013, Chris Muller wrote:
> As Dave mentioned, the immediate priority is simply ensuring continued operation of squeaksource.com. He is heading up the transition to a new server this week whether that's box3, box4 (Gandi) or a server hosted by Robert's group. If we cannot get it running on a new image and/or VM immediately, we'll just have to deal with the crashes/lockups until we can. > > While I think Squeak should host its own services, unfortunately what we have right now is unacceptably dangerous. It appears someone has hacked our SqueakSource implementation to bypass its original design for persistence to use "image-based" persistence. Image-based persistence, in itself, would be fine, except for the fact that our Linux-based SS system scripts carelessly KILL the SS process by PID as a means of opening the image under VNC -- potentially when the image could be in a half-saved state!
Which SqS are you referring to? Fabrizio once send Andreas and Me a (then current) image of the squeaksource.com (SS.C) and IIRC, it just had VNC running, so you probably are referring to source.squeak.org (S.S.O)?
Sorry, I also do not understand what you mean by the original design being “hacked” for image-based persistence.
Yes, I was referring to SSO, which is running a slightly newer/forked version of SqueakSource and using the save-the-image persistence.
SSO is also running on an old version of Squeak and old VM. Whatever is good for the SS upgrade will be good for SSO and vice-versa.
The only reason why SSO is running on an old VM is that the server's CPU doesn't support SSE2 instructions, so it can't run Cog VMs.
And don't forget that Bert implemented extra features for SqueakSource2 (the thing we use for SSO) to support the Trunk process.
What are the advantages of using Gemstone instead of Squeak?
Levente
> Such a misfortune could wipe out the entire community if we were unable to recover from a backup. Therefore, I fully support your idea to move to SqueakSource3 running on GemStone. > > But there is one more option. Last month, Tim Rowledge expressed interest in having a more-capable SqueakSource that could support what Magma-based MC repositories provide today: > > http://lists.squeakfoundation.org/pipermail/squeak-dev/2013-July/172243.html > > The advantage to this approach, besides having a working server example based on Squeak, is that all MCDefinitions across all Versions of each SSProject would be canonicalized and indexed inside the repository, so that the 'browse mc versions' and 'browse mc origin' functions which are afforded by Magma-based MCRepository's would be available on the trunk repository for everyone.
I had the Idea (but not yet enough pressure) to provide similar features for SS3.
McModel is self-contained. Besides substituting an RcDictionary for the MagmaPreallocatedDictionary, you might be able to lift it straight from Magma. It's indexing is sufficient to provide the minimum MCRepository subclass-responsibility API, plus the aforementioned (#historyOf: aMCDefinition) and (#originOf: aMCDefinition) functions, all without MC even knowing it exists. > I've already begun the work for this, but if there is no interest in this, I will simply bow out and support the idea that we go with the GemStone solution.
SS3 already cracks open MCZs for determining diffs, browsing code, finding parents, reading version informations. It would be easy to just cache this information and provide links inbetween. Remember, however, that both, a Magma-Based solution or a SS3-feature would have to respect acls in the sense, that you shouldn't be able to browse into a non-readable project by means of 'browse mc origin'.
Yes, I'd planned for each SSProject to have its own McModel instance, so access is allowed only through the existing Project level ACL. We have ODBMS' we should use them to exceed what we had before with only the filesystem.
- Chris
PS -- I attached McModel.st.gz in case you're interested.
Hi
Am 27.08.2013 um 00:16 schrieb Levente Uzonyi leves@elte.hu :
On Mon, 26 Aug 2013, Chris Muller wrote:
Sorry, I also do not understand what you mean by the original design being “hacked” for image-based persistence.
Yes, I was referring to SSO, which is running a slightly newer/forked version of SqueakSource and using the save-the-image persistence. SSO is also running on an old version of Squeak and old VM. Whatever is good for the SS upgrade will be good for SSO and vice-versa.
The only reason why SSO is running on an old VM is that the server's CPU doesn't support SSE2 instructions, so it can't run Cog VMs.
I see
And don't forget that Bert implemented extra features for SqueakSource2 (the thing we use for SSO) to support the Trunk process.
I wont forget, because I integrated them into SS3 :)
What are the advantages of using Gemstone instead of Squeak?
From my experience: Stability, Speed, and ease of Integration. - Stability: - when I switched from the squeak-based SS at my site to the SS3 based one, my unexpected downtimes went from twice a day for at last half an hour to once in three years. - Speed: - In our group, we experienced, that the response-time of SS3 was better than the image-based one. Recently we had to tweak things for performance, and a severa-thousand version project on ss3.gemstone.com went from over 50s load time to .4s (listing of all verisons) - Integration: - With Fcgi and daemontools, I was able to have a nice integration in my web infrastructure, reliably coming up again after failure (this is more of a GLASS feature) - I have introduced several hooks to easily extend and strip down SS3, you can have a minimal version without Filesystem support or commit-mails, or a full-fledged SS3 with additional plugins. Some students that took a class at our group developed direct CI-integration for SS3, for example (I presented their work at the ESUG in Edinburgh in 2010) (I primarily use the plugin mechanism to display a legally necessary imprint on our groups SS3).
Best -Tobias
box-admins@lists.squeakfoundation.org