I can't do a trunk update of any of my squeak 6 images right now. This is the case for several 6.0 release images on several platforms and a freshly downloaded 6.1alpha system.
The problem is that the securesocketstream>receiveData method gets a connection closed response. This isn't new and 'm sure I should remember more about it but... in fact the only notes I have that relate to the problem date back to '04 and I certainly don't remember any details.
The especially annoying thing is that using https://source.squeak.org/trunk/ in a web browser works fine, so it probably isn't that the server is dead. Is this another stupid SSL issue?
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful random insult:- Paralyzed from the neck up.
On Mon, Dec 26, 2022 at 04:30:52PM -0800, tim Rowledge wrote:
I can't do a trunk update of any of my squeak 6 images right now. This is the case for several 6.0 release images on several platforms and a freshly downloaded 6.1alpha system.
The problem is that the securesocketstream>receiveData method gets a connection closed response. This isn't new and 'm sure I should remember more about it but... in fact the only notes I have that relate to the problem date back to '04 and I certainly don't remember any details.
The especially annoying thing is that using https://source.squeak.org/trunk/ in a web browser works fine, so it probably isn't that the server is dead. Is this another stupid SSL issue?
Confirmed, I'm seeing the same. It seems specific to source.squeak.org/trunk as opposed to source.squeak.org/inbox. Something has changed quite recently, I'm pretty sure that updates were working earlier today.
Dave
On Mon, Dec 26, 2022 at 08:13:11PM -0500, David T. Lewis wrote:
On Mon, Dec 26, 2022 at 04:30:52PM -0800, tim Rowledge wrote:
I can't do a trunk update of any of my squeak 6 images right now. This is the case for several 6.0 release images on several platforms and a freshly downloaded 6.1alpha system.
The problem is that the securesocketstream>receiveData method gets a connection closed response. This isn't new and 'm sure I should remember more about it but... in fact the only notes I have that relate to the problem date back to '04 and I certainly don't remember any details.
The especially annoying thing is that using https://source.squeak.org/trunk/ in a web browser works fine, so it probably isn't that the server is dead. Is this another stupid SSL issue?
Confirmed, I'm seeing the same. It seems specific to source.squeak.org/trunk as opposed to source.squeak.org/inbox. Something has changed quite recently, I'm pretty sure that updates were working earlier today.
Opening a repository browser on trunk gives the ConnectionClosed symptom as described. So I tried it with an older Squeak 4.5 image, and it fails with different symptoms, the progress bar stops part way through retrieving the list of file names.
This is not SSL related, it seems to be something on the server that is specifically related to the trunk repository, or possibly something related to the (size of?) list of file names being retrieved.
Dave
On 2022-12-26, at 6:10 PM, David T. Lewis lewis@mail.msen.com wrote:
This is not SSL related, it seems to be something on the server that is specifically related to the trunk repository, or possibly something related to the (size of?) list of file names being retrieved.
Bizarrely - perhaps - that smacks of the '08 (not '04!) problems I still have emails about; in that case it was the response(s) being extraordinarily slow.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Unprecedented performance: Nothing ever ran this slow before.
Hi all,
The server appeared to be running fine. I was able to vnc into the image and everything appeared normal. Nothing abnormal in the log. The only anomaly was it was taking up a lot of RAM (7 GB) which is really abnormal, so I restarted it. After coming back up it's now taking between 1-2 GB which is normal for this sized repository.
However, the trunk directory listing is still failing!
After considerable live debugging, I could find no errors or abnormalities occuring in the Squeak image. But, dog gone it, Seaside apps with continuations and dynamic vars are really difficult to debug, so I may have missed something, but I don't think so).
So, it *appears* at this point to be some sort of HTTP response size limitation, possibly within the proxy server. Levente? There are currently over 14000 files in 'trunk'. All the other projects have far fewer files (e.g., 'inbox', 'treated', etc.) and appear to be working. So, I changed the production server code to only return the last 1000 of those 14000, it's "working" again.
I'm afraid I'm at a loss for what else the problem might be, or how to solve it, so, here are my suggestions:
- any HTTP / Proxy experts know of any server configuration parameters that could be affecting this? - If so, can we adjust it? Or break up trunk into smaller projects? - change the /ss code to only return one from each project round-robin until a total of 1000 have been retrieved. But then we couldn't access old versions except directly -- which may or may not be a problem...?
Thoughts? We're going to have to put our heads together on this one.
- Chris
On Mon, Dec 26, 2022 at 8:10 PM David T. Lewis lewis@mail.msen.com wrote:
On Mon, Dec 26, 2022 at 08:13:11PM -0500, David T. Lewis wrote:
On Mon, Dec 26, 2022 at 04:30:52PM -0800, tim Rowledge wrote:
I can't do a trunk update of any of my squeak 6 images right now. This is the case for several 6.0 release images on several platforms and a freshly downloaded 6.1alpha system.
The problem is that the securesocketstream>receiveData method gets a connection closed response. This isn't new and 'm sure I should remember more about it but... in fact the only notes I have that relate to the problem date back to '04 and I certainly don't remember any details.
The especially annoying thing is that using https://source.squeak.org/trunk/ in a web browser works fine, so it probably isn't that the server is dead. Is this another stupid SSL issue?
Confirmed, I'm seeing the same. It seems specific to source.squeak.org/trunk as opposed to source.squeak.org/inbox. Something has changed quite recently, I'm pretty sure that updates were working earlier today.
Opening a repository browser on trunk gives the ConnectionClosed symptom as described. So I tried it with an older Squeak 4.5 image, and it fails with different symptoms, the progress bar stops part way through retrieving the list of file names.
This is not SSL related, it seems to be something on the server that is specifically related to the trunk repository, or possibly something related to the (size of?) list of file names being retrieved.
Dave
Thanks Chris,
++box-admins
I opened an item on the Slack box-admins channel to follow up on this.
Dave
On Mon, Dec 26, 2022 at 11:02:16PM -0600, Chris Muller wrote:
Hi all,
The server appeared to be running fine. I was able to vnc into the image and everything appeared normal. Nothing abnormal in the log. The only anomaly was it was taking up a lot of RAM (7 GB) which is really abnormal, so I restarted it. After coming back up it's now taking between 1-2 GB which is normal for this sized repository.
However, the trunk directory listing is still failing!
After considerable live debugging, I could find no errors or abnormalities occuring in the Squeak image. But, dog gone it, Seaside apps with continuations and dynamic vars are really difficult to debug, so I may have missed something, but I don't think so).
So, it *appears* at this point to be some sort of HTTP response size limitation, possibly within the proxy server. Levente? There are currently over 14000 files in 'trunk'. All the other projects have far fewer files (e.g., 'inbox', 'treated', etc.) and appear to be working. So, I changed the production server code to only return the last 1000 of those 14000, it's "working" again.
I'm afraid I'm at a loss for what else the problem might be, or how to solve it, so, here are my suggestions:
- any HTTP / Proxy experts know of any server configuration
parameters that could be affecting this?
- If so, can we adjust it? Or break up trunk into smaller projects?
- change the /ss code to only return one from each project
round-robin until a total of 1000 have been retrieved. But then we couldn't access old versions except directly -- which may or may not be a problem...?
Thoughts? We're going to have to put our heads together on this one.
- Chris
On Mon, Dec 26, 2022 at 8:10 PM David T. Lewis lewis@mail.msen.com wrote:
On Mon, Dec 26, 2022 at 08:13:11PM -0500, David T. Lewis wrote:
On Mon, Dec 26, 2022 at 04:30:52PM -0800, tim Rowledge wrote:
I can't do a trunk update of any of my squeak 6 images right now. This is the case for several 6.0 release images on several platforms and a freshly downloaded 6.1alpha system.
The problem is that the securesocketstream>receiveData method gets a connection closed response. This isn't new and 'm sure I should remember more about it but... in fact the only notes I have that relate to the problem date back to '04 and I certainly don't remember any details.
The especially annoying thing is that using https://source.squeak.org/trunk/ in a web browser works fine, so it probably isn't that the server is dead. Is this another stupid SSL issue?
Confirmed, I'm seeing the same. It seems specific to source.squeak.org/trunk as opposed to source.squeak.org/inbox. Something has changed quite recently, I'm pretty sure that updates were working earlier today.
Opening a repository browser on trunk gives the ConnectionClosed symptom as described. So I tried it with an older Squeak 4.5 image, and it fails with different symptoms, the progress bar stops part way through retrieving the list of file names.
This is not SSL related, it seems to be something on the server that is specifically related to the trunk repository, or possibly something related to the (size of?) list of file names being retrieved.
Dave
Thanks, everybody.
So, I changed the production server code to only return the last 1000 of those 14000, it's "working" again.
Just in case this wasn't obvious, updating your image still doesn't work because many versions are missing. Round-robin or some date-based heuristic would surely be a better (yet more complex) workaround. :-)
Best, Christoph
________________________________ Von: Squeak-dev squeak-dev-bounces@lists.squeakfoundation.org im Auftrag von David T. Lewis lewis@mail.msen.com Gesendet: Dienstag, 27. Dezember 2022 20:07:32 An: Chris Muller Cc: box-admins@lists.squeakfoundation.org; The general-purpose Squeak developers list Betreff: Re: [squeak-dev] trunk update stream appears to have become inaccessible
Thanks Chris,
++box-admins
I opened an item on the Slack box-admins channel to follow up on this.
Dave
On Mon, Dec 26, 2022 at 11:02:16PM -0600, Chris Muller wrote:
Hi all,
The server appeared to be running fine. I was able to vnc into the image and everything appeared normal. Nothing abnormal in the log. The only anomaly was it was taking up a lot of RAM (7 GB) which is really abnormal, so I restarted it. After coming back up it's now taking between 1-2 GB which is normal for this sized repository.
However, the trunk directory listing is still failing!
After considerable live debugging, I could find no errors or abnormalities occuring in the Squeak image. But, dog gone it, Seaside apps with continuations and dynamic vars are really difficult to debug, so I may have missed something, but I don't think so).
So, it *appears* at this point to be some sort of HTTP response size limitation, possibly within the proxy server. Levente? There are currently over 14000 files in 'trunk'. All the other projects have far fewer files (e.g., 'inbox', 'treated', etc.) and appear to be working. So, I changed the production server code to only return the last 1000 of those 14000, it's "working" again.
I'm afraid I'm at a loss for what else the problem might be, or how to solve it, so, here are my suggestions:
- any HTTP / Proxy experts know of any server configuration
parameters that could be affecting this?
- If so, can we adjust it? Or break up trunk into smaller projects?
- change the /ss code to only return one from each project
round-robin until a total of 1000 have been retrieved. But then we couldn't access old versions except directly -- which may or may not be a problem...?
Thoughts? We're going to have to put our heads together on this one.
- Chris
On Mon, Dec 26, 2022 at 8:10 PM David T. Lewis lewis@mail.msen.com wrote:
On Mon, Dec 26, 2022 at 08:13:11PM -0500, David T. Lewis wrote:
On Mon, Dec 26, 2022 at 04:30:52PM -0800, tim Rowledge wrote:
I can't do a trunk update of any of my squeak 6 images right now. This is the case for several 6.0 release images on several platforms and a freshly downloaded 6.1alpha system.
The problem is that the securesocketstream>receiveData method gets a connection closed response. This isn't new and 'm sure I should remember more about it but... in fact the only notes I have that relate to the problem date back to '04 and I certainly don't remember any details.
The especially annoying thing is that using https://source.squeak.org/trunk/ in a web browser works fine, so it probably isn't that the server is dead. Is this another stupid SSL issue?
Confirmed, I'm seeing the same. It seems specific to source.squeak.org/trunk as opposed to source.squeak.org/inbox. Something has changed quite recently, I'm pretty sure that updates were working earlier today.
Opening a repository browser on trunk gives the ConnectionClosed symptom as described. So I tried it with an older Squeak 4.5 image, and it fails with different symptoms, the progress bar stops part way through retrieving the list of file names.
This is not SSL related, it seems to be something on the server that is specifically related to the trunk repository, or possibly something related to the (size of?) list of file names being retrieved.
Dave
So, I changed the production server code to only return the last 1000
of those 14000, it's "working" again.
Just in case this wasn't obvious, updating your image still doesn't work because many versions are missing. Round-robin or some date-based heuristic would surely be a better (yet more complex) workaround. :-)
Yup. And, it would alleviate our property of unsustainability. I'm curious about the proxy configuration, but this has me wanting to come up with something in the app code in any case. We're just sending that ever-growing list over the wire hundreds, possibly thousands of times, every day. I like your idea of having age be a factor.
The impact of such a change would occur in the Repository browser. There would be fewer Versions in the right-hand pane for each package. However, I think it would be able to use the ancestry to diff anything older than what the Repository browser showed, so it may not be a big deal.. Does anyone see any holes in this idea?
- Chris
From the perspective of someone who has only gotten to know SqueakSource from the client side: The possibility to enumerate *all* trunk versions is important for me in several situations. It would be quite a shame if one could no longer access the full ancestry of a version or browse an ancient version of any package through the UI. The release notes generator in the ReleaseBuilder could also be affected, I think.
With a view to modern REST and GraphQL APIs, would it be possible to add support for some parameters to the requests in question to enable pagination or filter the retrieved information vertically (i.e., leave out some metadata)?
Best,
Christoph
________________________________ Von: Chris Muller ma.chris.m@gmail.com Gesendet: Dienstag, 27. Dezember 2022 21:03:58 An: Thiede, Christoph Cc: The general-purpose Squeak developers list; box-admins@lists.squeakfoundation.org Betreff: Re: [squeak-dev] trunk update stream appears to have become inaccessible
So, I changed the production server code to only return the last 1000 of those 14000, it's "working" again.
Just in case this wasn't obvious, updating your image still doesn't work because many versions are missing. Round-robin or some date-based heuristic would surely be a better (yet more complex) workaround. :-)
Yup. And, it would alleviate our property of unsustainability. I'm curious about the proxy configuration, but this has me wanting to come up with something in the app code in any case. We're just sending that ever-growing list over the wire hundreds, possibly thousands of times, every day. I like your idea of having age be a factor.
The impact of such a change would occur in the Repository browser. There would be fewer Versions in the right-hand pane for each package. However, I think it would be able to use the ancestry to diff anything older than what the Repository browser showed, so it may not be a big deal.. Does anyone see any holes in this idea?
- Chris
Hi Christoph,
From the perspective of someone who has only gotten to know SqueakSource from the client side: The possibility to enumerate *all* trunk versions is important for me in several situations.
Would you mention or describe one or more of those situations, because...
It would be quite a shame if one could no longer access the full ancestry of a version or browse an ancient version of any package through the UI.
... this change would not affect the above use cases. They could still be accomplished via the "History" button, because it accesses those very old versions *directly*, by name or UUID. What we're talking about here is just the "list of files", NOT the list of ancestors, which we still have. The lists in the Repository browsers wouldn't show everything in the ancestry, but the ancestry is a better place to review that anyway. I'm not convinced there is _any_ use case that needs a list of *all* Repository files (Versions) since the beginning of time. If there is one, it's on an eternal linear degradation path, and some alternative approach should at least be considered.
The release notes generator in the ReleaseBuilder could also be affected, I think.
It shouldn't be. As above, it's in the ancestry.
With a view to modern REST and GraphQL APIs, would it be possible to add support for some parameters to the requests in question to enable pagination or filter the retrieved information vertically (i.e., leave out some metadata)?
I'm sure it is, but right now I'm just looking for TSTTCPW for getting trunk development back online. I think the round-robin with date-based balancing sounds like a good possibility for something that could be done quickly.
Best, Chris
Hi Chris,
sorry for the misunderstanding! Some use case I was thinking of would be this: Somewhere in the mailing list archive, I find a reference to a very old Monticello version that I need to check out in order to understand why something was built or changed. Usually, I would open a repository browser on Trunk and scroll down until I find that really old version, just because it is more intuitive than using the API via efficient #versionNamed:. Will this still be possible through the UI with your intended change, regardless of the number of versions in the package?
Apart from that, I can imagine that such a limitation would worsen the explorability of the database, but given that this is just meant to be a workaround and we have an urgent situation (of course, CI bundles and smalltalkCI for Trunk are broken as well) I don't think that this should be a real issue. In the long term, maybe add a small notice to the UI to indicate that the list is truncated.
Best,
Christoph
________________________________ Von: Chris Muller ma.chris.m@gmail.com Gesendet: Dienstag, 27. Dezember 2022 22:03:57 An: Thiede, Christoph Cc: The general-purpose Squeak developers list; box-admins@lists.squeakfoundation.org Betreff: Re: [squeak-dev] trunk update stream appears to have become inaccessible
Hi Christoph,
From the perspective of someone who has only gotten to know SqueakSource from the client side: The possibility to enumerate *all* trunk versions is important for me in several situations.
Would you mention or describe one or more of those situations, because...
It would be quite a shame if one could no longer access the full ancestry of a version or browse an ancient version of any package through the UI.
... this change would not affect the above use cases. They could still be accomplished via the "History" button, because it accesses those very old versions *directly*, by name or UUID. What we're talking about here is just the "list of files", NOT the list of ancestors, which we still have. The lists in the Repository browsers wouldn't show everything in the ancestry, but the ancestry is a better place to review that anyway. I'm not convinced there is _any_ use case that needs a list of *all* Repository files (Versions) since the beginning of time. If there is one, it's on an eternal linear degradation path, and some alternative approach should at least be considered.
The release notes generator in the ReleaseBuilder could also be affected, I think.
It shouldn't be. As above, it's in the ancestry.
With a view to modern REST and GraphQL APIs, would it be possible to add support for some parameters to the requests in question to enable pagination or filter the retrieved information vertically (i.e., leave out some metadata)?
I'm sure it is, but right now I'm just looking for TSTTCPW for getting trunk development back online. I think the round-robin with date-based balancing sounds like a good possibility for something that could be done quickly.
Best, Chris
We appear to be all fixed, but I don't know why! Did someone do something?
I was getting ready to append the following fix that would limit the output of that one operation to the most recent 5000 items. In SSVersionListing>>#setProjects:.
setProjects: aCollection projects := aCollection. associations := Array streamContents: [ :stream | projects do: [ :project | self appendProject: project to: stream ] ]. associations sort: [ :x :y | x value timestamp > y value timestamp ]. "Limit to the last 5000 versions." associations := associations copyFrom: 1 to: (associations size min: 5000)
I've had the above typed into the browser in the production image, ready to save, when I decided I wanted to observe the "broken-before --> fixed-after" state change, broke-before was (is) no longer broken!
Strange. So, I will hold off then. If this fixes trunk development for now, I'd rather invite the problem back to reveal itself again for better understanding than put in any arbitrary fixes to "hide" it. I'm not working in trunk right now, if the problem returns, please email me again.
- Chris
On Tue, Dec 27, 2022 at 3:23 PM Thiede, Christoph < Christoph.Thiede@student.hpi.uni-potsdam.de> wrote:
Hi Chris,
sorry for the misunderstanding! Some use case I was thinking of would be this: Somewhere in the mailing list archive, I find a reference to a very old Monticello version that I need to check out in order to understand why something was built or changed. Usually, I would open a repository browser on Trunk and scroll down until I find that really old version, just because it is more intuitive than using the API via efficient #versionNamed:. Will this still be possible through the UI with your intended change, regardless of the number of versions in the package?
Apart from that, I can imagine that such a limitation would worsen the explorability of the database, but given that this is just meant to be a workaround and we have an urgent situation (of course, CI bundles and smalltalkCI for Trunk are broken as well) I don't think that this should be a real issue. In the long term, maybe add a small notice to the UI to indicate that the list is truncated.
Best,
Christoph
*Von:* Chris Muller ma.chris.m@gmail.com *Gesendet:* Dienstag, 27. Dezember 2022 22:03:57 *An:* Thiede, Christoph *Cc:* The general-purpose Squeak developers list; box-admins@lists.squeakfoundation.org *Betreff:* Re: [squeak-dev] trunk update stream appears to have become inaccessible
Hi Christoph,
From the perspective of someone who has only gotten to know SqueakSource from the client side: The possibility to enumerate *all* trunk versions is important for me in several situations.
Would you mention or describe one or more of those situations, because...
It would be quite a shame if one could no longer access the full ancestry of a version or browse an ancient version of any package through the UI.
... this change would not affect the above use cases. They could still be accomplished via the "History" button, because it accesses those very old versions *directly*, by name or UUID. What we're talking about here is just the "list of files", NOT the list of ancestors, which we still have. The lists in the Repository browsers wouldn't show everything in the ancestry, but the ancestry is a better place to review that anyway. I'm not convinced there is _any_ use case that needs a list of *all* Repository files (Versions) since the beginning of time. If there is one, it's on an eternal linear degradation path, and some alternative approach should at least be considered.
The release notes generator in the ReleaseBuilder could also be affected, I think.
It shouldn't be. As above, it's in the ancestry.
With a view to modern REST and GraphQL APIs, would it be possible to add support for some parameters to the requests in question to enable pagination or filter the retrieved information vertically (i.e., leave out some metadata)?
I'm sure it is, but right now I'm just looking for TSTTCPW for getting trunk development back online. I think the round-robin with date-based balancing sounds like a good possibility for something that could be done quickly.
Best, Chris
On 2022-12-27, at 2:39 PM, Chris Muller asqueaker@gmail.com wrote:
We appear to be all fixed, but I don't know why!
It's happily updating for me now. I didn't change anything, promise.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: LA: Lockout Access
Very nice, I confirm that uploading new versions works as well again. Anyway, thanks a lot for your efforts, Chris, very much appreciated!
Best,
Christoph
________________________________ Von: Squeak-dev squeak-dev-bounces@lists.squeakfoundation.org im Auftrag von tim Rowledge tim@rowledge.org Gesendet: Mittwoch, 28. Dezember 2022 00:21:41 An: Chris Muller; The general-purpose Squeak developers list Betreff: Re: [squeak-dev] trunk update stream appears to have become inaccessible
On 2022-12-27, at 2:39 PM, Chris Muller asqueaker@gmail.com wrote:
We appear to be all fixed, but I don't know why!
It's happily updating for me now. I didn't change anything, promise.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: LA: Lockout Access
Hmm, slight strangeness for one or two updates though; trying to replace newer versions of method with older.
For example -
and
That seems like either a) a method was changed and then reverted b) a method version was wrongly saved
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful random insult:- Understands English as well as any parrot.
Hi all,
while trying to update a pretty old Trunk image, I am reproducibly seeing this error while processing update-mt.500.mcm:
NotFound: 'http://source.squeak.org/trunk/Collections-mt.974%28ct.926%29.mcd'
Is this a bug in the update map or the updater or are there still some problems with the server?
Best,
Christoph
________________________________ Von: Squeak-dev squeak-dev-bounces@lists.squeakfoundation.org im Auftrag von tim Rowledge tim@rowledge.org Gesendet: Mittwoch, 28. Dezember 2022 02:13:49 An: The general-purpose Squeak developers list Betreff: Re: [squeak-dev] trunk update stream appears to have become inaccessible
Hmm, slight strangeness for one or two updates though; trying to replace newer versions of method with older.
For example -
[cid:ADB09C63-2E56-4F86-8086-E69ABA3B74A7]
and
[cid:1D2CF8BB-24A7-45F5-9AE7-03667B7EC78B]
That seems like either a) a method was changed and then reverted b) a method version was wrongly saved
tim -- tim Rowledge; tim@rowledge.orgmailto:tim@rowledge.org; http://www.rowledge.org/tim Useful random insult:- Understands English as well as any parrot.
browse an ancient version of any package through the UI
This one could be affected. However, it could easily be fixed in the client by moving it to the ancestry list menu, where it could do the direct (by uuid) access.
I also just realized, for the round-robin, it would need to make sure to always include ALL that are NOT an ancestor of the newest. That way, we can see old things that might still want to be merged...
squeak-dev@lists.squeakfoundation.org
-
Chris Muller -
Chris Muller -
David T. Lewis -
Thiede, Christoph -
tim Rowledge