I was able to get VNC access to the image and look at a process browser, see attached.

I think that a good thing to do would be:

1) connect via VNC

2) save the image under a different name for later evaluation

3) immediately exit the image and let systemd restart it automatically

I will be away for a couple of hours and cannot follow up now, but will check back later.

Dave

On 2024-02-02 17:25, lewis@mail.msen.com wrote:

Adding Chris in case he may be able to take a quick look at the source.squeak.org image before we restart it.

The fix will be to restart the squeaksource service on andreas.box.squeak.org. It is using 100% cpu right now. A VNC connection to the image will probably show something interesting.

I am away with limited access but I will try to check back in an hour or so.

Dave

On 2024-02-02 16:24, leves wrote: Hi Marcel,

This time alan is not full, there's 80 GB free space on its disk. IIRC either the deletion strategy has been implemented by Yannis or a manual round of deletions happened.

However the mailing lists' cron jobs are consuming significant amount of CPU and RAM. Swap size is ~5GB, so that could be a reason why source.squeak.org is slow/down. The monthly job has already consumed 40 CPU hours. Since it's python, it's single threaded, so that's been running for 40 hours. The hourly job was struggling as well, it's been running for over 13 minutes.

However, the source.squeak.org image is also consuming 100% cpu, so there's probably some other issue with it. Also, squeaksource.com works, so it's clearly the source.squeak.org image that needs to be investigated.

Btw, source.squeak.org is started with parameter -memory 2048m. I suppose -maxoldspace would be a better option to use instead.

Levente

On 2024. 02. 02. 16:45, Taeumel, Marcel wrote: Hi board, hi box-admins

five months ago, I discussed a deletion strategy of our CI builds for files.squeak.org with Yannis. Unfortunately, he has not had the time to implement it yet.

Now, it seems, "alan" is full again. And we are experiencing problems with source.squeak.org and Monticello again.

We should bump "alan" with another 5 GB and get on with our deletion strategy. A lot of old stuff does not even need a fancy script. Here is my prior take on this, which I discussed with Yannis:

Ansonsten hier die alten Files, die *gelöscht *werden können:

• http://files.squeak.org/5.1beta/

http://files.squeak.org/5.1beta/ alles bis auf 16514 (32-bit und 64-bit)

• http://files.squeak.org/5.2alpha/

http://files.squeak.org/5.2alpha/ alles bis auf 18184 (32-bit und 64-bit)

• http://files.squeak.org/5.2beta/

http://files.squeak.org/5.2beta/ alles bis auf 18199 (32-bit und 64-bit)

• http://files.squeak.org/5.3alpha/

http://files.squeak.org/5.3alpha/ alles bis auf 19256 (32-bit und 64-bit)

• http://files.squeak.org/5.3beta/

http://files.squeak.org/5.3beta/ alles bis auf 19412 (32-bit und 64-bit)

• http://files.squeak.org/6.0alpha/

http://files.squeak.org/6.0alpha/ alles bis auf 21752 (32-bit und 64-bit)

• http://files.squeak.org/6.0beta/

http://files.squeak.org/6.0beta/ alles bis auf 22075 (32-bit und 64-bit)

Und für 6.1alpha (bzw. der aktuelle Trunk) würde ich *behalten *wollen:

• pro Monat drei Builds anfang/mitte/ende (32-bit und 64-bit)
• den aktuellen Monat komplett bzw. die letzten 30 Tage komplett

Und bitte irgendwo eine Liste führen, wo man manuell Builds vom aktuellen Trunk hinzufügen kann, die niemals gelöscht werden dürfen. Quasi pinning. Aber keine Ahnung, ob wir sowas mal brauchen. Ich denke, dass nur der aktuelle http://files.squeak.org/trunk/ http://files.squeak.org/trunk/ so eine automatische Lösch-Strategie braucht, weil tägliche Builds. den Rest machen wir manuell.

Best, Marcel

Hi all,

The source.squeak.org image has been on and off sluggish for a day or so, and it does not seem to be getting better. It's using about 1.8G memory and 100% CPU, and VNC connection to the image shows lots of processes similar to previous conditions (see below).

I am going to try to connect with VNC and save a copy of the image, then exit it to allow a restart.

CC Christophe who reported the issue on squeak-dev https://lists.squeakfoundation.org/archives/list/squeak-dev@lists.squeakfoun...

Dave

On 2024-02-02 20:56, lewis@mail.msen.com wrote:

The source.squeak.org image is still bogged down and unresponsive, so I am going to restart it as described below. Hopefully it will come back on line cleanly with the normal systemd restart procedure.

Dave

On 2024-02-02 18:03, lewis@mail.msen.com wrote:

I was able to get VNC access to the image and look at a process browser, see attached.

I think that a good thing to do would be:

1. connect via VNC

2. save the image under a different name for later evaluation

3. immediately exit the image and let systemd restart it automatically

I will be away for a couple of hours and cannot follow up now, but will check back later.

Dave

On 2024-02-02 17:25, lewis@mail.msen.com wrote:

Adding Chris in case he may be able to take a quick look at the source.squeak.org image before we restart it.

The fix will be to restart the squeaksource service on andreas.box.squeak.org. It is using 100% cpu right now. A VNC connection to the image will probably show something interesting.

I am away with limited access but I will try to check back in an hour or so.

Dave

On 2024-02-02 16:24, leves wrote: Hi Marcel,

This time alan is not full, there's 80 GB free space on its disk. IIRC either the deletion strategy has been implemented by Yannis or a manual round of deletions happened.

However the mailing lists' cron jobs are consuming significant amount of CPU and RAM. Swap size is ~5GB, so that could be a reason why source.squeak.org is slow/down. The monthly job has already consumed 40 CPU hours. Since it's python, it's single threaded, so that's been running for 40 hours. The hourly job was struggling as well, it's been running for over 13 minutes.

However, the source.squeak.org image is also consuming 100% cpu, so there's probably some other issue with it. Also, squeaksource.com works, so it's clearly the source.squeak.org image that needs to be investigated.

Btw, source.squeak.org is started with parameter -memory 2048m. I suppose -maxoldspace would be a better option to use instead.

Levente

On 2024. 02. 02. 16:45, Taeumel, Marcel wrote: Hi board, hi box-admins

five months ago, I discussed a deletion strategy of our CI builds for files.squeak.org with Yannis. Unfortunately, he has not had the time to implement it yet.

Now, it seems, "alan" is full again. And we are experiencing problems with source.squeak.org and Monticello again.

We should bump "alan" with another 5 GB and get on with our deletion strategy. A lot of old stuff does not even need a fancy script. Here is my prior take on this, which I discussed with Yannis:

Ansonsten hier die alten Files, die *gelöscht *werden können:

• http://files.squeak.org/5.1beta/

http://files.squeak.org/5.1beta/ alles bis auf 16514 (32-bit und 64-bit)

• http://files.squeak.org/5.2alpha/

http://files.squeak.org/5.2alpha/ alles bis auf 18184 (32-bit und 64-bit)

• http://files.squeak.org/5.2beta/

http://files.squeak.org/5.2beta/ alles bis auf 18199 (32-bit und 64-bit)

• http://files.squeak.org/5.3alpha/

http://files.squeak.org/5.3alpha/ alles bis auf 19256 (32-bit und 64-bit)

• http://files.squeak.org/5.3beta/

http://files.squeak.org/5.3beta/ alles bis auf 19412 (32-bit und 64-bit)

• http://files.squeak.org/6.0alpha/

http://files.squeak.org/6.0alpha/ alles bis auf 21752 (32-bit und 64-bit)

• http://files.squeak.org/6.0beta/

http://files.squeak.org/6.0beta/ alles bis auf 22075 (32-bit und 64-bit)

Und für 6.1alpha (bzw. der aktuelle Trunk) würde ich *behalten *wollen:

• pro Monat drei Builds anfang/mitte/ende (32-bit und 64-bit)
• den aktuellen Monat komplett bzw. die letzten 30 Tage komplett

Und bitte irgendwo eine Liste führen, wo man manuell Builds vom aktuellen Trunk hinzufügen kann, die niemals gelöscht werden dürfen. Quasi pinning. Aber keine Ahnung, ob wir sowas mal brauchen. Ich denke, dass nur der aktuelle http://files.squeak.org/trunk/ http://files.squeak.org/trunk/ so eine automatische Lösch-Strategie braucht, weil tägliche Builds. den Rest machen wir manuell.

Best, Marcel

Hi Dave,

Thanks for taking care of it.

The process you mentioned as a suspect is kicked off from SSFilesystem>>#saveRepositoryNow:, and names the Process as 'SSFileSystem>>#saveRepository:'. Did you see that process in the process listing? It's forked at priority 29, which is below the processing of client requests, so I'm highly doubtful that is the process causing the meltdown.

I just logged in and checked. The data.obj files look to be the correct size (91M) with today's date, as with the last few backups. Entries in ss.log look normal.

As I've mentioned in my prior emails in this thread, what I've been seeing is an unnamed Process that appears to be in a state of endless recursion, and competing with client requests also at priority 30.

Just to try to make some progress on this, I just disabled Magma in the image, so we're back to using strictly the filesystem repository. It will be really interesting to see whether the problem returns after this.

Best, Chris

On Sun, Mar 3, 2024 at 4:46 PM lewis@mail.msen.com wrote:

Unfortunately I was not able to get an image snapshot this time, but the service restarted normally so the source.squeak.org repositories should be accessible again.

Although I could not get a snapshot this time, I can say that the symptoms appeared similar those in the image that I saved last time. @Chris, based on the process listing I'm suspicious that we may have some kind of meltdown scenario associated with the scheduled saves of the SSRepository to a data.obj. Just a guess but maybe worth a look.

Dave

On 2024-03-03 22:25, lewis@mail.msen.com wrote:

Hi all,

The source.squeak.org image has been on and off sluggish for a day or so, and it does not seem to be getting better. It's using about 1.8G memory and 100% CPU, and VNC connection to the image shows lots of processes similar to previous conditions (see below).

I am going to try to connect with VNC and save a copy of the image, then exit it to allow a restart.

CC Christophe who reported the issue on squeak-dev https://lists.squeakfoundation.org/archives/list/squeak-dev@lists.squeakfoun...

Dave

On 2024-02-02 20:56, lewis@mail.msen.com wrote:

The source.squeak.org image is still bogged down and unresponsive, so I am going to restart it as described below. Hopefully it will come back on line cleanly with the normal systemd restart procedure.

Dave

On 2024-02-02 18:03, lewis@mail.msen.com wrote:

I was able to get VNC access to the image and look at a process browser, see attached.

I think that a good thing to do would be:

1. connect via VNC

2. save the image under a different name for later evaluation

3. immediately exit the image and let systemd restart it automatically

I will be away for a couple of hours and cannot follow up now, but will check back later.

Dave

On 2024-02-02 17:25, lewis@mail.msen.com wrote:

Adding Chris in case he may be able to take a quick look at the source.squeak.org image before we restart it.

The fix will be to restart the squeaksource service on andreas.box.squeak.org. It is using 100% cpu right now. A VNC connection to the image will probably show something interesting.

I am away with limited access but I will try to check back in an hour or so.

Dave

On 2024-02-02 16:24, leves wrote:

Hi Marcel,

This time alan is not full, there's 80 GB free space on its disk. IIRC either the deletion strategy has been implemented by Yannis or a manual round of deletions happened.

However the mailing lists' cron jobs are consuming significant amount of CPU and RAM. Swap size is ~5GB, so that could be a reason why source.squeak.org is slow/down. The monthly job has already consumed 40 CPU hours. Since it's python, it's single threaded, so that's been running for 40 hours. The hourly job was struggling as well, it's been running for over 13 minutes.

However, the source.squeak.org image is also consuming 100% cpu, so there's probably some other issue with it. Also, squeaksource.com works, so it's clearly the source.squeak.org image that needs to be investigated.

Btw, source.squeak.org is started with parameter -memory 2048m. I suppose -maxoldspace would be a better option to use instead.

Levente

On 2024. 02. 02. 16:45, Taeumel, Marcel wrote:

Hi board, hi box-admins --

five months ago, I discussed a deletion strategy of our CI builds for files.squeak.org with Yannis. Unfortunately, he has not had the time to implement it yet.

Now, it seems, "alan" is full again. And we are experiencing problems with source.squeak.org and Monticello again.

We should bump "alan" with another 5 GB and get on with our deletion strategy. A lot of old stuff does not even need a fancy script. Here is my prior take on this, which I discussed with Yannis:

Ansonsten hier die alten Files, die *gelöscht *werden können:

• http://files.squeak.org/5.1beta/ http://files.squeak.org/5.1beta/ alles bis auf 16514 (32-bit und 64-bit)
• http://files.squeak.org/5.2alpha/ http://files.squeak.org/5.2alpha/ alles bis auf 18184 (32-bit und 64-bit)
• http://files.squeak.org/5.2beta/ http://files.squeak.org/5.2beta/ alles bis auf 18199 (32-bit und 64-bit)
• http://files.squeak.org/5.3alpha/ http://files.squeak.org/5.3alpha/ alles bis auf 19256 (32-bit und 64-bit)
• http://files.squeak.org/5.3beta/ http://files.squeak.org/5.3beta/ alles bis auf 19412 (32-bit und 64-bit)
• http://files.squeak.org/6.0alpha/ http://files.squeak.org/6.0alpha/ alles bis auf 21752 (32-bit und 64-bit)
• http://files.squeak.org/6.0beta/ http://files.squeak.org/6.0beta/ alles bis auf 22075 (32-bit und 64-bit)

Und für 6.1alpha (bzw. der aktuelle Trunk) würde ich *behalten *wollen:

• pro Monat drei Builds anfang/mitte/ende (32-bit und 64-bit)
• den aktuellen Monat komplett bzw. die letzten 30 Tage komplett

Und bitte irgendwo eine Liste führen, wo man manuell Builds vom aktuellen Trunk hinzufügen kann, die niemals gelöscht werden dürfen. Quasi pinning. Aber keine Ahnung, ob wir sowas mal brauchen. Ich denke, dass nur der aktuelle http://files.squeak.org/trunk/ http://files.squeak.org/trunk/ so eine automatische Lösch-Strategie braucht, weil tägliche Builds. den Rest machen wir manuell.

Best, Marcel