On Fri, Oct 07, 2022 at 10:09:50AM -0400, rabbit wrote:
> Hi Levente,
> 
> Is this the same machine running SqueakSource? I get frequent intermittent
> failures with accessing projects there (SqueakSource.com). If it were also
> running out of FDs and at 100% CPU, that would explain it. It???s frustrating
> me as I install a bunch of packages off SS, and it fails in the middle often.
> 

No, the https://squeaksource.com and https://source.squeak.org services
are running on different machines.

I suspect (but do know know for sure) that the intermittent issues
on squeaksource.com are related to the server image exporting its
repository data on data.obj file. The code that does this has changed
in recent years and to be honest I don't know how it works. Specifically,
I do not know what actually triggers the repeated "save" operations
that seem to be causing problems. Tips and guidance welcome.

With respect to exhausing file descriptors on source.squeak.org, I
added a monitor process on squeaksource.com (the other service) that
automatically restarts the image if it is getting too low on file
descriptors. We could probably adapt this to work on source.squeak.org
also. The code is in http://www.squeaksource.com/SSImageInit and I
can work with Chris on it if the file descriptor issue turns out
to be a recurring problem.

Here is how the file descriptor monitor works on squeaksource.com.
It would need to be slightly modified for source.squeak.org because
on that server we never want to save the image file, so probably
just change "SmalltalkImage current snapshot: true andQuit: true]"
to "SmalltalkImage current snapshot: false andQuit: true]" in the
method below.

SSImageInit class>>startSocketMonitorProcess
	"Socket leak monitor process. If the number of open file descriptors in the
	VM process gets dangerously close to the per-process limit (normally 1024),
	we are at risk of putting the image into a state in which it cannot accept
	connections, and that may not allow a clean recovery even after an image
	restart. Thus if the file descriptor count exceeds a threshold of 800, save
	the image and exit. Assume that a supervisory script will detect the image
	exit and initiate a restart."

	| vmFileCount |
	self stopSocketLeakMonitorProcess.
	SocketLeakMonitorProcess := 
	[[vmFileCount := (FileDirectory on: '/proc/', OSProcess thisOSProcess pid asString, '/fd') entries size.
	"OSProcess trace: DateAndTime now asString, ' squeakvm has ', vmFileCount asString, ' open file descriptors'."
	vmFileCount > 800 ifTrue: [
		OSProcess trace: 'Too many open file handles, save image and exit'.
		"Save the image, exit and wait for the supervisory script to restart"
		WorldState addDeferredUIMessage: [SmalltalkImage current snapshot: true andQuit: true]].
	(Delay forSeconds: 3 * 3600) wait] repeat] fork name: 'the Socket leak monitor'.

Dave