Great! I too look forward to hearing Tips and Guidance Welcome! I 
totally and wholeheartedly agree! Welcome!

I started thinking about persistence. Kafka would excel at this, fault 
tolerant, replicated and durable. I am unsure whether a directory of 
files could be used as partition persistence...perhaps the Kafka events 
sector block written are links that could be used by an Installer to 
install a package in the squeaksource directory structure.

Further, I thought the way to approach ameliorating the shortage of FDs, 
which max out per machine, is to use a redirector to load balance the 
servicing machines and having a distributed image would be helpful, as 
well as a multi-threaded vm running separate Vats on different native 
threads, But the FD shortage issue is a big concern for my system as well.

I look to use Kafka replication with EventualLinda, yet without 
persistence, and eventual references and whenReactors: to use 
Capabilities-sending style. With remote promises it becomes distributed 
amount Vats. As FDs are machine restricted, I'm interested in the many 
Vat cluster experimentation. I wish to send replicated double-encrypted 
traffic through third party bridging services and have more than 1 
replicated PartitianSession separate stacks to different vats, talking 
to the same unique receivers of services responding to messages from 
senders. A Replicating Session would balance the PartitianSessions to 
*ensure fault-tolerant* eventual message sends, responses and GC 
traffic, for remote promises. The services would have multiple 
connections to it from the same sender and using a distributed 
EventualLinda the broadcast consumer service instances running as a 
distributed fail-safe application. I.E. SqueakSource.

It's all going to take a minute! ^,^

Tips and guidance welcome.

Have a good one; keep it, light.
Kindly,
rabbit
. .. … ‘…^,^

Sent from Callisto House :: decentralized mobile homeless solutions


On 10/7/22 11:06, David T. Lewis wrote:
> On Fri, Oct 07, 2022 at 10:09:50AM -0400, rabbit wrote:
>> Hi Levente,
>>
>> Is this the same machine running SqueakSource? I get frequent intermittent
>> failures with accessing projects there (SqueakSource.com). If it were also
>> running out of FDs and at 100% CPU, that would explain it. It???s frustrating
>> me as I install a bunch of packages off SS, and it fails in the middle often.
>>
> No, thehttps://squeaksource.com  andhttps://source.squeak.org  services
> are running on different machines.
>
> I suspect (but do know know for sure) that the intermittent issues
> on squeaksource.com are related to the server image exporting its
> repository data on data.obj file. The code that does this has changed
> in recent years and to be honest I don't know how it works. Specifically,
> I do not know what actually triggers the repeated "save" operations
> that seem to be causing problems. Tips and guidance welcome.
>
> With respect to exhausing file descriptors on source.squeak.org, I
> added a monitor process on squeaksource.com (the other service) that
> automatically restarts the image if it is getting too low on file
> descriptors. We could probably adapt this to work on source.squeak.org
> also. The code is inhttp://www.squeaksource.com/SSImageInit  and I
> can work with Chris on it if the file descriptor issue turns out
> to be a recurring problem.
>
> Here is how the file descriptor monitor works on squeaksource.com.
> It would need to be slightly modified for source.squeak.org because
> on that server we never want to save the image file, so probably
> just change "SmalltalkImage current snapshot: true andQuit: true]"
> to "SmalltalkImage current snapshot: false andQuit: true]" in the
> method below.
>
> SSImageInit class>>startSocketMonitorProcess
> 	"Socket leak monitor process. If the number of open file descriptors in the
> 	VM process gets dangerously close to the per-process limit (normally 1024),
> 	we are at risk of putting the image into a state in which it cannot accept
> 	connections, and that may not allow a clean recovery even after an image
> 	restart. Thus if the file descriptor count exceeds a threshold of 800, save
> 	the image and exit. Assume that a supervisory script will detect the image
> 	exit and initiate a restart."
>
> 	| vmFileCount |
> 	self stopSocketLeakMonitorProcess.
> 	SocketLeakMonitorProcess :=
> 	[[vmFileCount := (FileDirectory on: '/proc/', OSProcess thisOSProcess pid asString, '/fd') entries size.
> 	"OSProcess trace: DateAndTime now asString, ' squeakvm has ', vmFileCount asString, ' open file descriptors'."
> 	vmFileCount > 800 ifTrue: [
> 		OSProcess trace: 'Too many open file handles, save image and exit'.
> 		"Save the image, exit and wait for the supervisory script to restart"
> 		WorldState addDeferredUIMessage: [SmalltalkImage current snapshot: true andQuit: true]].
> 	(Delay forSeconds: 3 * 3600) wait] repeat] fork name: 'the Socket leak monitor'.
>
> Dave
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20221007/ca144dea/attachment.html>