<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">I got off track, I was thinking about persistence….<br><br><div dir="ltr"><div><div id="protonmail_signature_block" class="protonmail_signature_block" style="outline: none;"><div style="outline: none;"><div style="outline: none;"><br></div></div></div><div id="protonmail_mobile_signature_block" style="outline: none; min-height: 40px;"><div style="outline: none;"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div><div style="outline: none;"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div></div></div></div><div dir="ltr"><br><blockquote type="cite">On Oct 7, 2022, at 11:45, rabbit <rabbit@callistohouse.org> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><p>Great! I too look forward to hearing Tips and Guidance Welcome! I
totally and wholeheartedly agree! Welcome!<br></p></div></blockquote><div><br></div><div>+1!!!</div><blockquote type="cite"><div dir="ltr"><p>
</p>
<p>I started thinking about persistence. Kafka would excel at this,
fault tolerant, replicated and durable. I am unsure whether a
directory of files could be used as partition
persistence...perhaps the Kafka events sector block written are
links that could be used by an Installer to install a package in
the squeaksource directory structure. <br></p></div></blockquote>Hadoop durable replicated partitions are spread across replicated brokers. I was thinking we would replicate the actual SS directory contents of all projects/packages into Hadoop. The SS brokers could be replicated and monitored for restart or replica switch. In this way I’m thinking of replicating Kafka Consumers talking to Hadoop, writes and reads of MCZ & MCM files. This for good FD distribution as multiple web servers could also go wide. <div><br></div><div>Go wide with connections, consumers and persistence.</div><div><br></div><div><br><blockquote type="cite"><div dir="ltr"><p>
</p>
<p><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Capabilities-sending style.</blockquote></blockquote></blockquote></blockquote></p></div></blockquote><div>This has zero to do with Kafka replication. It’s properly a semantics aspect of the core local promises…</div><div><br></div><div> | ssDir sssEventual ssCryptoEventual ssPromisesEventual |</div><div> “In the machine serving ss files from it’s“ </div><div> “projectDirectory/packageFile structure…”</div><div> ssDir := <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">SqueakSourceDirectory on: ‘~/tribe/ssProjectDirectories/‘.</span></div><div> sssEventual := (SqueakSourceServer directory: ssDir) eventual.</div><div> </div><div> <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">“In a remote Vat, given the above server behind a web interface”</span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"> “Get a remote eventual for the”</span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"> “SqueakSourceServer and ask for projects“</span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"> “from it and send retrieve: or publish:“</span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"> “messages remotely, to the projects.”</span></div><div><br></div><div> (ssCryptoEventual := (RemoteServerInstaller on: ‘https://SqueakSource.com/squeakSourceServer/‘)</div><div> project: ‘Cryptography’)</div><div> then: [:project | project install: ‘RabbitZCryptoKitchenSink’]</div><div> then: [:project | self announce: <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">‘</span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">RabbitZCryptoKitchenSink installed!’]..</span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"> “Using #then:then: chaining convenience method.”</span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"> “As <</span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"> then: then1 then: then2</span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br></span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"> ^ (receiver then: then1) </span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"> then: then2</span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"> ></span></div><div><br></div><div><font color="#000000"><span style="caret-color: rgb(0, 0, 0);">(s</span></font>sPromisesEventual := (RemoteServerInstaller on: ‘https://SqueakSource.com/squeakSourceServer/‘)</div><div> project: ‘Promises’)</div><div> then: [:project | project allPackages]</div><div> then: [:packages | self announce: <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">‘Packages in Promises = ‘, packages asString</span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">’]..</span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br></span></div><div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Or something similar. Do you feel me?</span></div><blockquote type="cite"><div dir="ltr">
<p>It's all going to take a minute! ^,^<br>
</p>
<pre class="moz-quote-pre" wrap="">Tips and guidance welcome.</pre>
<div class="moz-signature">
<div>Have a good one; keep it, light.</div>
<div>Kindly,</div>
<div>rabbit</div>
<div>. .. … ‘…^,^</div>
<br>
<div>Sent from Callisto House :: decentralized mobile homeless
solutions</div>
<br>
<br>
</div>
<div class="moz-cite-prefix">On 10/7/22 11:06, David T. Lewis wrote:<br>
</div>
<blockquote type="cite" cite="mid:20221007150643.GA77986@shell.msen.com">
<pre class="moz-quote-pre" wrap="">On Fri, Oct 07, 2022 at 10:09:50AM -0400, rabbit wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hi Levente,
Is this the same machine running SqueakSource? I get frequent intermittent
failures with accessing projects there (SqueakSource.com). If it were also
running out of FDs and at 100% CPU, that would explain it. It???s frustrating
me as I install a bunch of packages off SS, and it fails in the middle often.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">No, the <a class="moz-txt-link-freetext" href="https://squeaksource.com">https://squeaksource.com</a> and <a class="moz-txt-link-freetext" href="https://source.squeak.org">https://source.squeak.org</a> services
are running on different machines.
I suspect (but do know know for sure) that the intermittent issues
on squeaksource.com are related to the server image exporting its
repository data on data.obj file. The code that does this has changed
in recent years and to be honest I don't know how it works. Specifically,
I do not know what actually triggers the repeated "save" operations
that seem to be causing problems. Tips and guidance welcome.
With respect to exhausing file descriptors on source.squeak.org, I
added a monitor process on squeaksource.com (the other service) that
automatically restarts the image if it is getting too low on file
descriptors. We could probably adapt this to work on source.squeak.org
also. The code is in <a class="moz-txt-link-freetext" href="http://www.squeaksource.com/SSImageInit">http://www.squeaksource.com/SSImageInit</a> and I
can work with Chris on it if the file descriptor issue turns out
to be a recurring problem.
Here is how the file descriptor monitor works on squeaksource.com.
It would need to be slightly modified for source.squeak.org because
on that server we never want to save the image file, so probably
just change "SmalltalkImage current snapshot: true andQuit: true]"
to "SmalltalkImage current snapshot: false andQuit: true]" in the
method below.
SSImageInit class>>startSocketMonitorProcess
"Socket leak monitor process. If the number of open file descriptors in the
VM process gets dangerously close to the per-process limit (normally 1024),
we are at risk of putting the image into a state in which it cannot accept
connections, and that may not allow a clean recovery even after an image
restart. Thus if the file descriptor count exceeds a threshold of 800, save
the image and exit. Assume that a supervisory script will detect the image
exit and initiate a restart."
| vmFileCount |
self stopSocketLeakMonitorProcess.
SocketLeakMonitorProcess :=
[[vmFileCount := (FileDirectory on: '/proc/', OSProcess thisOSProcess pid asString, '/fd') entries size.
"OSProcess trace: DateAndTime now asString, ' squeakvm has ', vmFileCount asString, ' open file descriptors'."
vmFileCount > 800 ifTrue: [
OSProcess trace: 'Too many open file handles, save image and exit'.
"Save the image, exit and wait for the supervisory script to restart"
WorldState addDeferredUIMessage: [SmalltalkImage current snapshot: true andQuit: true]].
(Delay forSeconds: 3 * 3600) wait] repeat] fork name: 'the Socket leak monitor'.
Dave
</pre>
</blockquote>
<span></span><br></div></blockquote><br><div><img alt="image0.jpeg" src="cid:85F401F1-92A4-4B0D-9ED2-2A79D2351A06"></div><div><br></div><div><img alt="image1.jpeg" src="cid:1DC4BED4-C345-43C1-A087-D8E51CB6BEA0-L0-001"><br></div></div></body></html>