I got off track, I was thinking about persistence….





> On Oct 7, 2022, at 11:45, rabbit <rabbit at callistohouse.org> wrote:
> 
> Great! I too look forward to hearing Tips and Guidance Welcome! I totally and wholeheartedly agree! Welcome!
> 

+1!!!
> I started thinking about persistence. Kafka would excel at this, fault tolerant, replicated and durable. I am unsure whether a directory of files could be used as partition persistence...perhaps the Kafka events sector block written are links that could be used by an Installer to install a package in the squeaksource directory structure. 
> 
Hadoop durable replicated partitions are spread across replicated brokers. I was thinking we would replicate the actual SS directory contents of all projects/packages into Hadoop. The SS brokers could be replicated and monitored for restart or replica switch. In this way I’m thinking of replicating Kafka Consumers talking to Hadoop, writes and reads of MCZ & MCM  files. This for good FD distribution as multiple web servers could also go wide. 

Go wide with connections, consumers and persistence.


>>>>> Capabilities-sending style.
This has zero to do with Kafka replication. It’s properly a semantics aspect of the core local promises…

  | ssDir sssEventual ssCryptoEventual ssPromisesEventual |
  “In the machine serving ss files from it’s“ 
  “projectDirectory/packageFile structure…”
  ssDir := SqueakSourceDirectory on: ‘~/tribe/ssProjectDirectories/‘.
  sssEventual := (SqueakSourceServer directory: ssDir) eventual.
 
  “In a remote Vat, given the above server behind a web interface”
  “Get a remote eventual for the”
  “SqueakSourceServer and ask for projects“
  “from it and send retrieve: or publish:“
  “messages remotely, to the projects.”

  (ssCryptoEventual := (RemoteServerInstaller on: ‘https://SqueakSource.com/squeakSourceServer/‘)
      project: ‘Cryptography’)
          then: [:project | project install: ‘RabbitZCryptoKitchenSink’]
          then: [:project | self announce: ‘RabbitZCryptoKitchenSink installed!’]..
          “Using #then:then: chaining convenience method.”
          “As <
              then: then1 then: then2

                  ^ (receiver then: then1) 
                      then: then2
          >

(ssPromisesEventual := (RemoteServerInstaller on: ‘https://SqueakSource.com/squeakSourceServer/‘)
      project: ‘Promises’)
          then: [:project | project allPackages]
          then: [:packages | self announce: ‘Packages in Promises = ‘, packages asString’]..

Or something similar. Do you feel me?
> It's all going to take a minute! ^,^
> 
> Tips and guidance welcome.
> Have a good one; keep it, light.
> Kindly,
> rabbit
> . .. … ‘…^,^
> 
> Sent from Callisto House :: decentralized mobile homeless solutions
> 
> 
>> On 10/7/22 11:06, David T. Lewis wrote:
>>> On Fri, Oct 07, 2022 at 10:09:50AM -0400, rabbit wrote:
>>> Hi Levente,
>>> 
>>> Is this the same machine running SqueakSource? I get frequent intermittent
>>> failures with accessing projects there (SqueakSource.com). If it were also
>>> running out of FDs and at 100% CPU, that would explain it. It???s frustrating
>>> me as I install a bunch of packages off SS, and it fails in the middle often.
>>> 
>> No, the https://squeaksource.com and https://source.squeak.org services
>> are running on different machines.
>> 
>> I suspect (but do know know for sure) that the intermittent issues
>> on squeaksource.com are related to the server image exporting its
>> repository data on data.obj file. The code that does this has changed
>> in recent years and to be honest I don't know how it works. Specifically,
>> I do not know what actually triggers the repeated "save" operations
>> that seem to be causing problems. Tips and guidance welcome.
>> 
>> With respect to exhausing file descriptors on source.squeak.org, I
>> added a monitor process on squeaksource.com (the other service) that
>> automatically restarts the image if it is getting too low on file
>> descriptors. We could probably adapt this to work on source.squeak.org
>> also. The code is in http://www.squeaksource.com/SSImageInit and I
>> can work with Chris on it if the file descriptor issue turns out
>> to be a recurring problem.
>> 
>> Here is how the file descriptor monitor works on squeaksource.com.
>> It would need to be slightly modified for source.squeak.org because
>> on that server we never want to save the image file, so probably
>> just change "SmalltalkImage current snapshot: true andQuit: true]"
>> to "SmalltalkImage current snapshot: false andQuit: true]" in the
>> method below.
>> 
>> SSImageInit class>>startSocketMonitorProcess
>> 	"Socket leak monitor process. If the number of open file descriptors in the
>> 	VM process gets dangerously close to the per-process limit (normally 1024),
>> 	we are at risk of putting the image into a state in which it cannot accept
>> 	connections, and that may not allow a clean recovery even after an image
>> 	restart. Thus if the file descriptor count exceeds a threshold of 800, save
>> 	the image and exit. Assume that a supervisory script will detect the image
>> 	exit and initiate a restart."
>> 
>> 	| vmFileCount |
>> 	self stopSocketLeakMonitorProcess.
>> 	SocketLeakMonitorProcess := 
>> 	[[vmFileCount := (FileDirectory on: '/proc/', OSProcess thisOSProcess pid asString, '/fd') entries size.
>> 	"OSProcess trace: DateAndTime now asString, ' squeakvm has ', vmFileCount asString, ' open file descriptors'."
>> 	vmFileCount > 800 ifTrue: [
>> 		OSProcess trace: 'Too many open file handles, save image and exit'.
>> 		"Save the image, exit and wait for the supervisory script to restart"
>> 		WorldState addDeferredUIMessage: [SmalltalkImage current snapshot: true andQuit: true]].
>> 	(Delay forSeconds: 3 * 3600) wait] repeat] fork name: 'the Socket leak monitor'.
>> 
>> Dave
>> 
>> 
> 




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20221007/006bc559/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image0.jpeg
Type: image/jpeg
Size: 16094 bytes
Desc: not available
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20221007/006bc559/attachment-0002.jpeg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image1.jpeg
Type: image/jpeg
Size: 247097 bytes
Desc: not available
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20221007/006bc559/attachment-0003.jpeg>