[Box-Admins] Re: [Board] Jens Lincke (from HPI) as new Trunk/Etoys developer

[Levente Uzonyi]

On Sat, 3 Sep 2016, Chris Muller wrote:

> It looks like its taking as long as 15 minutes to index a single VMMaker.oscog Version into Magma (see log output below).  With more than 3500
> Versions in VMMaker, my guess is this could take another few weeks!
> 
> So, I suggest that we go ahead and cut over to the new image using only SSFilesystem.  I will run a separate Squeak VM process to finish loading

If you do that, make sure you start the VM with

 	ionice -c3 nice squeak ...

ionice -c3 will make it only use the disk when no other processes use it. 
nice affects CPU usage in a similar way.

It would still take months to process everything. I'm not sure it would 
work out. Perhaps we'd better wait for the new server, which will probably 
have better CPUs. Or profile and optimize. :)

> the ss.magma database, when its done, we can switch back to the SSCompositeStorage, with SSFilesystem as the primary, and the SSMagma to provide
> the history.
> 
> This doesn't solve the problem of Ronie's PW not being reset in box4.squeak.org (unless, Bert, you did that).  However, it solves the problem of
> the system, when using a SSCompositeStorage, of needing to obtain the initial SSRepository instance upon startup from the SSMagma, which is
> currently still "behind", until it finally catches up.
> 
> How does that sound?

Sounds good to me.

> 
>  - Chris
> 
> 2016-09-03T17:59:20.018847+02:00 VMMaker.oscog-eem.711 of VMMaker --> committed to ss.magma
> 2016-09-03T18:04:31.069556+02:00 VMMaker-dtl.289 of VMMaker --> committed to ss.magma
> 2016-09-03T18:12:41.965925+02:00 VMMaker.oscog-eem.1519 of VMMaker --> committed to ss.magma
> 2016-09-03T18:13:25.429452+02:00 Cog-eem.288 of VMMaker --> committed to ss.magma
> 2016-09-03T18:27:22.797825+02:00 VMMaker-IgorStasenko.55 of VMMaker --> committed to ss.magma
> 2016-09-03T18:38:28.835509+02:00 VMMaker.oscog-eem.658 of VMMaker --> committed to ss.magma
> 2016-09-03T18:44:51.454197+02:00 VMMaker.oscog-eem.85 of VMMaker --> committed to ss.magma
> 2016-09-03T18:56:21.111055+02:00 VMMaker.oscog-eem.1193 of VMMaker --> committed to ss.magma
> 2016-09-03T19:02:42.47393+02:00 VMMaker.oscog-tpr.291 of VMMaker --> committed to ss.magma
> 2016-09-03T19:05:53.865053+02:00 VMMaker-tpr.53 of VMMaker --> committed to ss.magma
> 2016-09-03T19:21:28.711418+02:00 VMMaker.oscog-eem.149 of VMMaker --> committed to ss.magma
> ....

If I undestand these logs correctly, then it took almost 82 minutes to 
process 10 mczs. That's about 8 minutes and 12 seconds per mcz instead of 
15 minutes. But that's still quite a lot. "profile and optimize" :)

Levente

> 
> 
> On Fri, Sep 2, 2016 at 8:59 PM, Chris Muller <ma.chris.m at gmail.com> wrote:
>       After reconsidering, we should wait until the bulk load is complete
>       because, when using the new dual repository (one SSFilesystem and one
>       SSMagma) the code must choose the SSRepository loaded from Magma
>       instead of the one loaded from "data.obj" because that's the object
>       Magma monitors for changes.
>
>       Unfortunately its taking a long time to tear open the VMMaker
>       project's 17GB worth of MCZ's and examine every internal MCDefinition
>       (which, VMMaker has a lot in each one) to see if its already been
>       canonicalized.
>
>       If we deploy before its done then, for the reason above, some of the
>       Versions which have been committed since the bulk load began will not
>       be seen until the bulk load finishes.
>
>       I wish I would have put some kind progress indication somewhere; it
>       only logs the filenames its working on at the time.
>
>       On Fri, Sep 2, 2016 at 5:47 PM, Levente Uzonyi <leves at caesar.elte.hu> wrote:
>       > On Fri, 2 Sep 2016, Chris Muller wrote:
>       >
>       >> I wasn't planning to.  The metadata for versions will be captured via
>       >> the recovery process.  The other parts of the model, not, but
>       >> thankfully they don't change much.
>       >>
>       >> As far as the image and daemontools run system, I think we're ready to
>       >> switch over any time, however, the only thing I'm not sure about are
>       >> how the community emails work.  I think I saw the packages email
>       >> address in the SSRepository model, so maybe it will "just work"?  Do
>       >> either you know whether any special server configurations are needed
>       >> to be able to send SMTP mail?
>       >
>       >
>       > I suppose it's just a setting, so it may just work.
>       >
>       >>
>       >> On Fri, Sep 2, 2016 at 4:20 PM, Levente Uzonyi <leves at caesar.elte.hu>
>       >> wrote:
>       >>>
>       >>> Let's not forget that Ronie Salgado is still waiting for his password,
>       >>> which
>       >>> we should give him asap.
>       >>
>       >>
>       >> Hi Levente, I looked up the other email to refresh my memory --
>       >> SSEMailSubscription>>#writeHeaders is apparently where it writes the
>       >> "Date:" header, with this code:
>       >>
>       >> 1     stream nextPutAll: 'Date: ';
>       >> 2         nextPutAll: MailMessage dateStampNow;
>       >> 3          nextPut: Character space;
>       >> 4          nextPutAll: TimeZone default printHHMM;
>       >> 5          cr.
>       >>
>       >> So do we just need to remove line 4?  I can commit that change and
>       >> build a new image if that's the right thing to do..
>       >>
>       >
>       > No, the problem is with fractions in the seconds. And that should already be
>       > fixed in the new image.
>       >
>       > Levente
> 
> 
> 
>