Gosts objects

[Chris Muller]

Hi Norberto, thanks for the script.  With it, I was indeed able to
reproduce this assertion "failure" in the new release 40 (which is
what you are using, I hope):

> self assert: MagmaSession allInstances size = 2. "it fails. There is an
> extra session with id=nil, user=nil. Is it normal? "

The reason I put "failure" in quotes is because it isn't a bug.  The
session with id=nil, user=nil is a session that was created during
initialization but never used.  It was de-referenced, and therefore
eligible for garbage collection.  If you run

  Smalltalk garbageCollect

you will see it disappear.  Of course, Magma asserts no control over
when the garbage-collector runs.

Magma does a lot of "direct initialization" (as opposed to
lazy-initialization), and this leads to the creation of this extra
session object (which is never opened or connected) before replacing
it with another one when you ask the repository to open.  I agree it
could be slightly less wasteful to not create that one extra session,
and I have made a tweak to do this in r41Alpha.  It really doesn't
amount to much of an improvement but.. it is an improvement
nonetheless, thanks.

But my sense is you are not that concerned about this, but more about
this question:

> I just wanted to know
> wether it could be somethig wrong in the files, not in the
> objects in memory for this malcfunction. Could be a unfinished transaction
> waiting in the files for return to live? Because I didn't  created any extra
> transaction, just only one.

Each commit involves writes to multiple files.  The very first is to
write the serialized CommitPackage to the "commits.log" file, which
secures the transaction in the commit log, followed by an immediate
#flush of that file.  This assures that, no matter what happens (power
outage, etc.), the commit will be secured in the database.

Immediately after writing and flushing the CommitPackage to the
commits.log file, Magma then "applies" the commit to the various
randomly-accessed files of the repository.  Because a single commit
requires writing to multiple files, it is required that writes be
accomplished atomically.  In other words, if a commit requires writing
to 5 files, it is completely unacceptable for three out of five writes
to be completed and then a power-outage occurs, leaving the repository
in an inconsistent state.

Therefore MaAtomicFileStream (which wraps StandardFileStreams) was
developed to handle a series of writes across multiple files
atomically.  This is accomplished in three steps:

  1) write a "begin entry" (with control-information) record to the
"applied.images" file.
  2) flush all of the proposed deltas to the files to the single
"applied.images" file.
  3) write the deltas to the real files directly.
  4) write a "end entry" to the "applied.images" file.

This is a well-considered process that has been working for years.
The test case for it is quite robust; #testRollbackRecovery, and
covers every possible scenario including creation of new files.

However, the process is also somewhat burdensome.  To maximize
performance, rather than do this for every commit, Magma does it (by
default) every five seconds.  This is totally fine because
MaAtomicFileStreams cache the writes to them that occur during those
five seconds.  Any reads which come in during those five seconds are
answered directly from the cache transparently.  And all the data is
already secured in the commits.log file in case there is a crash of
any kind.

After five seconds (again, this is adjustable), the server flushes all
MaAtomicFileStreams to disk using the above-described 4-step process.

So, I hope this explains why you may see a (5 second) delay between
the time you #commit and the time you see a change in the files in the
filesystem.

Regards,
  Chris



On 9/21/07, Norberto Manzanos <nmanzanos at gmail.com> wrote:
> It's very difficult to replicate the whole situation.
> I just wanted to know
> wether it could be somethig wrong in the files, not in the
> objects in memory for this malcfunction. Could be a unfinished transaction
> waiting in the files for return to live? Because I didn't  created any extra
> transaction, just only one.
> I was testing wether the problem could be in a session that remain in
> memory.  I wrote this test.
>
> test00OpenSessions
>
> | session1 session2 |
> MagmaSession disconnectAndCloseAllConnectedSessions.
> MagmaSession cleanUp.
> Smalltalk garbageCollect.
> self assert: MagmaSession allInstances isEmpty.
> session1:=  MagmaSession openLocal: '\Squeak 3.9\magma'.
> session1 connectAs: 'User1' asString.
> self assert: MagmaSession allInstances size = 2. "it fails. There is an
> extra session with id=nil, user=nil. Is it normal? "
> session2:=MagmaSession openLocal:  '\Squeak 3.9\magma'.
> session2 connectAs: 'User2' asString.
> self assert: MagmaSession allInstances size = 3. "idem"
>
>
>
>
> Regards
> Norberto
>
>
> On 9/13/07, Chris Muller <asqueaker at gmail.com> wrote:
> > Norberto, given the lack of specific information, I can only speculate
> > from my gut that you were in a transaction more than one-level deep.
> > In that case, all changes remain solely in memory until you execute
> > the outermost #commit and you will not see any change in the
> > repository files until then.
> >
> > You said it wasn't the first time you've encountered a problem, so
> > would you please retrace your steps, documenting them into a script I
> > can follow that demonstrates a problem?  I would be happy to be help.
> >
> > Regards,
> >   Chris
> >
> >
> >
> > On 9/13/07, Norberto Manzanos <nmanzanos at gmail.com> wrote:
> > > Hi Chris and people.
> > >
> > > Something really strange has just happened with Magma.
> > >
> > > I had a repository. One of its collections had about 11000 objects.
> > > I made a massive adding of about 600 objects. 10 objects per
> > > transaction, refreshPersistentObjects... seting to false, readStrategy
> > > with depth 0.
> > > When the process finished, the size of the collection was about 11600.
> > > I noticed the files of the repository hadn't change. When I closed the
> > > session the collection size was 11000 again. Where were the other? I
> > > tried sending an #abort (but the whole process was inside
> > > transactions,so, what for?), then I sent a #cleanUp to MagmaSession (I
> > > don't known why, magic perhaps). Surprisingly, when I reconnected the
> > > session and requested the size again, it was 600!. I swear I didn't
> > > drink nothing but coffee, but when I closed and reconnected the
> > > session one more time, the collection had the 11000 of the beginning.
> > > The objects file hadn't changed and the size of the squeak image was
> > > about 150 Mb. So, they were allways in memory ... but where exactly?
> > > There was only one instance of MagmaRepositoryController and two
> > > instances of MagmaSession, one my session, and a __system session.
> > >
> > > This is not the first time that many objects disapears in the air.
> > > What I really want to understand is what could be the reason that
> > > several #add: to the repository inside a transaction could not be
> > > actually saved in the files and keeps in memory.
> > >
> > > Thanks in advance.
> > > Norberto
> > > _______________________________________________
> > > Magma mailing list
> > > Magma at lists.squeakfoundation.org
> > >
> http://lists.squeakfoundation.org/mailman/listinfo/magma
> > >
> >
>
>
> _______________________________________________
> Magma mailing list
> Magma at lists.squeakfoundation.org
> http://lists.squeakfoundation.org/mailman/listinfo/magma
>
>