Hi Norberto, thanks for the script. With it, I was indeed able to reproduce this assertion "failure" in the new release 40 (which is what you are using, I hope):

self assert: MagmaSession allInstances size = 2. "it fails. There is an extra session with id=nil, user=nil. Is it normal? "

The reason I put "failure" in quotes is because it isn't a bug. The session with id=nil, user=nil is a session that was created during initialization but never used. It was de-referenced, and therefore eligible for garbage collection. If you run

Smalltalk garbageCollect

you will see it disappear. Of course, Magma asserts no control over when the garbage-collector runs.

Magma does a lot of "direct initialization" (as opposed to lazy-initialization), and this leads to the creation of this extra session object (which is never opened or connected) before replacing it with another one when you ask the repository to open. I agree it could be slightly less wasteful to not create that one extra session, and I have made a tweak to do this in r41Alpha. It really doesn't amount to much of an improvement but.. it is an improvement nonetheless, thanks.

But my sense is you are not that concerned about this, but more about this question:

I just wanted to know wether it could be somethig wrong in the files, not in the objects in memory for this malcfunction. Could be a unfinished transaction waiting in the files for return to live? Because I didn't created any extra transaction, just only one.

Each commit involves writes to multiple files. The very first is to write the serialized CommitPackage to the "commits.log" file, which secures the transaction in the commit log, followed by an immediate #flush of that file. This assures that, no matter what happens (power outage, etc.), the commit will be secured in the database.

Immediately after writing and flushing the CommitPackage to the commits.log file, Magma then "applies" the commit to the various randomly-accessed files of the repository. Because a single commit requires writing to multiple files, it is required that writes be accomplished atomically. In other words, if a commit requires writing to 5 files, it is completely unacceptable for three out of five writes to be completed and then a power-outage occurs, leaving the repository in an inconsistent state.

Therefore MaAtomicFileStream (which wraps StandardFileStreams) was developed to handle a series of writes across multiple files atomically. This is accomplished in three steps:

1) write a "begin entry" (with control-information) record to the "applied.images" file. 2) flush all of the proposed deltas to the files to the single "applied.images" file. 3) write the deltas to the real files directly. 4) write a "end entry" to the "applied.images" file.

This is a well-considered process that has been working for years. The test case for it is quite robust; #testRollbackRecovery, and covers every possible scenario including creation of new files.

However, the process is also somewhat burdensome. To maximize performance, rather than do this for every commit, Magma does it (by default) every five seconds. This is totally fine because MaAtomicFileStreams cache the writes to them that occur during those five seconds. Any reads which come in during those five seconds are answered directly from the cache transparently. And all the data is already secured in the commits.log file in case there is a crash of any kind.

After five seconds (again, this is adjustable), the server flushes all MaAtomicFileStreams to disk using the above-described 4-step process.

So, I hope this explains why you may see a (5 second) delay between the time you #commit and the time you see a change in the files in the filesystem.

Regards, Chris

On 9/21/07, Norberto Manzanos nmanzanos@gmail.com wrote:

It's very difficult to replicate the whole situation. I just wanted to know wether it could be somethig wrong in the files, not in the objects in memory for this malcfunction. Could be a unfinished transaction waiting in the files for return to live? Because I didn't created any extra transaction, just only one. I was testing wether the problem could be in a session that remain in memory. I wrote this test.

test00OpenSessions

| session1 session2 | MagmaSession disconnectAndCloseAllConnectedSessions. MagmaSession cleanUp. Smalltalk garbageCollect. self assert: MagmaSession allInstances isEmpty. session1:= MagmaSession openLocal: '\Squeak 3.9\magma'. session1 connectAs: 'User1' asString. self assert: MagmaSession allInstances size = 2. "it fails. There is an extra session with id=nil, user=nil. Is it normal? " session2:=MagmaSession openLocal: '\Squeak 3.9\magma'. session2 connectAs: 'User2' asString. self assert: MagmaSession allInstances size = 3. "idem"

Regards Norberto

On 9/13/07, Chris Muller asqueaker@gmail.com wrote:

...

Norberto, given the lack of specific information, I can only speculate from my gut that you were in a transaction more than one-level deep. In that case, all changes remain solely in memory until you execute the outermost #commit and you will not see any change in the repository files until then.

You said it wasn't the first time you've encountered a problem, so would you please retrace your steps, documenting them into a script I can follow that demonstrates a problem? I would be happy to be help.

Regards, Chris

On 9/13/07, Norberto Manzanos nmanzanos@gmail.com wrote:

...

Hi Chris and people.

Something really strange has just happened with Magma.

I had a repository. One of its collections had about 11000 objects. I made a massive adding of about 600 objects. 10 objects per transaction, refreshPersistentObjects... seting to false, readStrategy with depth 0. When the process finished, the size of the collection was about 11600. I noticed the files of the repository hadn't change. When I closed the session the collection size was 11000 again. Where were the other? I tried sending an #abort (but the whole process was inside transactions,so, what for?), then I sent a #cleanUp to MagmaSession (I don't known why, magic perhaps). Surprisingly, when I reconnected the session and requested the size again, it was 600!. I swear I didn't drink nothing but coffee, but when I closed and reconnected the session one more time, the collection had the 11000 of the beginning. The objects file hadn't changed and the size of the squeak image was about 150 Mb. So, they were allways in memory ... but where exactly? There was only one instance of MagmaRepositoryController and two instances of MagmaSession, one my session, and a __system session.

This is not the first time that many objects disapears in the air. What I really want to understand is what could be the reason that several #add: to the repository inside a transaction could not be actually saved in the files and keeps in memory.

Thanks in advance. Norberto _______________________________________________ Magma mailing list Magma@lists.squeakfoundation.org

http://lists.squeakfoundation.org/mailman/listinfo/magma

...

...

---

Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma