Magma-Pier documentation

[Keith Hodges]

Magma-Pier documentation
---------------------------------------------

Magma provides a virtually transparent persistency mechanism, in which 
the application, in this case Pier, need hardly be aware of the fact 
that not all of the objects which it uses are kept in memory all of the 
time. As far as Pier is concerned Magma will stub out objects in memory, 
and load them in, a process termed reifying, or reification, as needed, 
entirely transparently.

Pier provides very controlled access to its model, including awareness 
of persistency issues, so it makes sense to make use of this in the 
Pier-Magma integration.

However, Pier will have to curb its tendency to traverse its entire 
model for some of its operations (e.g. page removal) in order to play 
sensibly and scalably in a Pier-Magma installation.

The Database Structure
===============
The root of the Magma database is an instance of PRMagmaRepository. This 
is a minimal root object designed with the intention of being able to 
merge with root objects from other projects. For example GJallar will 
have a root object of its own. To adopt or merge with Pier's 
requirements, all that is needed is for their root object to have an 
accessor #pier, onto a standard Dictionary.

PRMagmaRepository is a subclass of PRObject, and so has properties by 
default, if you can think of a reason or need to use them they are there 
but they are not currently used.

When running Seaside-Magma-Pier, the aforementioned Dictionary in the 
root of the database is directly available from the Seaside session via 
the convenience method #pierPersistencies.  (in the latest version, 
previously it was just #pier)

"session pierPersistencies", yields a dictionary of, PRMagmaPersistency 
instances, keyed by PRKernel name, so to obtain the PRKernel instance of 
your choice:

(session pierPersistencies at: 'MyPier') kernel

This does the job fairly logically.

ReadStrategies in Magma-Seaside
=====================
The root object of the repository also provides the Magma read strategy 
for the repository via #buildReadStrategyOn: . A ReadStrategy tells 
Magma how many objects to load in simultaneously.

For example, in Pier we tell Magma to reify all of the objects in a 
PRDocument structure in one database access/hit, since we will need them 
all to display or process the page.  For more details about the Pier 
read strategy see below.

This functionality, whereby the root object supplies the database 
read-strategy, is provided in WAReadStrategyDefault as a standard way of 
providing a read strategy for seaside applications in the package 
Magma-Seaside.

There is an alternative WAMagmaReadStrategyWithLogging which provides 
some logging features, and this can be selected in the application 
configuration by over-riding the 'read strategy' setting. Logging is 
invaluable in fine-tuning Magma read strategies.

Magma and Seaside Sessions
==================
Since Pier manages its own model locking, it is perfectly ok to share a 
single Magma session between all seaside sessions. This functionality is 
provided by WAMagmaSharedSession.

Other Seaside applications (Gjallar again!) have more complex 
arrangements using session-pools, but the basic WAMagmaSharedSession 
provided in the Magma-Seaside package is fine for Pier.

When running using WAMagmaSharedSession a single Magma session is opened 
for each application (#obtainMagmaSession), and is available from the 
seaside session via #magmaSession. The root of the database from the 
magmaSession obtained via #root.

The Pier-Magma package defines the convenience method #pierPersistencies 
as already mentioned.

Pier-Magma Integration
===============

The model locking performed by Pier is handled by the persistency 
scheme, PRPersistency and its subclasses. The default #execute: method is:

PRPersistency-i-#execute: aCommand
    self mutex critical: [ self apply: aCommand ].

The change required to support Magma is minimal:

PRMagmaPersistency-i-#execute: aCommand
    self mutex critical: [ self session commit: [ self apply: aCommand ] ]
   
Thats it!

What is In memory vs What is in the DB
=========================
When using PRMagmaPersistency, the master list of PRKernel instances, 
kept in PRKernel-instances, contains a 'stub'. This is an instance of 
PRKernel, having #persistency as an instance of PRMagmaPersistency (also 
a stub), but no data, i.e. #root is nil.

Whenever Pier wishes to obtain the real object, which is in the 
database, it calls, #realize on the stub, which calls #realizeKernel on 
its (stub) PRMagmaPersistency, returning the real PRKernel object, with 
a non-nil root, from the database. The real PRKernel object, also has a 
real #persistency, which is what is used to perform commands and logging 
etc.

[The model locking mutex which is used at present is the one in the 
persisted PRMagmaPersistency, if this causes problems, then it will be 
possible to use the mutex in the  memory resident (stub) 
PRMagmaPersistency instead].

Behind The Scenes - Saving The Model
========================
Before persisting some objects, Magma calls #maPreserialize.
[ I am warned that this approach may be deprecated in the future! -- kph ].

This is used:
1) to remove the mutex from PRPersistency
(Disabled at present. The mutex used to be in PRKernel and was lazily 
initialized, so removing it before persisting was fine, it would just be 
reinstated when next used.
If it becomes necessary to re-enable this, then we will have to see 
about asking Lukas to lazily initialize the mutex once more)
2) To remove any decorations of PRChildren if they are empty.  The 
current model implementation should not be creating empty PRChildren 
decorations anyway, but this is just in case.
3) Context's are tidied up before they are persisted. Properties are 
removed, and commands are copied, and command's answers are copied. This 
is to support logging.

Behind the scenes - loading the model
========================

This is the read strategy that Magma-Pier uses at present:

strategy minimumDepth: 0;
    "forVariableNamed: 'pier' onAny: PRMagmaRepository readToDepth: 2 ;"
        forVariableNamed: 'kernel' onAny: PRPersistency readToDepth: 1;
        onAny: PRKernel readToDepth: 1 ;
        forVariableNamed: 'decorations' onAny: PRDecorated readToDepth: 1;
        forVariableNamed: 'title' onAny: PRStructure readToDepth: 1;

        forVariableNamed: 'children' onAny: PRChildren readToDepth:
            (self readPathSkeletonInOneHit ifTrue: [ 1 ] ifFalse: [ 0 ]);

        onAny: PRDocumentItem readToDepth: 1; "reads all page items in 
one hit"

        onAny: PRComponent readToDepth: 1;
yourself.

If the flag #readPathSkeletonInOneHit is set, Magma, realizes all of the 
tree of objects necessary to resolve a url-path, in one database access 
cycle/hit. Depending upon other settings, this data may remain in 
memory. Otherwise, resolving a url-path occurs one level at a time.

As you would expect the items required to display a page, are loaded in 
one hit.

The ideal situation would be to have a path lookup cache which can 
perform the path look-up in one hit, and possibly obtain the whole page 
data in the same hit. This should be relatively straight forward, using 
an indexed MagmaCollection.

Logging
=====
The logging implementation used in PRImagePersistency has changed since 
the Pier-Magma integration was written, and so it follows the previous 
implementation.

It stores a snapshot of the original data-structure, together with a 
collection of the context/commands that are applied to it.

(it probably isnt working as intended at the moment, review required)

======
feedback comments to keith_hodges at yahoo.co.uk