Indexing

[Chris Muller]

> Btw, I am slightly confused about this. There is
> MagmaServerLoader,
> MagmaTesterLoader and MagmaClientLoader. I presume
> these three MCs use
> MC deps to refer to the latest of all components
> they consist of. I
> understand that server is a superset of client etc.
> I guess that using
> any of these would give me Magma 1.1 - or rather -
> the latest of all
> packages, right?

Yep.  MagmaClientLoader has just the client packages,
used to ONLY connect to a *remote* server. 
MagmaServerLoader includes all the packages in client
plus some extras for the server.  These are needed to
either host a server or connect using #openLocal:. 
MagmaTesterLoader includes all in server (and client)
plus a bunch extra for the test cases.

> And then I presume that Magma1.0 is an MC referring
> to a frozen set of
> older snapshots, but does it correspond to
> MagmaServerLoader or
> MagmaClientLoader or ... what?

Better to think of them not as "older" (because
they're actually newer with these patches this week)
but as "minus the security code".

Rather than create three separate Loader packages for
1.0, I just created one that includes everything (a la
"Magma1.0TesterLoader").  The premise is that soon 1.1
will be the best one to use.  With these last two
fix-updates to 1.0, it is now branched from 1.1 on
SqueakSource.  I'm not planning to release another 1.1
for a few weeks yet.

So this is another reason to stay with 1.0 for now.  I
have merged the fixes into my own local 1.1 but not
planning to commit it to SqueakSource yet until I'm
done with this iteration.

> Anyway, I just loaded Magma1.0-cmm.4 and my app
> still works. :)
> 
> (Cees and others are now using Monticello
> Configurations, perhaps that
> is an option for you too - a config is just a list
> of specific
> snapshots)

I have no preference either way other than I really
don't want to have a SqueakSource server running right
now just to use MC-Configs..  When they support
File-based repositories I'll check them out again.

> Ok, so in my app I have MagmaCollections in three
> different places and
> given the number of instances of my domain objects
> at the moment I
> should have this number of MagmaCollections:
> 
> 	(Q2Model allInstances size * 2) + (Q2Process
> allInstances size) ==> 9
> 
> Note: Q2Model has two MagmaCollection instvars and
> Q2Process one.
> 
> This gives me 9 right now. MagmaCollection
> allInstances size gives me
> 378! And MagmaCollectionChanges allInstances size
> gives 379.

The next time this happens, see how many instances of
MagmaSession you have.  Remember, they all have their
own copy of all the MagmaCollections and changes.

There have been intermittent issues with cleanup of
old sessions over the years, it may be back..  It was
always related to Block/Method contexts holding old
Sessions in one of their (temp-var?) references.. 
There is a utility method, MagmaSession
class>>#cleanUp which enumerates all instances of
these contexts does a fine job of getting rid of the
ones; print-it to see the before/after instance count.


> Hmmmmm, ok - now I cleaned out my Magma db directory
> (it had tons of
> older MagmaCollection files - indexes that is -
> around 370-ish). Now it
> looks much better. 

Now this confuses me.  "Cleaning up" the the directory
files alone should have no effect on the number of
instances in the image..  ??

> I still have "twice too many"
> MagmaCollections in my
> image though:
> ...
> That is the expected
> number given a single
> MagmaSession. The second MagmaSession seems to be an
> extra internal
> session used by Magma (right?) and perhaps that
> session is for some
> reason also materializing the collections - which
> would explain the
> double amount (18 instead of 9).

Exactly right.  Magma has a meta-model that is
maintained via its own transaction mechanism.  The
meta-model includes such things as the
class-definitions, the magma-collections and their
indexes, the code-base for the repository, etc.  See
MagmaRepositoryDefinition.  It is the root of the
"meta side".

When a new class-definition or large-collection is
added, the server refreshes its own "internal" session
because it must know about them to do its work
properly.

> > No, this does not sound correct.  All
> LargeCollections
> > should be monitored as soon as they're persistent.
>  If
> > they're not persistent, changing keys doesn't
> matter. 
> > But again, I'm talking codeless here..
> 
> So a MagmaSession always "knows" all
> MagmaCollections in a db,
> regardless of if they have been navigated and
> materialized in the
> session yet?

Since all the MagmaCollections are part of the
MagmaRepositoryDefinition (the meta root), and this
definition is faulted down and materialized upon
connect, the answer is yes, each connected
MagmaSession always knows all MagmaCollections in a
db.

> Btw, in my app you can actually have the server
> "build" a separate Magma
> db, then download it, unzip and reconnect to it on
> the clients locally -

Wow, you can tell me more about this?  This is
obviously part of the "working offline" function,
right?

This might be painful if you are planning to try to
"merge" the offline work back into the "master" later.

I have planned, for 1.2, an efficient server-to-server
protocol that will allow large chunks of domains to be
transported between repositories without having to go
through the client; and, further, to be able to "sync"
up with the original repository.  I hope to have this
done by summer.

> so it was nice to see that you are very careful with
> predicting "odd"
> scenarios - because I then stumbled onto this
> exception (perfectly
> correctly - because I needed to reconnect etc)
> signalled in
> MagmaSession>>validateRemoteId   :
> 
> 	"Cannot connect because the repository has been
> replaced."
> 
> :)

I never imagined anyone would run into that condition
so soon.  :)  Glad you are putting it through some
good paces.

So I gather you discovered you just need to connect
with a new MagmaSession instance instead of trying to
reuse the old one.

> Hmmm, let me see now... above you are saying (I
> guess) that only
> monitored MagmaCollections will be reindexed. And
> AFAICT from the code
> the monitored collections are the ones we have
> materialized in the
> session. But above you wrote "All LargeCollections
> should be monitored
> as soon as they're persistent." which seems
> contradictory.

This question is hopefully answered now (above).  All
MagmaCollections in the db are monitored as soon as
you connect because they're part of the meta
RepositoryDef.  All newly craeted ones since the
connect are monitored as soon as they become
persistent via your commit.  Non-persistent
collections with indices do not suffer from key-change
side-effects.

> PS. Very happy with Magma so far. :) And yes, the
> second demo the other
> day went fine and we have a GO for the project! And
> most likely we will
> open source it too.

Fantastic!  Someday I hope my Java-Oracle cohorts will
at least *listen* to an alternative for five-minutes
without smirk and ridicule (about which they know
NOTHING).  In the meantime, we spend hours and
hundreds of e-mails every day toiling over
column-lengths, types, slow-BLOBs and CLOBs,
constraint order, naming-abbreviation "standards", DBA
fights, etc. etc.  Blecch!

 - Chris