Indexing

[goran at krampe.se]

Hi Chris!

Been hectic days here, haven't had time to followup. But here goes:

Chris Muller <chris at funkyobjects.org> wrote:
> Howdy Göran!
> 
> What version of Magma are you using?

1.0

Btw, I am slightly confused about this. There is MagmaServerLoader,
MagmaTesterLoader and MagmaClientLoader. I presume these three MCs use
MC deps to refer to the latest of all components they consist of. I
understand that server is a superset of client etc. I guess that using
any of these would give me Magma 1.1 - or rather - the latest of all
packages, right?

And then I presume that Magma1.0 is an MC referring to a frozen set of
older snapshots, but does it correspond to MagmaServerLoader or
MagmaClientLoader or ... what?

Anyway, I just loaded Magma1.0-cmm.4 and my app still works. :)

(Cees and others are now using Monticello Configurations, perhaps that
is an option for you too - a config is just a list of specific
snapshots)

> > AFAICT MaTransaction>>markRead:using: calls
> > monitorLargeCollectionChanges:, but the WeakSet in
> > MaTransaction just
> > keeps growing, I assume it uses identity instead but
> > perhaps should use
> > equality?. Looks like a bug, anyway that is not my
> > issue here...
> 
> I'm trying to understand this without having access to
> Squeak right now..  not remembering what WeakSet you
> may be talking about; I know the MaTransaction has a
> 'readSet' which is a WeakIdentityKeyDictionary but
> what is the name of the variable referencing the
> WeakSet?  I don't remember, I'm afraid I'll have to
> wait until this weekend to comment, sorry.

The WeakSet I am referring to is largeCollectionChanges.

In markRead:using: it says at the end:

...
anObject maIsLargeCollection 
		ifTrue:
			[ self monitorLargeCollectionChanges: anObject changes.
			anObject session: session ].
^anObject

And in MaTransaction>>monitorLargeCollectionChanges: we have:

monitorLargeCollectionChanges: aMaLargeCollectionChanges

	largeCollectionChanges add: aMaLargeCollectionChanges


Ok, so in my app I have MagmaCollections in three different places and
given the number of instances of my domain objects at the moment I
should have this number of MagmaCollections:

	(Q2Model allInstances size * 2) + (Q2Process allInstances size) ==> 9

Note: Q2Model has two MagmaCollection instvars and Q2Process one.

This gives me 9 right now. MagmaCollection allInstances size gives me
378! And MagmaCollectionChanges allInstances size gives 379.

Hmmmmm, ok - now I cleaned out my Magma db directory (it had tons of
older MagmaCollection files - indexes that is - around 370-ish). Now it
looks much better. I still have "twice too many" MagmaCollections in my
image though:

{(Q2Model allInstances size * 2) + (Q2Process allInstances size).
MagmaSession allInstances size.
MagmaCollection allInstances size.
MagmaCollectionChanges allInstances size}
 ===> #(9 2 18 19)

Q2Model allInstances size is 1 - which is my domain model root object.
It has two MagmaCollections. Then I have Q2Process - 7 instances with
one MagmaCollection each. That is the expected number given a single
MagmaSession. The second MagmaSession seems to be an extra internal
session used by Magma (right?) and perhaps that session is for some
reason also materializing the collections - which would explain the
double amount (18 instead of 9). And yes, I have files on disk
indicating 9 MagmaCollections (there are 9 unique numbers used in the
.hdx filenames). 

Anyway, this looks "reasonable" and funny enough my experience that
spurred this email (the index not being updated) seems to have magically
disappeared. It might have been related to these older index files
laying around? Odd.

> > So the code in captureOldHashesFor: seems to only
> > capture the hashes for
> > the large collections that are "monitored", but in
> > my case this happens
> > to only include the collection I navigated through
> > (since I then read
> > it), but not the other! Is this logic really
> > correct? Or do I need to
> > "trick" the session into reindexing my other
> > collection too somehow?
> 
> No, this does not sound correct.  All LargeCollections
> should be monitored as soon as they're persistent.  If
> they're not persistent, changing keys doesn't matter. 
> But again, I'm talking codeless here..

So a MagmaSession always "knows" all MagmaCollections in a db,
regardless of if they have been navigated and materialized in the
session yet?

> Brent are you around?  Didn't we just fix a bug
> related to this recently?
> 
> Sorry you had this problem Göran.  I will investigate
> it this weekend and have an answer/fix for you.

No problem, I haven't had many issues at all with Magma so far - and in
this case it was probably because there were older files in the db dir
(my guess).

Btw, in my app you can actually have the server "build" a separate Magma
db, then download it, unzip and reconnect to it on the clients locally -
so it was nice to see that you are very careful with predicting "odd"
scenarios - because I then stumbled onto this exception (perfectly
correctly - because I needed to reconnect etc) signalled in
MagmaSession>>validateRemoteId   :

	"Cannot connect because the repository has been replaced."

:)

> > The current result seems to be that object A is only
> > reindexed in the
> > collection I navigated through.  I assume there is
> > currently no way for
> > Magma to know which collections it should re-index
> > on its own.
> 
> Nope, no way.  I tried real hard for Magma to detect
> this automatically but eventually concluded it was
> impossible without severely affecting performance.
> 
> I'll get back to you tomorrow or Sunday..  Be sure to
> tell me what version you're using.
> 
>  - Chris

Hmmm, let me see now... above you are saying (I guess) that only
monitored MagmaCollections will be reindexed. And AFAICT from the code
the monitored collections are the ones we have materialized in the
session. But above you wrote "All LargeCollections should be monitored
as soon as they're persistent." which seems contradictory.

I am at a loss right now. Right now my app seems to work nicely - but
perhaps that is just because I materialize all these collections in my
sessions right now.

regards, Göran

PS. Very happy with Magma so far. :) And yes, the second demo the other
day went fine and we have a GO for the project! And most likely we will
open source it too.