Dear all,
I'm using Magma and finding out how transparent it can be. I'm trying Magma to see if the DDD principles can be applied to it. To retrieve, store and save aggegrate root objects a Repository object is used. A 'transient' Repository object is nothing more than a wrapper around an OrderedCollection, which contains store save and find methods. The client who is using this Repository object does not know which implementation is behind it. After retrieving the initial root aggegrate the client can manipulate the sub objects of the aggegrate root.
I managed to achieve Read Transparency. I specialised the Repository objects so they use a MagmaCollection instead of an ordinary OrderedCollection. You just do a findXXX method on the Repository, and that method is delegated to a MagmaCollection which does a read based on some defined index.
To do a Write Transparency, I added a decorator around the store / save methods of the Repository which automatically wraps the original Repository object in a session commit:[] block; enabling Magma to see the changes made to the 'saved' object and get them persisted. AddTransactionAroundMutators>>doesNotUnderstand: aMessage |result| (self isMutator: aMessage) ifTrue: [repositoriesController commit: [result := aMessage sendTo: decorated]] ifFalse: [result := aMessage sendTo: decorated]. ^result. In which AddTransactionAroundMutators is a decorator around the Repository object.
Persisting works, however I noticed that the hash indexes did not get updated this way. On the wiki it is mentioned that MagmaSession>>#noteOldKeysFor: should be called before any change to the object is made. This is off course, using my idiom, impossible to do, since the user using the Repository does not even know that there is a MagmaSession. It just knows the API of the Repository object. The Seaside GUI I'm using is manipulating an object (which it got from a findXXX query on a repository) directly and if it wants to save the changes it calls Repository>>save (which got decorated). The GUI using the Repositories does not know Magma and I tend to keep it that way.
After some searching, I managed to circumvent this by a dirty hack: AddTransactionAroundMutators>>doesNotUnderstand: aMessage |result| (self isMutator: aMessage) ifTrue: [ (self isUpdatingMutator: aMessage) "denotes saveXXX methods => updates to existing objects" ifTrue: [ |session session2 firstArg oidForFirstArg reReadObject| session := repositoriesController session. session2 := repositoriesController session2. firstArg := aMessage arguments first. oidForFirstArg := session oidFor: firstArg. reReadObject := session2 realObjectFor: oidForFirstArg. repositoriesController commit: [session noteOldKeysFor: firstArg usingHashesFromObject: reReadObject. result := aMessage sendTo: decorated]] ifFalse: [ repositoriesController commit: [result := aMessage sendTo: decorated]]] ifFalse: [result := aMessage sendTo: decorated]. ^result.
MagmaSession>>noteOldKeysFor: anObject usingHashesFromObject: anOtherObject guard critical: [ transaction ensureOldHashesCapturedFor: anObject usingHashesFromObject: anOtherObject]
MaTransaction>>ensureOldHashesCapturedFor: anObject usingHashesFromObject: anOtherObject (potentialKeysChange includesKey: anObject) ifTrue: [ ^ self ]. self captureOldHashesFor: anObject usingHashesFromObject: anOtherObject
MaTransaction>>captureOldHashesFor: anObject usingHashesFromObject: anOtherObject largeCollectionChanges do: [ :eachChanges | (eachChanges collection canIndex: anObject) ifTrue: [ eachChanges collection indexesDo: [ :eachIndex | | oldKeys oldHashes | oldKeys := potentialKeysChange at: anObject ifAbsentPut: [ IdentityDictionary new ]. oldHashes := oldKeys at: eachIndex attribute ifAbsentPut: [ OrderedCollection new ]. oldHashes add: { eachChanges. eachIndex. eachIndex indexHashesFor: anOtherObject } ] ] ].
So I'm using a second MagmaSession to reload the object I'm about to commit and I'm noting to the first session to noteOldKeysFor: anObject usingHashingFromObject: theJustReloadedObjectFromTheSecondSession. Like this, the first session gets the hashes of the unmodified object in the repository, and the hashes get updated correctly automatically. It is a working solution (at least with my very limited testing), but I find it very complicated. Surely there must be a cleaner way to do an automatic hashkey update.
Other things I already tried is to enable WriteBarrier, but could not get that working. I also tried to do readRealObject: oid on the same session; but this always returned a cached version, with the already updated fields in. The only thing that worked was to create a new session and make sure that the object is fresh.
Can anyone help me on this one? I'm finding it a bit strange that Magma can correctly save my changes, but not determine if a hash key update is necessary.
Thanks for any help.
Kind Regards,
Bart
Hi,
I'm using Magma and finding out how transparent it can be. I'm trying Magma to see if the DDD principles can be applied to it. To retrieve, store and save aggegrate root objects a Repository object is used. A 'transient' Repository object is nothing more than a wrapper around an OrderedCollection, which contains store save and find methods. The client who is using this Repository object does not know which implementation is behind it. After retrieving the initial root aggegrate the client can manipulate the sub objects of the aggegrate root.
I managed to achieve Read Transparency. I specialised the Repository objects so they use a MagmaCollection instead of an ordinary OrderedCollection.
Use of a MagmaCollection vs. an OrderedCollection should be based on application requirements. Either type of collection works equally well whether transient or persistent. I recommend just picking the best one for the needs of that particular collection and sticking with it.
To do a Write Transparency, I added a decorator around the store / save methods of the Repository which automatically wraps the original Repository object in a session commit:[] block; enabling Magma to see the changes made to the 'saved' object and get them persisted. AddTransactionAroundMutators>>doesNotUnderstand: aMessage |result| (self isMutator: aMessage) ifTrue: [repositoriesController commit: [result := aMessage sendTo: decorated]] ifFalse: [result := aMessage sendTo: decorated]. ^result.
Hmm, to me, this is not worth the complexity. If you want your domain to be in control of the transactions (as opposed to the user), I suggest signaling MagmaSessionRequests. See http://wiki.squeak.org/squeak/5605 if you haven't already.
So I'm using a second MagmaSession to reload the object I'm about to commit and I'm noting to the first session to noteOldKeysFor: anObject usingHashingFromObject: theJustReloadedObjectFromTheSecondSession. Like this, the first session gets the hashes of the unmodified object in the repository, and the hashes get updated correctly automatically. It is a working solution (at least with my very limited testing), but I find it very complicated. Surely there must be a cleaner way to do an automatic hashkey update.
Wow, please do not do this approach, I agree it is way too complicated and definitely not the right way to use Magma.
Please use MagmaSessionRequest (signalNoteOldKeysFor:, et al) and handle it in whatever app-level controller you're using. It is benign if it isn't handled, so transient applications will work transparently.
Other things I already tried is to enable WriteBarrier, but could not get that working.
WriteBarrier is a separate package which I don't believe works on Pharo. I do not maintain this package.
Can anyone help me on this one? I'm finding it a bit strange that Magma can correctly save my changes, but not determine if a hash key update is necessary.
Detecting key changes automatically was impossible without slowing down commits; the
MagmaSessionRequest signalNoteOldKeysFor: myAttribute
is so easy to just put into the appropriate setter, so it wasn't a good trade-off to compromise performance for this minor increase in transparency.
Regards, Chris
Hi,
Thanks for your reply.
Hmm, to me, this is not worth the complexity. If you want your domain
to be in control of the transactions (as opposed to the user), I suggest signaling MagmaSessionRequests. See http://wiki.squeak.org/squeak/5605 if you haven't already.
I read that page, but did not fully comprehend it. I will look into it.
Wow, please do not do this approach, I agree it is way too complicated
and definitely not the right way to use Magma.
Please use MagmaSessionRequest (signalNoteOldKeysFor:, et al) and handle it in whatever app-level controller you're using. It is benign if it isn't handled, so transient applications will work transparently.
I will give this a try. Did not know this existed.
Detecting key changes automatically was impossible without slowing down commits; the
MagmaSessionRequest signalNoteOldKeysFor: myAttribute
is so easy to just put into the appropriate setter, so it wasn't a good trade-off to compromise performance for this minor increase in transparency.
On: http://wiki.squeak.org/squeak/5605 you mention that option 3) autocommit is completely transparent. I disagree, since the signalNoteOldKeys message on a session is absolutely necessary to make it work correctly. I can understand your reasoning about the performance hit if you have to check everything.
I will try out option 2) because the other I suggested is just too complex. I still have my doubts about the transparency of this option. Out of experience I know that most code is modified after being written. What the developer now has to remember is that if an index is added to some magma collection is to add the signalNoteOldKeys: in the mutator of it. Thats error prone. That is the reason I tried to do that completely automatically. You set up it correct one time and move on, just don't have to think about it again.
The index I'm trying to create makes this more error prone: I have a root object which is in a MagmaCollection. It has a keywords index. The keywords are made up of a list which contains multiple fields of the root object but also of a list of fields of the sub objects of this root. Using this, I can implement a kind of google search box in which you can find keywords dispersed over my entire domain and let Magma retrieve the correct aggegrate root, and I would not have to integrate any kind of full text search technology to do this. The reason I wanted it to be transparent is that if I for instance add a new field on this keywords list, I now have to remember the signalNoteKeys on this setter, (which is error prone ...).
You say that it is slow. Did Magma once provide this auto notify keys functionality? I think I could maybe live with the bit slower performance ...
Please don't get me wrong. I think Magma is great. Just got to know it better. And if it turns out that it is not completely transparent, I shall happily adapt the somewhat less transparent solution.
Kind Regards,
Bart
On: http://wiki.squeak.org/squeak/5605 you mention that option 3) autocommit is completely transparent. I disagree, since the signalNoteOldKeys message on a session is absolutely necessary to make it work correctly. I can understand your reasoning about the performance hit if you have to check everything.
Yeah, I agree with you. I've removed that sentence.
I will try out option 2) because the other I suggested is just too complex. I still have my doubts about the transparency of this option. Out of experience I know that most code is modified after being written. What the developer now has to remember is that if an index is added to some magma collection is to add the signalNoteOldKeys: in the mutator of it.
For clarification, signalNoteOldKeys: need only be called when instVar an object will cause one of its indexed attributes to change. Nothing needs to be done when "an index is added to some magma collection."
Using the Smalltalk browser to browse "definitions" of a particular instVar, all changes to an indexed attribute is presented. Usually, attributes of an object are modified in just one or two places (like #initialize and a setter). So this is the only place you would need to put it, and only for indexed attributes.
Incidentally, there is a parallel "lack-of-transparency" in standard Smalltalk and even, if I am not mistaken, Java. If you were to use a standard Dictionary / HashTable to look up objects, when something causes an objects #hash / hashCode() to change, the Dictionary must be rehashed; I actually cannot remember if this is true in Java but it is with Smalltalk.
Thats error prone. That is the reason I tried to do that completely automatically. You set up it correct one time and move on, just don't have to think about it again.
It is an unfortunate piercing of the transparency. For me its impact has been very limited and isolated; but then, I mostly only use MagmaCollections for keyword searching my domain models. I put a #signalNoteOldKeysFor: in my #description: setter and, done.
The index I'm trying to create makes this more error prone: I have a root object which is in a MagmaCollection. It has a keywords index. The keywords are made up of a list which contains multiple fields of the root object but also of a list of fields of the sub objects of this root. Using this, I can implement a kind of google search box in which you can find keywords dispersed over my entire domain and let Magma retrieve the correct aggegrate root, and I would not have to integrate any kind of full text search technology to do this. The reason I wanted it to be transparent is that if I for instance add a new field on this keywords list, I now have to remember the signalNoteKeys on this setter, (which is error prone ...).
Ah, thank you for the explanation. May I offer a couple of suggestions?
Option 1) Can the "sub-objects" know their parent / "root" object? If so, instead of indexing your root object by it's keywords and all of its sub-objects key-words, just make the root object respond with its *own* keywords. Make each sub-object respond with its *own* keywords too. Add the sub-objects to the MagmaCollection as well as the root object. When a Reader of objects is found, display them all of them (a heterogeneous list) or, if the sub-objects are not wanted, traverse up their parents to the "roots" and put them into a Set (to avoid presenting duplicates to the user). This way, you only need #signalNoteOldKeysFor: in just the keywords setter of each type of object that can be searched.
Option 2) Throw the MagmaCollection, indexes, and all the calls to #signalNoteOldKeys: out the window. Implement #maContextKeywordsDo: on all domain objects you want to have keyword search capability (otherwise, the printString of the object will act as its keywords!). As in Option 1, each domain object only values the passed in Block with its *own* keywords.
You can then send, #maNewSearchContext to any object in order to search its "sub-objects". This search object runs in the background, provides progress indication, provides results as they are found (so research can begin concurrently), and even orders the results according to how well they matched (e.g., whole match first, then front match, finally substring match). The results, themselves, are a searchable context. Multiple contexts can even be grouped into "CompositeContexts" so that one keyword can search a number of sources.
To learn more about this, start at the class MaAbstractContext, its class comment. This framework is pretty easy to use and it works.
Option 3) Implement your own visitor to do a search similar to (2).
You say that it is slow. Did Magma once provide this auto notify keys functionality? I think I could maybe live with the bit slower performance
It won't hurt anything to send #noteOldKeysFor: even if the key, in fact, did not change. I know there are places to hook to send #noteOldKeysFor: to every object so that Magma will appear to automatically detect it.. But please evaluate other options first, as I would like to wrap up this e-mail for now.. :)
Man, I sure wish more folks had your good patience! Someone else recently said Magma is "dog-slow". It may be, but I like dogs. :) Hopefully Cog will help someday!
Cheers, Chris
Hi Chris,
thanks yet again for your clarifications. I really appreciate it.
I tried the solution using the MagmaSessionRequest signalCommit: for my Repository store and save methods and signalNoteOldKeysFor: for the mutators which generated the keywords.
I only had to adapt my MagmaIntegrationTest and to add a WARequestFilter for my seaside app to bootstrap the MagmaSessionRequest.
Using this repositories abstraction, I even have a gui which uses the in memory version and a version which is using the Magma version. Since the gui only knows the Repositories interface (store: save: findxxx) methods, it just works in both cases. Which is nice.
Incidentally, there is a parallel "lack-of-transparency" in standard
Smalltalk and even, if I am not mistaken, Java. If you were to use a standard Dictionary / HashTable to look up objects, when something causes an objects #hash / hashCode() to change, the Dictionary must be rehashed; I actually cannot remember if this is true in Java but it is with Smalltalk.
You are absolutely right, same in Java world. Updating a field which influences the hash key, does not automatically update any collection where this hash happens to be used. I was just again thinking in the SQL world , where if you update a column which has an index on it defined, the index gets updated automatically.
Thats error prone. That is the reason I tried to do that completely
automatically.
You set up it correct one time and move on, just don't have to think
about
it again.
It is an unfortunate piercing of the transparency. For me its impact has been very limited and isolated; but then, I mostly only use MagmaCollections for keyword searching my domain models. I put a #signalNoteOldKeysFor: in my #description: setter and, done.
I'm doing the same. If all the rest of Magma is transparent, it is still very impressive. You just do some initial setUp and further more, just expand the domain.
The index I'm trying to create makes this more error prone: I have a root object which is in a MagmaCollection. It has a keywords index. The
keywords
are made up of a list which contains multiple fields of the root object
but
also of a list of fields of the sub objects of this root. Using this, I
can
implement a kind of google search box in which you can find keywords dispersed over my entire domain and let Magma retrieve the correct
aggegrate
root, and I would not have to integrate any kind of full text search technology to do this. The reason I wanted it to be transparent is that
if I
for instance add a new field on this keywords list, I now have to
remember
the signalNoteKeys on this setter, (which is error prone ...).
Ah, thank you for the explanation. May I offer a couple of suggestions?
Option 1) Can the "sub-objects" know their parent / "root" object? If so, instead of indexing your root object by it's keywords and all of its sub-objects key-words, just make the root object respond with its *own* keywords. Make each sub-object respond with its *own* keywords too. Add the sub-objects to the MagmaCollection as well as the root object. When a Reader of objects is found, display them all of them (a heterogeneous list) or, if the sub-objects are not wanted, traverse up their parents to the "roots" and put them into a Set (to avoid presenting duplicates to the user). This way, you only need #signalNoteOldKeysFor: in just the keywords setter of each type of object that can be searched.
I will try this option.
Option 2) Throw the MagmaCollection, indexes, and all the calls to #signalNoteOldKeys: out the window. Implement #maContextKeywordsDo: on all domain objects you want to have keyword search capability (otherwise, the printString of the object will act as its keywords!). As in Option 1, each domain object only values the passed in Block with its *own* keywords.
You can then send, #maNewSearchContext to any object in order to search its "sub-objects". This search object runs in the background, provides progress indication, provides results as they are found (so research can begin concurrently), and even orders the results according to how well they matched (e.g., whole match first, then front match, finally substring match). The results, themselves, are a searchable context. Multiple contexts can even be grouped into "CompositeContexts" so that one keyword can search a number of sources.
To learn more about this, start at the class MaAbstractContext, its class comment. This framework is pretty easy to use and it works.
I have a question about this. I looked into it, and I'm a bit concerned about scalability. I've did some testing and Magma can easily using a MagmaCollection, and an index on it, search through a million records and find the first 20 matching ones in 250 milliseconds. Thats actually pretty fast !! And since Magma is also scalable in the number of nodes you can add, you can more or less guarantee that every user can have a response time which is in that range. Which is very nice!
So now the question. I noticed that this context framework uses a collectionreader, enumerates all elements in database, and sees if the the keywords match (with a match percentage). So it can be that the one you're actually looking for is just not found, since there is a timeout on the searching time (5 minutes - I don't know anybody that is willing to wait that long for a response). Is this correct what I'm saying, or did I not completely understand it?
Man, I sure wish more folks had your good patience! Someone else recently said Magma is "dog-slow". It may be, but I like dogs. :) Hopefully Cog will help someday!
Thank you again for your never ending effort explaining and improving Magma!
Kind Regards,
Bart
Hi, sorry for the delay.
I have a question about this. I looked into it, and I'm a bit concerned about scalability. I've did some testing and Magma can easily using a MagmaCollection, and an index on it, search through a million records and find the first 20 matching ones in 250 milliseconds. Thats actually pretty fast !! And since Magma is also scalable in the number of nodes you can add, you can more or less guarantee that every user can have a response time which is in that range. Which is very nice! So now the question. I noticed that this context framework uses a collectionreader, enumerates all elements in database, and sees if the the keywords match (with a match percentage). So it can be that the one you're actually looking for is just not found, since there is a timeout on the searching time (5 minutes - I don't know anybody that is willing to wait that long for a response). Is this correct what I'm saying, or did I not completely understand it?
Your understanding correct, with one clarification:
collectionreader, enumerates all elements in database, and sees if the the
enumerates all elements in the reader...
Yes, the nature of the searching is linear, so that the framework can endow _any object_ with keyword search capabilities with no coding (by traversing its physical object graph). That's why it's a background with results-as-you-go.
But I should have to remembered you are working in a web environment, and so this option is perhaps not appropriate. Whether a web interface _can_ support a highly-interactive UI with background search results populating in real-time is superceded by whether it _should_; in the natural, expected behavior of web applications. Option 1 is the best choice for web.
- Chris
magma@lists.squeakfoundation.org
-
Bart Gauquie -
Chris Muller