Magma November 2006

magma@lists.squeakfoundation.org

7 participants
15 discussions

[ANN] MagmaServerLoader-cmm.25
by Chris Muller 08 Mar '07

08 Mar '07

I am pleased to announce the latest Magma upgrade! The main thrust of this release are bug fixes but, primarily, one particular bug which required a significant amount of work; complete transparency of "pseudo-persistent" MagmaCollectionReaders. By pseudo-persistent I mean Readers that are the result-sets of a query on a new, automatically created MagmaCollection borned from using one of the "luxury" query features (either distinctness or sorting by other than the most-optimal attribute) on another MagmaCollection. Although these new Reader/Collection pairs, are not (initially, at least) truly persistent because they are not referenced by the persistent object model, they do, like persistent collections, require support by back-end (hash-index) files and managed by a server object. Until now, attempting to further query these "result set" Readers or attach them to the persistent graph did not work. Now it does. Further, thanks to Hilaire's great suggestion, further querying of Readers is fully symmetrical with MagmaCollections, i.e., you can use #where: and friends on a Reader. I don't think Brent will like this but.. along with this is the notion that the back-end files for "result sets" are no longer deleted simply because the creator session disconnects. They can't be. Just because a session disconnects doesn't mean its done with the MagmaCollectionReader in its image. For example, the disconnect could just be a side-effect of saving the image. So how are back-end files ever removed? Via MagmaCompressor. So, applications that allow users to run many hundreds of "luxury" queries, will accumulate quite a few result-set support files. This would pose a problem for the Magma server, up until now, because the server would open all known MagmaCollection and index files upon starting, and any queries would create (and open) files that would remain open until the server was shut down. No longer. Now, all files are now opened lazily! The Magma server should start more quickly now. Further, to mitigate the sheer number of open file handles for query result sets, Magma now employs a "file pool" to limit the number of simultaneously-open files. The user may set the maximum number of open files via #filePoolSize: (the default is 60), and oldest-accessed files are closed. Not only does this allow a good approach to dealing with many query result sets, it is more ISP-friendly (some ISP's restrict the number of simultaneously-open files you may have). For those familiar with Magma's meta-model (MagmaRepositoryDefinition), it no longer includes 'allLargeCollections'. This turned out to be significant addition-by-subtraction. There is a reduction in the model, code (including some special-case code) and number of byte-code instructions the computer must execute. A win win win all-around. Caveats: Ah, of course. Besides the aforementioned potential file explosion, it is now important to let Magma have the entire directory. __Do not keep any of your own important files in any Magma repository directory__ because MagmaRepositoryController>>#delete: now simply deletes all files in the directory. Upgrading: Guess what?! All existing databases are completely compatible with the new code. Why? Because, in spite of the change in Magma's meta model, the class mapping permits "dynamic evolution" of the class model, and removal of an instance variable is actually the simplest case. Other fixes: - Fixed bug with MagmaCollection>>#asArray:. Thanks to Göran's team! - Fixed at:ifOutOfBounds: sometimes giving error instead of valuing the outOfBoundsBlock reported by Göran. - Added Form and BitMap to default ReadStrategy, 99999 depth, thanks Hilaire for reporting. - ensure MaTransactionalFileStreams are always binary, this fixes the "Outage occurred" message in 3.9. I spent a lot of time on this and believe it to be a quality release. It is, therefore, the new Magma baseline which I've posted to both the "Magma" project as well as the "MagmaTester" project. Enjoy!

3 2

next steps..
by Chris Muller 22 Nov '06

22 Nov '06

The queries are done and turned out a wonderful improvement to 1.0. Now I have found my previous dusty image where I was before having begun that wonderful detour. Ah yes, like a time capsule, everything just as I had it. Monticello shows 10 packages with lots of changes. Great, now I can see not just versioned code but the actual trajectory I was working on. This is the "1.1" version, a.k.a., "Magma with security." Magma works well behind a web server, why complicate things? Well, let me clarify, unless you "turn on" the security, operation of 1.1 will be almost identical to 1.0 without security. Also, I will not allow performance to be compromised more than.. at most 5%-10% when running with security off (with privacy security on, I expect performance to drop dramatically). At a minimum, 1.0 and 1.1 will share the same file-format and execution environment. If this can be achieved, there will no need to stay with 1.0. The key is to make sure it defaults to "most usable, least secure" since that is more naturally compatible with project life-cycles (starting simple and then needing more serious things once the project gets more serious). Behind a web-app will still be just as usable and just as fast. But before repositories of objects directly accessible from within the Squeak environment can be shared across a public network, we gotta have security. That's the core requirement, and that's what is next for Magma. Wouldn't anyone else like to see World menu -> open... -> Mailing List Browser. It connects to a publicly-hosted db with a rich model of Threads, Messages, Posters, SqueakPeople, etc. People could query for answers and run the (for example) doIt examples riight there, directly in the image..

1 0

[Q] Is there any tutorial/example for Magma + Seaside integration?
by Sungjin Chun 14 Nov '06

14 Nov '06

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, as subject says, I'm newbie to Magma. I've found some simple tutorial on wiki but I need more information. Thanks in advance. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFWUDgQqspS1+XJHgRApuUAKC3VRWkoho+XdTPsA7QZIMpX4mKuQCg3j7d RNtNx8dCM48BLOyogP/ei04= =xjIj -----END PGP SIGNATURE-----

1 0

Re: Old version(s) of Magma
by Chris Muller 14 Nov '06

14 Nov '06

Hi Aaron, copying Magma list. I do indeed have some old versions of Magma around. I have attached two versions, Magma 1.0 gamma7 (dated 4/2004) and Magma 1.0 gamma7.2 (dated 10/2004) But before you spend any time on these, let's please talk about this. I take it you are really are interested in <=3.6 *compatibility*, not an old version of Magma, right? I ask this because I expect the current version of Magma runs on 3.7, and I actually haven't put in much effort for 3.6. The current version of Magma has come a long way since those circa Squeak-3.6 versions and is the best version in terms of quality, performance, features and stability. So if the current version can work on <=3.6 then you definitely want that right? In reality, it probably won't work on 3.6 out of the box, but given the choice of putting energy into 1) making current Magma compatible with older Squeak, or 2) dealing with old bugs / older file format / fewer features and less stability, I think you'd be happer with option 1. I'll add, I'd be much more interested in helping out with option 1 vs. 2. Regards, Chris ----- Original Message ---- From: Aaron Reichow <areichow(a)ipharos.com> To: chris(a)funkyobjects.org Sent: Monday, November 13, 2006 9:10:20 AM Subject: Old version(s) of Magma Chris- Hi! I'm wondering if you have any zips/tarballs of any old versions of Magma, possible along with whatever documentation might come along with the old release(s)? I am going to be using an older version of Squeak- probably 3.2, but possibly anything between 3.0 and 3.6. I might have a copy myself, but I was wondering if you send me a complete archive of whatever I might need to get Magma going on my older images? Thanks a ton!! Regards, Aaron Aaron Reichow || iPS Developer @ RSI || areichow(a)ipharos.com "Liberty will not descend to a people, a people must raise themselves to Liberty." -- emma goldman

1 0

Re: dataHandler error
by Chris Muller 10 Nov '06

10 Nov '06

Hi Hilaire, sorry for a delayed response. As you can see I have been very busy trying to get this latest release out the door. I'm quite excited that you are trying Magma over the Internet. I consider support for two-tier applications the future of Magma. So that we as a group can have direct access to a shared repository of objects directly from an image, not just through a web browser, for example. But I have not yet had the opportunity to coordinate with someone far away to attempt this. I don't know for sure what is causing the dataHandler error you are seeing. How does the error manifest, as a debugger? Also, I would be *very* interested in running the MagmaBenchmarker over the Internet. I can configure the code so that your helper in the remote location can simply start the server with "one click" and then you can kick off the benchmark which will print the statistics. Would you be willing to do this? Let me know privately and we can talk more about the details. Thanks! Chris ----- Original Message ---- From: Hilaire Fernandes <hilaire2006(a)laposte.net> To: magma(a)lists.squeakfoundation.org Sent: Sunday, November 5, 2006 9:58:35 AM Subject: dataHandler error I am conducting test of the use of Magma with one server and one client in two separate places in the Internet (France<-->Taiwan link), throught not so fast link. So far so good, and as wrote Göran, Index are very helpful. From time to time I got some error message like this one in the server: dataHandler: selected socket fd=45 flags=0x2 would block (why?) Then a MaNetworkError 'Connection closed while waiting for response.' in the client side. The situation is a bit special: To reduce the latency from the client side, I fork the commits in separate processes. Looking at the process browser, I can see that Magma does it job to queue the commit (I guess it is what is about Semaphore use). Could the problem be related to the use of only one session for several forked commit? Hilaire _______________________________________________ Magma mailing list Magma(a)lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma

1 0

Re: [ANN] MagmaServerLoader-cmm.25
by Chris Muller 10 Nov '06

10 Nov '06

Hi Vladimir, ok, I now learned something about Monticello. It is not enough to simply delete your package-cache directory and then to load from squeaksource, to verify that all dependent packages are present; because any other Repository you have added to Monticello is also searched for dependencies even if you aren't loading from that repository! Anyway, thanks for pointing it out. I have included that missing package and now Magma loads fine. Please let me know if you have any other questions or problems. - Chris ----- Original Message ---- From: Vladimir Plšek <vladimir.plsek(a)siemens.com> To: magma(a)lists.squeakfoundation.org Sent: Thursday, November 9, 2006 8:07:50 AM Subject: Re: [ANN] MagmaServerLoader-cmm.25 hi, I loaded new MagmaTesteLoader from Magma mc repository. There was one error during loading about misssing "Ma special collections-cmm.72.mcz", which I didn't find on mc. I loaded lower version of it. Everything looks fine. Old magma data are working, but when I try to create new one "MagmaRepositoryController create:(FileDirectory default fullNameFor: 'magma-test') root: Dictionary new." I got i truble. Squeak got's in endless loop, after terminating it a got message about "MaFileRecordBroker(Object)>>doesNotUnderstand: #filesDo:". am i doing something wrong? vlado plsek Chris Muller wrote: > I am pleased to announce the latest Magma upgrade! The main thrust of > this release are bug fixes but, primarily, one particular bug which > required a significant amount of work; complete transparency of > "pseudo-persistent" MagmaCollectionReaders. By pseudo-persistent I mean > Readers that are the result-sets of a query on a new, automatically > created MagmaCollection borned from using one of the "luxury" query > features (either distinctness or sorting by other than the most-optimal > attribute) on another MagmaCollection. Although these new > Reader/Collection pairs, are not (initially, at least) truly persistent > because they are not referenced by the persistent object model, they do, > like persistent collections, require support by back-end (hash-index) > files and managed by a server object. > > Until now, attempting to further query these "result set" Readers or > attach them to the persistent graph did not work. Now it does. Further, > thanks to Hilaire's great suggestion, further querying of Readers is fully > symmetrical with MagmaCollections, i.e., you can use #where: and friends > on a Reader. > > I don't think Brent will like this but.. along with this is the notion > that the back-end files for "result sets" are no longer deleted simply > because the creator session disconnects. They can't be. Just because a > session disconnects doesn't mean its done with the MagmaCollectionReader > in its image. For example, the disconnect could just be a side-effect of > saving the image. So how are back-end files ever removed? Via > MagmaCompressor. > > So, applications that allow users to run many hundreds of "luxury" > queries, will accumulate quite a few result-set support files. This would > pose a problem for the Magma server, up until now, because the server > would open all known MagmaCollection and index files upon starting, and > any queries would create (and open) files that would remain open until the > server was shut down. No longer. > > Now, all files are now opened lazily! The Magma server should start more > quickly now. Further, to mitigate the sheer number of open file handles > for query result sets, Magma now employs a "file pool" to limit the number > of simultaneously-open files. The user may set the maximum number of open > files via #filePoolSize: (the default is 60), > and oldest-accessed files are closed. Not only does this allow a good > approach to dealing with many query result sets, it is more ISP-friendly > (some ISP's restrict the number of simultaneously-open files you may > have). > > For those familiar with Magma's meta-model (MagmaRepositoryDefinition), it > no longer includes 'allLargeCollections'. This turned out to be > significant addition-by-subtraction. There is a reduction in the model, > code (including some special-case code) and number of byte-code > instructions the computer must execute. A win win win all-around. > > Caveats: > > Ah, of course. Besides the aforementioned potential file explosion, it is > now important to let Magma have the entire directory. __Do not keep any > of your own important files in any Magma repository directory__ because > MagmaRepositoryController>>#delete: now simply deletes all files in the > directory. > > Upgrading: > > Guess what?! All existing databases are completely compatible with the > new code. Why? Because, in spite of the change in Magma's meta model, > the class mapping permits "dynamic evolution" of the class model, and > removal of an instance variable is actually the simplest case. > > Other fixes: > > - Fixed bug with MagmaCollection>>#asArray:. Thanks to Göran's team! > - Fixed at:ifOutOfBounds: sometimes giving error instead of valuing the > outOfBoundsBlock reported by Göran. - Added Form and BitMap to default > ReadStrategy, 99999 depth, thanks Hilaire for reporting. - ensure > MaTransactionalFileStreams are always binary, this fixes the "Outage > occurred" message in 3.9. > > I spent a lot of time on this and believe it to be a quality release. It > is, therefore, the new Magma baseline which I've posted to both the > "Magma" project as well as the "MagmaTester" project. > > Enjoy! _______________________________________________ Magma mailing list Magma(a)lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma

1 0

Hmmm, missed bug fix?
by goran＠krampe.se 07 Nov '06

07 Nov '06

Hi! I am not sure, but I use squeaksource.com/Magma - MagmaServerLoader-cmm.22. In that code I can't see this fix: http://www.gjallar.se/gjallar/47 regards, Göran

1 0

Re: Head on MagmaCollection
by Chris Muller 07 Nov '06

07 Nov '06

Not directly, because only #where: is something for MagmaCollection's which returns a Reader. Depending on your indexes, you can use & (and) to maximize query performance. But still nesting support is a very good idea so that MagmaCollections and Readers are 'symmetrical" w.r.t. where querying. To that end, it is something I have already been working on and am very close. Depending on whether you use "luxury" features like distinct: true or sorting on a non-optimized attribute, it must give you back a Reader on a *new* MagmaCollection. So if you then ask the reader for this new collection you can then do further where: query's on that. But I should probably take it one step further so you can use #where: on a reader, good idea.. ----- Original Message ---- From: Hilaire Fernandes <hilaire2006(a)laposte.net> To: magma(a)lists.squeakfoundation.org Sent: Friday, October 27, 2006 1:47:46 PM Subject: Re: Head on MagmaCollection Chris Muller a écrit : > If you have not explored MagmaCollections yet you should know they > provide Magma the much of the benefits of relational databases > without losing its ODBMS transparency at all. Central to this > philosophy is the fact that the where: syntax, for quickly creating > subsets of massive collections is, in most cases, identical to Is it correct to nest where: method? I guess it could be useful to speed up a query when your select criterias are deep in the tree. Hilaire _______________________________________________ Magma mailing list Magma(a)lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma

2 1

dataHandler error
by Hilaire Fernandes 05 Nov '06

05 Nov '06

I am conducting test of the use of Magma with one server and one client in two separate places in the Internet (France<-->Taiwan link), throught not so fast link. So far so good, and as wrote Göran, Index are very helpful. From time to time I got some error message like this one in the server: dataHandler: selected socket fd=45 flags=0x2 would block (why?) Then a MaNetworkError 'Connection closed while waiting for response.' in the client side. The situation is a bit special: To reduce the latency from the client side, I fork the commits in separate processes. Looking at the process browser, I can see that Magma does it job to queue the commit (I guess it is what is about Semaphore use). Could the problem be related to the use of only one session for several forked commit? Hilaire

1 0

Re: Magma question
by Chris Muller 04 Nov '06

04 Nov '06

Hey Göran, congratulations Gjaller. I saw your presentation (the BOF video) and it looks very impressive! I think we should push (eventaully, at least) to use Gjaller to manage the Squeak development process and bugs. > The more interesting fact is that Magma indexes seem to do their job > perfectly fine with 600 cases in Gjallar. The indexes have been very heavily tested, I don't expect you will have any problems with 10000, 100K or even much more (other than, perhaps, the time to load them!). As for the buffer reading, is it possible perhaps that you were asking for the #root in there somewhere? That allways forces a buffer read. If you were only sending #at: to a MagmaCollectionReader, then I agree it doesn't seem like it should be doing any buffer reads.. I'll do some more thinking and poking around, or if you have more detail about what operations you were doing during the profile that would probably help. Regards, Chris ----- Original Message ---- From: "goran(a)krampe.se" <goran(a)krampe.se> To: afunkyobject(a)yahoo.com; magma(a)lists.squeakfoundation.org Sent: Friday, November 3, 2006 8:05:22 AM Subject: Magma question Hi Chris! Ok, so I am frantically getting Gjallar into some deployable shape here. :) Yesterday I played with creating 600 cases programmatically - kinda complex objects, one per commit - took 25 min which isn't all bad - and it was doing other silly things too. The more interesting fact is that Magma indexes seem to do their job perfectly fine with 600 cases in Gjallar. One of these nights I will push 10000 into it and we can see what happens. BUT... one silly question: If I use a reader with an index query - and the cases that the reader finds are already cached into the session - should it really be doing buffer reading? While doing one of these operations using the profiler I kept hitting "Q2Case allInstances size" in a workspace and it constantly showed 85 - just like before. So no new Q2Case objects were created but the profiler spent 10 seconds in Magma doing buffer reads it seems. I had expected it to just get some kind of oid, discover it already has the object in the imag - and that would be it. I can of course provide more info if you need it. regards, Göran _______________________________________________ Magma mailing list Magma(a)lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Magma November 2006