Magma November 2010

magma@lists.squeakfoundation.org

10 participants
12 discussions

[ANN] ConfigurationOfMagma for Pharo 1.0, Pharo 1.1 and Pharo 1.2
by Miguel Enrique Cobá Martínez 08 Dec '10

08 Dec '10

Hi all, a new version of ConfigurationOfMagma is available for Pharo. Notes: Pharo 1.0 ---------------------- - 1.1r1 is the last versions that works in Pharo 1.0. This is because upstream Magma code depends on code that only exists on Pharo 1.1 and superior. - Pharo 1.0 uses Preferences to enable/disable the underscore as assigment in code. This release captures current value of this preference. Enables it for loading Magma and restores it to the original value after installing Magma. Pharo 1.1, Pharo 1.2 ---------------------- - The configuration for these Pharo releases is based on the one from Pharo 1.0. - 1.1r2 uses a new baseline because Magma has a new package dependency (Ma files additions). - Pharo 1.1 and superior uses Scanner allowUnderscoreAsAssignment instead of Preferences. The current value is captured and restored after installation of Magma. - Pharo 1.1 and superior doesn't include FlapTab class anymore. Magma adds some methods to it so the Configuration adds an empty class named FlapTab before installing and removes from the system after installing. This avoid a warning prompt when installing Magma. - This Configuration includes support for installing Magma releases previous to 1.1r2. Those versions should install correctly on Pharo 1.1 and greater but if a lot of problems are detected, the support for those old versions will be removed, and only 1.1r2 and newer versions will be supported. Now that Pharo has Universes, Magma has different ConfigurationOfMagma in each universe for Pharo. Each of those Configuration install the appropriate packages in Pharo. Installing Magma Pharo 1.0 --------------- Evaluate: Gofer new squeaksource: 'UniverseForPharo10'; package: 'ConfigurationOfMagma'; load. And then, one of: ConfigurationOfMagma project latestVersion load: 'Client'. ConfigurationOfMagma project latestVersion load: 'Server'. ConfigurationOfMagma project latestVersion load: 'Tester'. latestVersion is 1.1r1 Pharo 1.1 --------------- Evaluate: Gofer new squeaksource: 'UniverseForPharo11'; package: 'ConfigurationOfMagma'; load. And then, one of: ConfigurationOfMagma project latestVersion load: 'Client'. ConfigurationOfMagma project latestVersion load: 'Server'. ConfigurationOfMagma project latestVersion load: 'Tester'. latestVersion is 1.1r2 Pharo 1.2 --------------- Evaluate: Gofer new squeaksource: 'UniverseForPharo12'; package: 'ConfigurationOfMagma'; load. And then, one of: ConfigurationOfMagma project latestVersion load: 'Client'. ConfigurationOfMagma project latestVersion load: 'Server'. ConfigurationOfMagma project latestVersion load: 'Tester'. latestVersion is 1.1r2 Comments and bugs accepted. Enjoy. -- Miguel Cobá http://miguel.leugim.com.mx

4 4

Re: Magma upgrater
by Chris Muller 04 Dec '10

04 Dec '10

Try this: MagmaSession initialize and then run your upgrade. I've seen this error a couple of times, now I think what's happening is that the class-initialization for MaObjectBuffer got run sometime _after_ MagmaSession. Under normal loading procedures that would not happen, but if a new version of Ma object serialization was loaded after Magma, it could happen. Let me know, Chris On Sun, Nov 28, 2010 at 3:50 PM, Facundo Vozzi <facundov79(a)gmail.com> wrote: > Sorry, I forgot attach the error. There you go. > Thanks, > Facu > > > On Sun, Nov 28, 2010 at 2:57 PM, Chris Muller <ma.chris.m(a)gmail.com> wrote: >> >> What error? Screenshot? >> >> On Fri, Nov 26, 2010 at 3:17 PM, Facundo Vozzi <facundov79(a)gmail.com> >> wrote: >> > Hi Chris, >> > I did it but I'm getting the error that adjunt in the screnshot. Do you >> > know >> > what I'm doing wrong? >> > Thanks you, >> > Facundo >> > On Fri, Nov 26, 2010 at 3:33 PM, Chris Muller <asqueaker(a)gmail.com> >> > wrote: >> >> >> >> Hi Facundo, please use the "Magma upgrader" package available on the >> >> "Magma tester" project of squeaksource. >> >> >> >> After loading it: >> >> >> >> (MagmaUpgrader source: '/path/to/old/repository') compressTo: >> >> '/path/to/upgraded/repository' >> >> >> >> - Chris >> >> >> >> On Fri, Nov 26, 2010 at 6:54 AM, Facundo Vozzi <facundov79(a)gmail.com> >> >> wrote: >> >> > Hi, >> >> > how is the way for upgrade my magma repository from version 12 >> >> > to version 14 >> >> > in order to use it with the version 1.2a1 of magma? >> >> > Someone was able to test version 1.2a1 in Pharo 1.1.1? >> >> > Thanks in advance, >> >> > Facu >> >> > >> >> > _______________________________________________ >> >> > Magma mailing list >> >> > Magma(a)lists.squeakfoundation.org >> >> > http://lists.squeakfoundation.org/mailman/listinfo/magma >> >> > >> >> > >> > >> > > >

4 6

Re: hard-drive read-performance
by Chris Muller 29 Nov '10

29 Nov '10

Yep, it's called MagmaCompressor. But still, if you have a multi-gigabyte repository and a HD buffer of, what, a few K? or even a client-cache of a 100MB, there is no way around needing to read off the HD.. On Sun, Nov 28, 2010 at 8:31 PM, Elliot Finley <efinley.lists(a)gmail.com> wrote: > maybe a defrag utility for Magma that places all objects in a collection > close together on the disk? > > On Wed, Nov 24, 2010 at 10:00 AM, Chris Muller <ma.chris.m(a)gmail.com> wrote: >> >> When reading any object off the hard drive (represented as the >> 'byteArray' of a single MaObjectBuffer), Magma always reads 280 bytes. >> Since the #physicalSize is in the object header, it is then able to >> check the contents of the buffer to determine the size of the whole >> object and, if necessary, read more bytes in order to get the whole >> object. See MaObjectFiler>>#read:bytesInto:and:startingAt:filePosition: >> for this behavior. >> >> 280 bytes is enough for about 40 pointer references, allowing most >> objects to be read in just one disk access. I refer to it as the >> #trackSize, to remind me it is supposed to be how many bytes I think >> can the HD read in one operation without overrunning its own internal >> buffers and becoming inefficient. I was curious whether this number >> is optimized in 2010, so I ran the following script: >> >> ----------- >> |stats random| stats:=OrderedCollection new. random := Random new. >> nextPos:=100. >> (FileDirectory on: '/home/cmm/test3/cube.001.magma') fileNamed: >> 'objects.2.dat' do: >> [ : stream | | ba fileSize | ba := ByteArray new: 10000. >> fileSize := stream size. >> 100 to: 10000 by: 100 do: >> [ : n | >> stream position: 0. >> Transcript cr; show: (stats add: n->([stream >> maRead: n "bytes" >> bytesFromPosition: 1 >> of: ba >> atFilePosition: (random nextInt: fileSize ] >> bench)) ]]. >> stats >> ------------ >> >> Note that "objects.2.dat" is a real Magma file, 1.8GB in size. The >> goal of the script is bench how fast Squeak can read object buffers >> off the hard-drive when we obviously won't get many (if any) HD cache >> hits. >> >> I have a cheap, Western Digital Caviar HD, which produced the following >> output: >> >> 100->'119 per second.' >> 200->'98.5 per second.' >> 300->'106 per second.' >> 400->'106 per second.' >> 500->'101 per second.' >> 600->'102 per second.' >> 700->'99.9 per second.' >> 800->'103 per second.' >> 900->'104 per second.' >> 1000->'99 per second.' >> 1100->'97.9 per second.' >> 1200->'104 per second.' >> 1300->'111 per second.' >> 1400->'99.8 per second.' >> 1500->'107 per second.' >> 1600->'108 per second.' >> 1700->'95.6 per second.' >> 1800->'103 per second.' >> 1900->'108 per second.' >> 2000->'102 per second.' >> 2100->'103 per second.' >> 2200->'107 per second.' >> ... >> 3000->'98.7 per second.' >> 4000->'102 per second.' >> 5000->'106 per second.' >> 6000->'104 per second.' >> 7000->'101 per second.' >> 8000->'102 per second.' >> 9000->'102 per second.' >> 10000->'107 per second.' >> >> For curiousity, I also modified the script to read very small buffers >> from the HD, here are the results: >> >> 4->'137 per second.' >> 12->'146 per second.' >> 20->'154 per second.' >> 28->'143 per second.' >> >> (The HD busy light was solid ON during the test). >> >> At first I was puzzled because Magma has demonstrated much faster >> objects-per-second read rates than these, even including >> materialization, what gives? >> >> It's the HD buffering. Most of the time, objects are "clustered" >> closely together, so that reading one object causes the "next" object >> which will be read to already be in the HD's buffer. Here's the same >> script, except reading mostly "sequentially" through the file instead >> of from a random location: >> >> |stats random nextPos| stats:=OrderedCollection new. random := Random new. >> nextPos:=100. >> (FileDirectory on: '/home/cmm/test3/cube.001.magma') fileNamed: >> 'objects.2.dat' do: >> [ : stream | | ba fileSize | ba := ByteArray new: 10000. >> fileSize := stream size. >> #(4 12 20 28 100 200 300 400 500) >> [ : n | >> stream position: 0. >> Transcript cr; show: (stats add: n->([stream >> maRead: n "bytes" >> bytesFromPosition: 1 >> of: ba >> atFilePosition: ("random nextInt: fileSize" >> (nextPos := >> nextPos+n+10)) ] bench)) ]]. >> stats >> >> Now look at the results: >> >> "Reading sequentially rather than at a random position." >> 4->'1,160,000 per second.' >> 12->'1,210,000 per second.' >> 20->'1,100,000 per second.' >> 28->'973,000 per second.' >> ... >> 100->'1,030,000 per second.' >> 200->'321,000 per second.' >> 300->'215,000 per second.' >> 400->'160,000 per second.' >> 500->'227,000 per second.' >> >> Conclusions: >> >> - Hard-disk seek is definitely a bottleneck with Magma, or any >> Squeak application that requires random-access to a file. >> - When objects are clustered closely together, read performance can >> be dramatically better. >> - HD's with fast seek times, such as newer solid-state drives, might >> perform dramatically better. >> - I should consider reducing the trackSize from 280 bytes to ~100 >> bytes (or make it customizable); because the rate drops really fast >> after that and even a second read required could still be faster than >> an initial read. >> >> - Chris >> _______________________________________________ >> Magma mailing list >> Magma(a)lists.squeakfoundation.org >> http://lists.squeakfoundation.org/mailman/listinfo/magma > >

1 0

Magma read performance
by Chris Muller 29 Nov '10

29 Nov '10

There has been interest in Magma's read and query performance lately, so I thought I would share results of a recent benchmark test. It's an actual application which does very heavy reading and writing to a Magma repository. This test was 24GB of repository work, over two days. My goal was to determine, once the persistent model becomes many times the size of RAM and HD access appeared to become the limiting factor: - how fast is Magma, in terms of "objects per second" and in "kilobytes per second"? - how fast is this relative to the speed when the repository was empty? Conclusion: - Magma started at 4K objects per second (empty repository), 282K per second. - Finished with 2.3K objects per second (6GB repository), 750K per second. - Memory consumption by the image never exceeded 300MB. It is important to note, these times are from the client MagmaSession point-of-view, including the full server roundtrip plus materialization. Also, as can be seen from the attached data, there were many requests which only brought back one or two objects which, while dramatically lowering the overall reported throughput, is actually a real-world scenario for applications. Verbose description: When the test first started the repository was tiny, and there were only ~4K server requests during a 5-minute monitored period. The total number of objects read across all ~4K requests was ~17M for an average throughput of 4K objects per second (ops). As the model grew in size and complexity, more and more objects during the 5-minute monitored intervals were required to perform application-posting of the same number of input records; doubling to ~8K server requests during a 5-minute period, however only a few more objects brought back (total 21M), for an average of 2500 ops. 32 hours after that, the rate of reads dropped to about 2300 ops. This is due to two factors: - the objects were larger (e.g., more pointers to other objects) - they were less clustered (having been replaced with objects which could not be co-located with the original object) The fact that objects got bigger (e.g., more pointers to other objects) is corroborated by another stat, "the number of *bytes* per second" read off the HD in order to access those objects. Initially, when the test first started, there were about 4K *requests* for objects in one of the first five-minute periods, which were read at a rate of 282K per second. 24 hours later, there were only 2 times as many requests for objects, but were read-and-materialized at a rate of 750K bytes / second. - Chris

3 4

Magma upgrater
by Facundo Vozzi 26 Nov '10

26 Nov '10

Hi, how is the way for upgrade my magma repository from version 12 to version 14 in order to use it with the version 1.2a1 of magma? Someone was able to test version 1.2a1 in Pharo 1.1.1? Thanks in advance, Facu

2 1

hard-drive read-performance
by Chris Muller 24 Nov '10

24 Nov '10

When reading any object off the hard drive (represented as the 'byteArray' of a single MaObjectBuffer), Magma always reads 280 bytes. Since the #physicalSize is in the object header, it is then able to check the contents of the buffer to determine the size of the whole object and, if necessary, read more bytes in order to get the whole object. See MaObjectFiler>>#read:bytesInto:and:startingAt:filePosition: for this behavior. 280 bytes is enough for about 40 pointer references, allowing most objects to be read in just one disk access. I refer to it as the #trackSize, to remind me it is supposed to be how many bytes I think can the HD read in one operation without overrunning its own internal buffers and becoming inefficient. I was curious whether this number is optimized in 2010, so I ran the following script: ----------- |stats random| stats:=OrderedCollection new. random := Random new. nextPos:=100. (FileDirectory on: '/home/cmm/test3/cube.001.magma') fileNamed: 'objects.2.dat' do: [ : stream | | ba fileSize | ba := ByteArray new: 10000. fileSize := stream size. 100 to: 10000 by: 100 do: [ : n | stream position: 0. Transcript cr; show: (stats add: n->([stream maRead: n "bytes" bytesFromPosition: 1 of: ba atFilePosition: (random nextInt: fileSize ] bench)) ]]. stats ------------ Note that "objects.2.dat" is a real Magma file, 1.8GB in size. The goal of the script is bench how fast Squeak can read object buffers off the hard-drive when we obviously won't get many (if any) HD cache hits. I have a cheap, Western Digital Caviar HD, which produced the following output: 100->'119 per second.' 200->'98.5 per second.' 300->'106 per second.' 400->'106 per second.' 500->'101 per second.' 600->'102 per second.' 700->'99.9 per second.' 800->'103 per second.' 900->'104 per second.' 1000->'99 per second.' 1100->'97.9 per second.' 1200->'104 per second.' 1300->'111 per second.' 1400->'99.8 per second.' 1500->'107 per second.' 1600->'108 per second.' 1700->'95.6 per second.' 1800->'103 per second.' 1900->'108 per second.' 2000->'102 per second.' 2100->'103 per second.' 2200->'107 per second.' ... 3000->'98.7 per second.' 4000->'102 per second.' 5000->'106 per second.' 6000->'104 per second.' 7000->'101 per second.' 8000->'102 per second.' 9000->'102 per second.' 10000->'107 per second.' For curiousity, I also modified the script to read very small buffers from the HD, here are the results: 4->'137 per second.' 12->'146 per second.' 20->'154 per second.' 28->'143 per second.' (The HD busy light was solid ON during the test). At first I was puzzled because Magma has demonstrated much faster objects-per-second read rates than these, even including materialization, what gives? It's the HD buffering. Most of the time, objects are "clustered" closely together, so that reading one object causes the "next" object which will be read to already be in the HD's buffer. Here's the same script, except reading mostly "sequentially" through the file instead of from a random location: |stats random nextPos| stats:=OrderedCollection new. random := Random new. nextPos:=100. (FileDirectory on: '/home/cmm/test3/cube.001.magma') fileNamed: 'objects.2.dat' do: [ : stream | | ba fileSize | ba := ByteArray new: 10000. fileSize := stream size. #(4 12 20 28 100 200 300 400 500) [ : n | stream position: 0. Transcript cr; show: (stats add: n->([stream maRead: n "bytes" bytesFromPosition: 1 of: ba atFilePosition: ("random nextInt: fileSize" (nextPos := nextPos+n+10)) ] bench)) ]]. stats Now look at the results: "Reading sequentially rather than at a random position." 4->'1,160,000 per second.' 12->'1,210,000 per second.' 20->'1,100,000 per second.' 28->'973,000 per second.' ... 100->'1,030,000 per second.' 200->'321,000 per second.' 300->'215,000 per second.' 400->'160,000 per second.' 500->'227,000 per second.' Conclusions: - Hard-disk seek is definitely a bottleneck with Magma, or any Squeak application that requires random-access to a file. - When objects are clustered closely together, read performance can be dramatically better. - HD's with fast seek times, such as newer solid-state drives, might perform dramatically better. - I should consider reducing the trackSize from 280 bytes to ~100 bytes (or make it customizable); because the rate drops really fast after that and even a second read required could still be faster than an initial read. - Chris

2 1

Updating a 'live' environment
by Amir Ansari 17 Nov '10

17 Nov '10

What is the best practise for updating a 'live' environment? More specifically: suppose I have a database that is accessed by several instances of an application (say, a load-balanced Seaside app), but need to change the objects comprising the database in any of the following ways: (1) Add/remove instance variables to/from a class that is persisted in a MagmaCollection; (2) Add/remove indexes to/from a particular MagmaCollection (perhaps after the changes in (1)); (3) Re-factor some of the classes, thus making minor changes to the schema ... how could the database be updated? Is there also some way to update all the existing objects in the database to reflect the new changes? Amir

2 2

Magma collection - where:
by Facundo Vozzi 16 Nov '10

16 Nov '10

Hi, I've this code: - self customers select: [:aCustomer | aCustomer name asUppercase beginsWith: firstsLetterOfName asUppercase] It works ok and with good perfomance but "customers" is a magma collection so I would like use where: because the collection has a index on #name attribute. - self customers where: [:each | aCustomer name = firstsLetterOfName asUppercase] It works but I had two problems, the first one is that the meaningfulCharacters of MaSearchStringIndex is 5 and then it works if I write exactly the 5 firsts letter of the name. The second problem is that I can't send asUppercase beacuse name is a MaClause. So, my question is: Can I use where: for this funcionality or I need use select:? If you think that where: is appropiate, Do I need create a new king of index to reach that? Thank you, Facu p.s.: I'm using the first code in the method of customersOfNameBeginsWith: selector and i used it on a autocompletation jquery input control.

3 5

Magma, Seaside and Magrite integration
by Facundo Vozzi 15 Nov '10

15 Nov '10

Hi all, I added magma session to my application seaside session (a subclass of WASession) so each user has his own magma session. My problem is with magritte description because I need, on the class side, set list of options. By example: User class >>descriptionRol "Answer a magritte meta description for the receiver rol attribute." ^MAMultipleOptionDescription new accessor: #roles; label: 'Roles'; priority: 200; componentClass: MAListCompositonComponent; options: MySystem singleton roles; beSorted; yourself The problem is "MySystem singleton roles" beacuse is in other magma session when the user add or edit an instance of User on the seaside application. I "hack" my seaside components to override the options with a collection of instances (by example: Rol allInSession: aUserSession) inside of the user session but it's very ugly. Just a test. How do you configure the maggrite description to deal with this problem? Is a common practice add a new magma session in every session session? Thanks in advance, Facu

3 3

[ANN] Magma 1.2 alpha1
by Chris Muller 14 Nov '10

14 Nov '10

I have posted a new configuration of the latest Magma version. It is in the "MagmaTester" project of squeaksource, under 1.2alpha1. If you are doing Magma development, please use this version as there have been many performance improvements over the current "stable" version. For Pharo users, please load the attached WeakOrderedCollection, which Magma needs. This has been integrated into the Squeak trunk and I hope Pharo will do the same (just 3 new methods). I have been too busy to test Magma on the latest Pharo so let me know if the test suite doesn't pass. I'd like to reiterate that I support for Magma on this list. There have been cases of folks using Magma incorrectly and then drawing questionable conclusions before ever asking on this list. - Chris

3 5

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Magma November 2010