Magma

magma@lists.squeakfoundation.org

722 discussions

Start a nNew thread

MagmaPreallocatedDictionary
by Facundo Vozzi 13 Dec '10

13 Dec '10

Hi Chris, list, how can I do to send #select: to a MagmaPreallocatedDictionary? Thanks, Facu

2 3

Re: Magma with style
by Facundo Vozzi 12 Dec '10

12 Dec '10

Now I'm getting a performance problem. I'm bulk loading transactions in sets (from a file) of 1250. Each of one has references to 3 objects, 3 numbers and 2 dates. When I test it without setting the customer it take to me 614 milliseconds to create the 1250 instances but if I look up for each client according to the client code it take to me 204158 milliseconds (3.40 minutes). Customers is a MagmaCollection and this is the method of the message that I'm sending 1250 times: customerAtDocumentNumber: unString sex: otroString "Devuelve un invividuo o una organizacion dependiendo de si unString corresponde a un número de documento o a un número de cuit. NOTA: se debe tener en cuenta el sexo por existir números de dni repetidos entre varones y mujeres." | aFewCustomers | aFewCustomers := self customers where: [ :each | (each documentNumber = unString)]. ^customers detect: [:each | each documentNumber = unString and: [each sex = otroString]] ifNone: [] Customers has an index on documentNumber. Now I go to profile it but I need to find how to do it in Pharo. I thing that I can switch customers from MagmaCollection to MagmaDictionary and set the document number + sex how the key. What do you think? See you, Facu On Thu, Dec 9, 2010 at 7:36 PM, Facundo Vozzi <facundov79(a)gmail.com> wrote: > >> >> If you do a copy, then you will be creating multiple instances of >> various "system entities", which I'm guessing don't want to do; you >> would rather have just one instance, referenced from multiple >> transactions, right? >> > > Yes, you're right I don't want multiple instances. > >> >> So all you need to do is just look-up the *equivalent* Code in session >> which is doing the loading. It will be the one identical instance in >> the repository. >> > > Yes, to do that I'm passing the session which is doing the loading by > argument. But I think that my error here is that I'm implementing the > protocol to get one entitiy instance by code or name on the Entity class > side. I think is > better implement it, by example #clientOfCode: aCode, in my system instance > which is the root of each session. In this manner I don't need pass the > user session by argument. I think that my traditional database mapping > thinking betrayed me > > >> But yes, this is one of the use-cases for which FP's were invented, it >> can be a more elegant solution, especially if the number of Codes is >> always changing or increasing. See >> MagmaTestCase>>#testForwardingProxy. >> > > Yes, I will see it right now. > > Facu > > >> >> - Chris >> >> >> > Thanks for your help, >> > Facu >> > >> > On Thu, Dec 9, 2010 at 4:35 PM, Chris Muller <asqueaker(a)gmail.com> >> wrote: >> >> >> >> I'm not sure I fully understand the context, but a third option might >> >> be to use ForwardingProxies. Especially if the shared objects can be >> >> updated by only one session (like a batch updater, for example). Then >> >> every other (user client) session will share the same instances in >> >> memory. >> >> >> >> HTH, >> >> Chris >> >> >> >> On Wed, Dec 8, 2010 at 1:13 PM, Facundo Vozzi <facundov79(a)gmail.com> >> >> wrote: >> >> > Hi all, >> >> > I'm refactoring my code and I found that I have two ways to do the >> same >> >> > in >> >> > distint parts of my system. >> >> > 1) Entity class >> atCode: aCode >> >> > "Answer the receiver instance with code equal to aCode or evaluate >> >> > aBlock if >> >> > it doesn't exists." >> >> > ^(self all detect: [:one | one code = aCode]) copy >> >> > or >> >> > 2) Entity class>> atCode: aCode inSession: anUserSession >> >> > "Answer the receiver instance with code equal to aCode or evaluate >> >> > aBlock if >> >> > it doesn't exists." >> >> > >> >> > ^(self allInSession: anUserSession) detect: [:one | one code = >> >> > aCode] >> >> > The difference is that 1 is using the shared session so I need copy >> the >> >> > object to be used (referenced by other object) on anUserSession. The >> >> > uggly >> >> > of 2 is that I have to pass anUserSession as argument. >> >> > Either way works well, is really the same? >> >> > Thanks in advance, >> >> > Facu >> >> > (*) I eliminate aBlock for absent elements for this example >> >> > _______________________________________________ >> >> > Magma mailing list >> >> > Magma(a)lists.squeakfoundation.org >> >> > http://lists.squeakfoundation.org/mailman/listinfo/magma >> >> > >> >> > >> > >> > >> > >

3 8

Magma with style
by Facundo Vozzi 09 Dec '10

09 Dec '10

Hi all, I'm refactoring my code and I found that I have two ways to do the same in distint parts of my system. 1) Entity class >> atCode: aCode "Answer the receiver instance with code equal to aCode or evaluate aBlock if it doesn't exists." ^(self all detect: [:one | one code = aCode]) copy or 2) Entity class>> atCode: aCode inSession: anUserSession "Answer the receiver instance with code equal to aCode or evaluate aBlock if it doesn't exists." ^(self allInSession: anUserSession) detect: [:one | one code = aCode] The difference is that 1 is using the shared session so I need copy the object to be used (referenced by other object) on anUserSession. The uggly of 2 is that I have to pass anUserSession as argument. Either way works well, is really the same? Thanks in advance, Facu (*) I eliminate aBlock for absent elements for this example

2 2

Help for local repository usage
by Denis Kudriashov 09 Dec '10

09 Dec '10

Hello. I use magma db for local usage. Should I use only single session? I ask because now I use separate sessions for processing different kind of objects in my root dictionary. At application startup I open 4 sessions for each kind of objects. And I have many processes that operate objects by this sessions. Problem that sometimes strange magma exception MagmaAlreadyInTransaction happen. I cant understand reason because magma sessions is thread safelly. Best regards, Denis

2 1

[ANN] ConfigurationOfMagma for Pharo 1.0, Pharo 1.1 and Pharo 1.2
by Miguel Enrique Cobá Martínez 08 Dec '10

08 Dec '10

Hi all, a new version of ConfigurationOfMagma is available for Pharo. Notes: Pharo 1.0 ---------------------- - 1.1r1 is the last versions that works in Pharo 1.0. This is because upstream Magma code depends on code that only exists on Pharo 1.1 and superior. - Pharo 1.0 uses Preferences to enable/disable the underscore as assigment in code. This release captures current value of this preference. Enables it for loading Magma and restores it to the original value after installing Magma. Pharo 1.1, Pharo 1.2 ---------------------- - The configuration for these Pharo releases is based on the one from Pharo 1.0. - 1.1r2 uses a new baseline because Magma has a new package dependency (Ma files additions). - Pharo 1.1 and superior uses Scanner allowUnderscoreAsAssignment instead of Preferences. The current value is captured and restored after installation of Magma. - Pharo 1.1 and superior doesn't include FlapTab class anymore. Magma adds some methods to it so the Configuration adds an empty class named FlapTab before installing and removes from the system after installing. This avoid a warning prompt when installing Magma. - This Configuration includes support for installing Magma releases previous to 1.1r2. Those versions should install correctly on Pharo 1.1 and greater but if a lot of problems are detected, the support for those old versions will be removed, and only 1.1r2 and newer versions will be supported. Now that Pharo has Universes, Magma has different ConfigurationOfMagma in each universe for Pharo. Each of those Configuration install the appropriate packages in Pharo. Installing Magma Pharo 1.0 --------------- Evaluate: Gofer new squeaksource: 'UniverseForPharo10'; package: 'ConfigurationOfMagma'; load. And then, one of: ConfigurationOfMagma project latestVersion load: 'Client'. ConfigurationOfMagma project latestVersion load: 'Server'. ConfigurationOfMagma project latestVersion load: 'Tester'. latestVersion is 1.1r1 Pharo 1.1 --------------- Evaluate: Gofer new squeaksource: 'UniverseForPharo11'; package: 'ConfigurationOfMagma'; load. And then, one of: ConfigurationOfMagma project latestVersion load: 'Client'. ConfigurationOfMagma project latestVersion load: 'Server'. ConfigurationOfMagma project latestVersion load: 'Tester'. latestVersion is 1.1r2 Pharo 1.2 --------------- Evaluate: Gofer new squeaksource: 'UniverseForPharo12'; package: 'ConfigurationOfMagma'; load. And then, one of: ConfigurationOfMagma project latestVersion load: 'Client'. ConfigurationOfMagma project latestVersion load: 'Server'. ConfigurationOfMagma project latestVersion load: 'Tester'. latestVersion is 1.1r2 Comments and bugs accepted. Enjoy. -- Miguel Cobá http://miguel.leugim.com.mx

4 4

Re: Magma upgrater
by Chris Muller 04 Dec '10

04 Dec '10

Try this: MagmaSession initialize and then run your upgrade. I've seen this error a couple of times, now I think what's happening is that the class-initialization for MaObjectBuffer got run sometime _after_ MagmaSession. Under normal loading procedures that would not happen, but if a new version of Ma object serialization was loaded after Magma, it could happen. Let me know, Chris On Sun, Nov 28, 2010 at 3:50 PM, Facundo Vozzi <facundov79(a)gmail.com> wrote: > Sorry, I forgot attach the error. There you go. > Thanks, > Facu > > > On Sun, Nov 28, 2010 at 2:57 PM, Chris Muller <ma.chris.m(a)gmail.com> wrote: >> >> What error? Screenshot? >> >> On Fri, Nov 26, 2010 at 3:17 PM, Facundo Vozzi <facundov79(a)gmail.com> >> wrote: >> > Hi Chris, >> > I did it but I'm getting the error that adjunt in the screnshot. Do you >> > know >> > what I'm doing wrong? >> > Thanks you, >> > Facundo >> > On Fri, Nov 26, 2010 at 3:33 PM, Chris Muller <asqueaker(a)gmail.com> >> > wrote: >> >> >> >> Hi Facundo, please use the "Magma upgrader" package available on the >> >> "Magma tester" project of squeaksource. >> >> >> >> After loading it: >> >> >> >> (MagmaUpgrader source: '/path/to/old/repository') compressTo: >> >> '/path/to/upgraded/repository' >> >> >> >> - Chris >> >> >> >> On Fri, Nov 26, 2010 at 6:54 AM, Facundo Vozzi <facundov79(a)gmail.com> >> >> wrote: >> >> > Hi, >> >> > how is the way for upgrade my magma repository from version 12 >> >> > to version 14 >> >> > in order to use it with the version 1.2a1 of magma? >> >> > Someone was able to test version 1.2a1 in Pharo 1.1.1? >> >> > Thanks in advance, >> >> > Facu >> >> > >> >> > _______________________________________________ >> >> > Magma mailing list >> >> > Magma(a)lists.squeakfoundation.org >> >> > http://lists.squeakfoundation.org/mailman/listinfo/magma >> >> > >> >> > >> > >> > > >

4 6

Re: hard-drive read-performance
by Chris Muller 29 Nov '10

29 Nov '10

Yep, it's called MagmaCompressor. But still, if you have a multi-gigabyte repository and a HD buffer of, what, a few K? or even a client-cache of a 100MB, there is no way around needing to read off the HD.. On Sun, Nov 28, 2010 at 8:31 PM, Elliot Finley <efinley.lists(a)gmail.com> wrote: > maybe a defrag utility for Magma that places all objects in a collection > close together on the disk? > > On Wed, Nov 24, 2010 at 10:00 AM, Chris Muller <ma.chris.m(a)gmail.com> wrote: >> >> When reading any object off the hard drive (represented as the >> 'byteArray' of a single MaObjectBuffer), Magma always reads 280 bytes. >> Since the #physicalSize is in the object header, it is then able to >> check the contents of the buffer to determine the size of the whole >> object and, if necessary, read more bytes in order to get the whole >> object. See MaObjectFiler>>#read:bytesInto:and:startingAt:filePosition: >> for this behavior. >> >> 280 bytes is enough for about 40 pointer references, allowing most >> objects to be read in just one disk access. I refer to it as the >> #trackSize, to remind me it is supposed to be how many bytes I think >> can the HD read in one operation without overrunning its own internal >> buffers and becoming inefficient. I was curious whether this number >> is optimized in 2010, so I ran the following script: >> >> ----------- >> |stats random| stats:=OrderedCollection new. random := Random new. >> nextPos:=100. >> (FileDirectory on: '/home/cmm/test3/cube.001.magma') fileNamed: >> 'objects.2.dat' do: >> [ : stream | | ba fileSize | ba := ByteArray new: 10000. >> fileSize := stream size. >> 100 to: 10000 by: 100 do: >> [ : n | >> stream position: 0. >> Transcript cr; show: (stats add: n->([stream >> maRead: n "bytes" >> bytesFromPosition: 1 >> of: ba >> atFilePosition: (random nextInt: fileSize ] >> bench)) ]]. >> stats >> ------------ >> >> Note that "objects.2.dat" is a real Magma file, 1.8GB in size. The >> goal of the script is bench how fast Squeak can read object buffers >> off the hard-drive when we obviously won't get many (if any) HD cache >> hits. >> >> I have a cheap, Western Digital Caviar HD, which produced the following >> output: >> >> 100->'119 per second.' >> 200->'98.5 per second.' >> 300->'106 per second.' >> 400->'106 per second.' >> 500->'101 per second.' >> 600->'102 per second.' >> 700->'99.9 per second.' >> 800->'103 per second.' >> 900->'104 per second.' >> 1000->'99 per second.' >> 1100->'97.9 per second.' >> 1200->'104 per second.' >> 1300->'111 per second.' >> 1400->'99.8 per second.' >> 1500->'107 per second.' >> 1600->'108 per second.' >> 1700->'95.6 per second.' >> 1800->'103 per second.' >> 1900->'108 per second.' >> 2000->'102 per second.' >> 2100->'103 per second.' >> 2200->'107 per second.' >> ... >> 3000->'98.7 per second.' >> 4000->'102 per second.' >> 5000->'106 per second.' >> 6000->'104 per second.' >> 7000->'101 per second.' >> 8000->'102 per second.' >> 9000->'102 per second.' >> 10000->'107 per second.' >> >> For curiousity, I also modified the script to read very small buffers >> from the HD, here are the results: >> >> 4->'137 per second.' >> 12->'146 per second.' >> 20->'154 per second.' >> 28->'143 per second.' >> >> (The HD busy light was solid ON during the test). >> >> At first I was puzzled because Magma has demonstrated much faster >> objects-per-second read rates than these, even including >> materialization, what gives? >> >> It's the HD buffering. Most of the time, objects are "clustered" >> closely together, so that reading one object causes the "next" object >> which will be read to already be in the HD's buffer. Here's the same >> script, except reading mostly "sequentially" through the file instead >> of from a random location: >> >> |stats random nextPos| stats:=OrderedCollection new. random := Random new. >> nextPos:=100. >> (FileDirectory on: '/home/cmm/test3/cube.001.magma') fileNamed: >> 'objects.2.dat' do: >> [ : stream | | ba fileSize | ba := ByteArray new: 10000. >> fileSize := stream size. >> #(4 12 20 28 100 200 300 400 500) >> [ : n | >> stream position: 0. >> Transcript cr; show: (stats add: n->([stream >> maRead: n "bytes" >> bytesFromPosition: 1 >> of: ba >> atFilePosition: ("random nextInt: fileSize" >> (nextPos := >> nextPos+n+10)) ] bench)) ]]. >> stats >> >> Now look at the results: >> >> "Reading sequentially rather than at a random position." >> 4->'1,160,000 per second.' >> 12->'1,210,000 per second.' >> 20->'1,100,000 per second.' >> 28->'973,000 per second.' >> ... >> 100->'1,030,000 per second.' >> 200->'321,000 per second.' >> 300->'215,000 per second.' >> 400->'160,000 per second.' >> 500->'227,000 per second.' >> >> Conclusions: >> >> - Hard-disk seek is definitely a bottleneck with Magma, or any >> Squeak application that requires random-access to a file. >> - When objects are clustered closely together, read performance can >> be dramatically better. >> - HD's with fast seek times, such as newer solid-state drives, might >> perform dramatically better. >> - I should consider reducing the trackSize from 280 bytes to ~100 >> bytes (or make it customizable); because the rate drops really fast >> after that and even a second read required could still be faster than >> an initial read. >> >> - Chris >> _______________________________________________ >> Magma mailing list >> Magma(a)lists.squeakfoundation.org >> http://lists.squeakfoundation.org/mailman/listinfo/magma > >

1 0

Magma read performance
by Chris Muller 29 Nov '10

29 Nov '10

There has been interest in Magma's read and query performance lately, so I thought I would share results of a recent benchmark test. It's an actual application which does very heavy reading and writing to a Magma repository. This test was 24GB of repository work, over two days. My goal was to determine, once the persistent model becomes many times the size of RAM and HD access appeared to become the limiting factor: - how fast is Magma, in terms of "objects per second" and in "kilobytes per second"? - how fast is this relative to the speed when the repository was empty? Conclusion: - Magma started at 4K objects per second (empty repository), 282K per second. - Finished with 2.3K objects per second (6GB repository), 750K per second. - Memory consumption by the image never exceeded 300MB. It is important to note, these times are from the client MagmaSession point-of-view, including the full server roundtrip plus materialization. Also, as can be seen from the attached data, there were many requests which only brought back one or two objects which, while dramatically lowering the overall reported throughput, is actually a real-world scenario for applications. Verbose description: When the test first started the repository was tiny, and there were only ~4K server requests during a 5-minute monitored period. The total number of objects read across all ~4K requests was ~17M for an average throughput of 4K objects per second (ops). As the model grew in size and complexity, more and more objects during the 5-minute monitored intervals were required to perform application-posting of the same number of input records; doubling to ~8K server requests during a 5-minute period, however only a few more objects brought back (total 21M), for an average of 2500 ops. 32 hours after that, the rate of reads dropped to about 2300 ops. This is due to two factors: - the objects were larger (e.g., more pointers to other objects) - they were less clustered (having been replaced with objects which could not be co-located with the original object) The fact that objects got bigger (e.g., more pointers to other objects) is corroborated by another stat, "the number of *bytes* per second" read off the HD in order to access those objects. Initially, when the test first started, there were about 4K *requests* for objects in one of the first five-minute periods, which were read at a rate of 282K per second. 24 hours later, there were only 2 times as many requests for objects, but were read-and-materialized at a rate of 750K bytes / second. - Chris

3 4

Magma upgrater
by Facundo Vozzi 26 Nov '10

26 Nov '10

Hi, how is the way for upgrade my magma repository from version 12 to version 14 in order to use it with the version 1.2a1 of magma? Someone was able to test version 1.2a1 in Pharo 1.1.1? Thanks in advance, Facu

2 1

hard-drive read-performance
by Chris Muller 24 Nov '10

24 Nov '10

When reading any object off the hard drive (represented as the 'byteArray' of a single MaObjectBuffer), Magma always reads 280 bytes. Since the #physicalSize is in the object header, it is then able to check the contents of the buffer to determine the size of the whole object and, if necessary, read more bytes in order to get the whole object. See MaObjectFiler>>#read:bytesInto:and:startingAt:filePosition: for this behavior. 280 bytes is enough for about 40 pointer references, allowing most objects to be read in just one disk access. I refer to it as the #trackSize, to remind me it is supposed to be how many bytes I think can the HD read in one operation without overrunning its own internal buffers and becoming inefficient. I was curious whether this number is optimized in 2010, so I ran the following script: ----------- |stats random| stats:=OrderedCollection new. random := Random new. nextPos:=100. (FileDirectory on: '/home/cmm/test3/cube.001.magma') fileNamed: 'objects.2.dat' do: [ : stream | | ba fileSize | ba := ByteArray new: 10000. fileSize := stream size. 100 to: 10000 by: 100 do: [ : n | stream position: 0. Transcript cr; show: (stats add: n->([stream maRead: n "bytes" bytesFromPosition: 1 of: ba atFilePosition: (random nextInt: fileSize ] bench)) ]]. stats ------------ Note that "objects.2.dat" is a real Magma file, 1.8GB in size. The goal of the script is bench how fast Squeak can read object buffers off the hard-drive when we obviously won't get many (if any) HD cache hits. I have a cheap, Western Digital Caviar HD, which produced the following output: 100->'119 per second.' 200->'98.5 per second.' 300->'106 per second.' 400->'106 per second.' 500->'101 per second.' 600->'102 per second.' 700->'99.9 per second.' 800->'103 per second.' 900->'104 per second.' 1000->'99 per second.' 1100->'97.9 per second.' 1200->'104 per second.' 1300->'111 per second.' 1400->'99.8 per second.' 1500->'107 per second.' 1600->'108 per second.' 1700->'95.6 per second.' 1800->'103 per second.' 1900->'108 per second.' 2000->'102 per second.' 2100->'103 per second.' 2200->'107 per second.' ... 3000->'98.7 per second.' 4000->'102 per second.' 5000->'106 per second.' 6000->'104 per second.' 7000->'101 per second.' 8000->'102 per second.' 9000->'102 per second.' 10000->'107 per second.' For curiousity, I also modified the script to read very small buffers from the HD, here are the results: 4->'137 per second.' 12->'146 per second.' 20->'154 per second.' 28->'143 per second.' (The HD busy light was solid ON during the test). At first I was puzzled because Magma has demonstrated much faster objects-per-second read rates than these, even including materialization, what gives? It's the HD buffering. Most of the time, objects are "clustered" closely together, so that reading one object causes the "next" object which will be read to already be in the HD's buffer. Here's the same script, except reading mostly "sequentially" through the file instead of from a random location: |stats random nextPos| stats:=OrderedCollection new. random := Random new. nextPos:=100. (FileDirectory on: '/home/cmm/test3/cube.001.magma') fileNamed: 'objects.2.dat' do: [ : stream | | ba fileSize | ba := ByteArray new: 10000. fileSize := stream size. #(4 12 20 28 100 200 300 400 500) [ : n | stream position: 0. Transcript cr; show: (stats add: n->([stream maRead: n "bytes" bytesFromPosition: 1 of: ba atFilePosition: ("random nextInt: fileSize" (nextPos := nextPos+n+10)) ] bench)) ]]. stats Now look at the results: "Reading sequentially rather than at a random position." 4->'1,160,000 per second.' 12->'1,210,000 per second.' 20->'1,100,000 per second.' 28->'973,000 per second.' ... 100->'1,030,000 per second.' 200->'321,000 per second.' 300->'215,000 per second.' 400->'160,000 per second.' 500->'227,000 per second.' Conclusions: - Hard-disk seek is definitely a bottleneck with Magma, or any Squeak application that requires random-access to a file. - When objects are clustered closely together, read performance can be dramatically better. - HD's with fast seek times, such as newer solid-state drives, might perform dramatically better. - I should consider reducing the trackSize from 280 bytes to ~100 bytes (or make it customizable); because the rate drops really fast after that and even a second read required could still be faster than an initial read. - Chris

2 1

← Newer
1
...
8
9
10
11
12
13
14
...
73
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Magma