Magma

magma@lists.squeakfoundation.org

722 discussions

Magma tester-cmm.184
by Chris Muller 18 Aug '05

18 Aug '05

>From the package version notes: - renamed MaIndexDefinition to MagmaCollectionIndex. Renamed all indexes ending in "Definition", chopped it. - To support a significant performance improvement, all MagmaCollectionIndexes must have their recordSize set to a power of two. Thanks Andrew Gaylard! - Moved MaHashIndex test-cases back into the special-collections package. - Minor fixes and cleanups. - Here are the latest benchmarks with the new MaHashIndex performance improvement: 'The date is 17 August 2005 10:21:22 pm Hardware Details: computer : IBM R40 cpu : Pentium-M speed : 1.3GHz memory : 768MB disk : internal IDE OS Details: osVersion : NT platformName : Win32 platformSubtype : IX86 vmVersion : Squeak3.7 of ''4 September 2004'' [latest update: #5989] imageName : C:\Development\Chris\Development\Squeak\latest3.8.image Image Details: version : Squeak3.8 lastUpdate : 6665 Code Package Details (from Monticello): ** modified ** Name: Magma tester-cmm.181 Author: cmm Time: 12 August 2005, 5:23:19 pm UUID: 60a56420-0e4a-934b-8ec9-83141ce9116c Ancestors: Magma tester-cmm.180 Magma server details: flushAggressively : false MagmaSession Details: isLocal : true allowWriteBarrier : false refreshPersistentObjectsEvenWhenChangedOnlyByMe : false Benchmarker Details: thousands : 1000 Benchmarks: --- readTests peakRefreshRate : 4679.73635288153 per second. singleObjectRead : 3240.15196960608 per second. oneThousandElementArrayRead : 211.4308553157474 per second. oneThousandElementArrayOfObjectsRead : 7.007981312049835 per second. oneMillionObjectPointersRead : 5 seconds. oneThousandLevelsDeepRead : 5.91715976331361 per second. --- writeTests peakCommitRate : 23.92821535393819 per second. oneThousandElementArrayCommit : 17.5438596491228 per second. a1001BufferCommit : 9.98003992015968 per second. aOneMillionBufferCommit : 16 seconds. --- magmaArrayTests getMagmaArraySize : 8824.4351129774 per second. updateThousandsOfMagmaArrayElements : 0.1958863858961802 per second. --- magmaCollectionTests addThousandsOfObjectsTenAtATime : 8.376 seconds. addThousandsOfObjectsOneHundredAtATime : 4.458 seconds. addThousandsOfObjectsAtOnce : 9.258 seconds.' 'The date is 17 August 2005 10:36:36 pm Hardware Details: computer : IBM R40 cpu : Pentium-M speed : 1.3GHz memory : 768MB disk : internal IDE OS Details: osVersion : NT platformName : Win32 platformSubtype : IX86 vmVersion : Squeak3.7 of ''4 September 2004'' [latest update: #5989] imageName : C:\Development\Chris\Development\Squeak\latest3.8.image Image Details: version : Squeak3.8 lastUpdate : 6665 Code Package Details (from Monticello): ** modified ** Name: Magma tester-cmm.181 Author: cmm Time: 12 August 2005, 5:23:19 pm UUID: 60a56420-0e4a-934b-8ec9-83141ce9116c Ancestors: Magma tester-cmm.180 Magma server details: flushAggressively : false MagmaSession Details: isLocal : true allowWriteBarrier : true refreshPersistentObjectsEvenWhenChangedOnlyByMe : false Benchmarker Details: thousands : 1000 Benchmarks: --- readTests peakRefreshRate : 4599.15824915825 per second. singleObjectRead : 3179.564087182564 per second. oneThousandElementArrayRead : 215.2708374975015 per second. oneThousandElementArrayOfObjectsRead : 6.9916100679185 per second. oneMillionObjectPointersRead : 5 seconds. oneThousandLevelsDeepRead : 5.95592614651578 per second. --- writeTests peakCommitRate : 92.1078921078921 per second. oneThousandElementArrayCommit : 39.4108280254777 per second. a1001BufferCommit : 23.35329341317365 per second. aOneMillionBufferCommit : 17 seconds. --- magmaArrayTests getMagmaArraySize : 7832.83343331334 per second. updateThousandsOfMagmaArrayElements : 0.194024058983314 per second. --- magmaCollectionTests addThousandsOfObjectsTenAtATime : 5.001 seconds. addThousandsOfObjectsOneHundredAtATime : 3.898 seconds. addThousandsOfObjectsAtOnce : 10.157 seconds.' 'The date is 17 August 2005 11:03:51 pm Hardware Details: computer : IBM R40 cpu : Pentium-M speed : 1.3GHz memory : 768MB disk : internal IDE OS Details: osVersion : NT platformName : Win32 platformSubtype : IX86 vmVersion : Squeak3.7 of ''4 September 2004'' [latest update: #5989] imageName : C:\Development\Chris\Development\Squeak\latest3.8.image Image Details: version : Squeak3.8 lastUpdate : 6665 Code Package Details (from Monticello): ** modified ** Name: Magma tester-cmm.181 Author: cmm Time: 12 August 2005, 5:23:19 pm UUID: 60a56420-0e4a-934b-8ec9-83141ce9116c Ancestors: Magma tester-cmm.180 MagmaSession Details: isLocal : false allowWriteBarrier : false refreshPersistentObjectsEvenWhenChangedOnlyByMe : false Benchmarker Details: thousands : 1000 Benchmarks: --- readTests peakRefreshRate : 229.3082766893243 per second. singleObjectRead : 247.6514091545073 per second. oneThousandElementArrayRead : 114.5937312836894 per second. oneThousandElementArrayOfObjectsRead : 7.578779417630634 per second. oneMillionObjectPointersRead : 7 seconds. oneThousandLevelsDeepRead : 5.238649592549476 per second. --- writeTests peakCommitRate : 16.90869305748956 per second. oneThousandElementArrayCommit : 12.97405189620759 per second. a1001BufferCommit : 7.50394944707741 per second. aOneMillionBufferCommit : 32 seconds. --- magmaArrayTests getMagmaArraySize : 47.75716694772344 per second. updateThousandsOfMagmaArrayElements : 0.2635046113306983 per second. --- magmaCollectionTests addThousandsOfObjectsTenAtATime : 11.912 seconds. addThousandsOfObjectsOneHundredAtATime : 5.69 seconds. addThousandsOfObjectsAtOnce : 11.47 seconds.' 'The date is 17 August 2005 11:18:51 pm Hardware Details: computer : IBM R40 cpu : Pentium-M speed : 1.3GHz memory : 768MB disk : internal IDE OS Details: osVersion : NT platformName : Win32 platformSubtype : IX86 vmVersion : Squeak3.7 of ''4 September 2004'' [latest update: #5989] imageName : C:\Development\Chris\Development\Squeak\latest3.8.image Image Details: version : Squeak3.8 lastUpdate : 6665 Code Package Details (from Monticello): ** modified ** Name: Magma tester-cmm.181 Author: cmm Time: 12 August 2005, 5:23:19 pm UUID: 60a56420-0e4a-934b-8ec9-83141ce9116c Ancestors: Magma tester-cmm.180 MagmaSession Details: isLocal : false allowWriteBarrier : true refreshPersistentObjectsEvenWhenChangedOnlyByMe : false Benchmarker Details: thousands : 1000 Benchmarks: --- readTests peakRefreshRate : 226.109556177529 per second. singleObjectRead : 243.1027588964414 per second. oneThousandElementArrayRead : 113.8406231276213 per second. oneThousandElementArrayOfObjectsRead : 7.55317034386802 per second. oneMillionObjectPointersRead : 7 seconds. oneThousandLevelsDeepRead : 5.19376747902517 per second. --- writeTests peakCommitRate : 36.97042366107114 per second. oneThousandElementArrayCommit : 23.14445331205108 per second. a1001BufferCommit : 14.6883684001588 per second. aOneMillionBufferCommit : 26 seconds. --- magmaArrayTests getMagmaArraySize : 19.23076923076923 per second. updateThousandsOfMagmaArrayElements : 10 --- magmaCollectionTests addThousandsOfObjectsTenAtATime : 8.015 seconds. addThousandsOfObjectsOneHundredAtATime : 5.559 seconds. addThousandsOfObjectsAtOnce : 10.839 seconds.' Enjoy, Chris

3 2

Possible Uses of Magma
by Daniel Salama 17 Aug '05

17 Aug '05

I've been playing with the newer release of Magma. So far my tests have gone well. However, I have a few questions of applicability that I hope someone can help me answer. First, I have a need to deploy an application for a small call center, where every call is recorded. Currently, the system creates an entry in a MySQL table indicating the source of the call (caller- id in case of someone calling in, or the extension of the agent in case the agent is making an outbound call), the destination (again, in the case of an outbound call, it would be the destination number, or in the case of an incoming call, it would be the agent's extension number), a timestamp, and a file name. When a call comes in or is made, the system makes an entry in this table and stores an MP3 file in a directory of the system. What I was wondering if it would be applicable and make sense, is to store all this information in a Magma database. Meaning, store the call meta information as well as the actual audio content. At first, I don't think there would be a problem. However, I was looking at the current system and they have approximately 200GB of audio content stored in over 100,000 files (in a period of 6 months). As additional requirements, the system should allow for the "easy" search of call records by either date, agent, or phone number. Also, the actual call "object" should be stored in the customer profile so that when reviewing the customer record, historical call recordings can be pulled. Is this too much to ask for? Will searches be too slow for such volume of objects being stored? The way I see it, there really isn't a complex object hierarchy being stored. It's mostly flat list. The second question of applicability I have is something that came to mind when reading the book "The Data Model Resource Book" by Len Silverston. This two-volume book shows universal data models applicable to most industries. Their target is relational databases but their models are easier represented as complex object models. As a matter of fact, they go through very complex relational data models in order to represent things I think would be much easier represented with object models. Anyway, just as I went through the book, I figured it would be interesting to model some of their ideas using Squeak and Magma, which brought me to the following questions based on some generalized ideas of an object model: The book "models" people, companies, entities, etc as a general object called a Party. Therefore, a party could be "sub-typed" as a Person, Organization, etc, where a Person can be further "sub-typed" into entities like Employee, Contractor, Family Member, Contact, Customer, Prospect, etc. Organizations can also be further sub-typed into things like Internal Organizations, Competitors, Agents, Distributors, Supplier, Customers, etc. You get the idea. Basically, each of these entities can be continuously be specialized into more and more specific models, as the need arises, and the application's business rules should know how these objects should/could interact. Not that I'm planning on using these types of models for real life applications, but it is certainly joyful reading about it. They definitely made me think about how I could have modeled some of the apps I've written in the past. Keeping the above general description of such a tiny portion of the object model in mind, I thought, well, I could definitely create a Party class with many different subclasses that can be each further subclassed and so on and so forth. Party class could store general attributes and methods common to all, including knowledge of how to persist itself and search, etc. At first, it sounds interesting, but then when the needs of searching comes to surface, there may be a chocking point in performance. Assuming a busy company uses this system, they could end up with several hundreds of thousands or millions of Party objects created relatively quickly. A user wanting to search for a customer, could specify a search criteria by, say, last name begins with "XXXX". How do you think Magma would perform? The way I think I would do this is to store all Party objects in a large collection with additional collections acting as "indices" for customers, etc. However, reading up on Magma's docs, I came across the MagmaCollectionReader and the ability to create customer indices. Would this work better? Could I create a customer index in Magma that would only store sub-classes of Party? For example, an index that would allow me to search by phone number of the class Customer, which is a sub-class of Party? Sorry for the long post. I just have a lot in my mind and was hoping someone with more experience could either answer my questions or at least point me in the right direction. Thanks, Daniel

3 3

Re: Possible Use of Magma
by Brent Pinkney 15 Aug '05

15 Aug '05

Hi, You should ask this on the magma mailing list. > Storing call records as MP3 file. Trivial. Magma can store any arbitrary object up to (just under) 16MB. > As additional requirements, the system should allow for the "easy" > search of call records by either date, agent, or phone number. Also, the > actual call "object" should be stored in the customer profile so that > when reviewing the customer record, historical call recordings can be > pulled. This is not a problem, store your CellRecods in a MagmaCollection. A MagmaCollection is like an OrderedCollection but can have any number of indices. An indexed search can be very fast. Brent

1 0

security requirements
by Chris Muller 09 Aug '05

09 Aug '05

After a lot of reading and thinking a lot about security for Magma, some vague ideas have formed in my head for requirements: #1 it should punish senders of bad, mal-formed or forged requests. #2 Byte and word buffers should be encrypted on disk in case the file-system is compromised. #3 messages should be secure across the public network full of nosey wire-tappers. #4 Security within the object-model (for business applications - individual object-access based on role hierarchy). Are there any other requirements I should be considering? Can secure sockets handle #3? If not encryption/decryption would burden the program way up in the Smalltalk layer; not good. I've given some thought to implementation for these but am mostly focused on #1 at this time. I've posed an invitation for discussion about #1 on squeak-dev, but what does anyone think of these other requirements? - Chris

1 0

Monticello remote repository template bug
by Michael Roberts 27 Jul '05

27 Jul '05

Hi, the template for the remote repository MCRemoteMagmaRepository class host: 'localhost' port: 51969 incorrectly sends #host:port: to Metaclass. Do you want to drop the sending of #species in #creationTemplate? If you manually change it it works fine. Cheers Mike

2 2

A quick DNU question
by Michael Roberts 27 Jul '05

27 Jul '05

Hi, I've loaded Magma client (125) and server (105) from Monticello into separate 3.8 images and I get a DNU in the client image when I try and connect. The client eventually ends up in MaOidManager>>getNextOid which sends #lastUserObjectOid to MaOidCalculator. This isn't defined in my client image. However, it seems that it is extended in Magma server. Does this need to be pushed down into Magma client? Also, is there a way of knowing which versions are considered stable? I just grabbed the latest versions but I would grab more 'blessed' versions if these numbers were indicated somewhere? Thanks! Mike

2 5

Magma-tester.cmm.162
by Chris Muller 27 Jul '05

27 Jul '05

I have posted a new version with the four bug fixes integrated. Thanks Brent and Michael. - Chris

1 0

[BUG] [FIX] MaIntegerIndex >> keySize
by Brent Pinkney 27 Jul '05

27 Jul '05

Hi Chris, Can you please ensure that MaIntegerIndex implements #keySize. Cerrently it is not implemented, but the parent class MaIndexDefinition calls #subclassResponsibility in its implementation of #keySize. Something like: MaIntegerIndex >> keySize ^ keySize ! Thanks Brent Pinkney

2 1

Re: memory
by Chris Muller 26 Jul '05

26 Jul '05

> 2. Traversing a Reader 'hogs' memory on the server > -------------------------------------------------- > This is our biggest problem. If I have a script which iterates over a > reader, doing something with the tree reachable each object and then > stubbing it out at the bottom of the loop, the server image's memory > footprint just keeps growing. The client image's memory is stable. > > e.g. > > | reader | > reader := (session root things read: #name). > 1 to: reader size do: [ :i | | thing | > thing := reader at: i. > "do other things with thing here..." > "materialize a lot of objects but no transactions" > session stubOut: thing. > session finalizeOids. > Smalltalk garbageCollect. > ]. I did find a memory "leak" that occurs for commits, I'll explain. We must now re-serialize commit-packages to support disaster-recovery, I was not "finalizingOids" of the serializer that takes care of this work. The attached change-set corrects this. (Note, in this case, you'll see I was actually able to use #resetOidManager, which is quicker for clean-up than #finalizeOids, but you not usable for a MagmaSessions serializer..). I was not able to reproduce the case above only enumerating a reader to cause the server to leak memory.. Let me share my test scripts and, if it is still happening for you, perhaps you can modify the scripts such that they recreate the problem and then send them back to me.. ** Script 1 - Create a huge MagmaSet of UUID's, indexed on #asString36. | path s set | path _ 'c:\temp\bigger'. MagmaRepositoryController create: path root: (MagmaSet equivalenceIndex: ((MaSearchStringIndexDefinition attribute: #asString36) keySize: 192 ; yourself)). s _ MagmaSession openLocal: path. s connectAs: 'cmm'. s begin. set _ s root. 1 to: 1000000 do: [ : n | set add: UUID new. n \\ 100 = 0 ifTrue: [ s commitAndBegin ; finalizeOids ] ]. s commit. s NOTE, in the interests of time, I broke this after it had added 493000+ UUID's to run the next scripts.. ** Script 2 - Start the server ** path _ 'c:\temp\bigger'. MagmaServerConsole new open: path ; processOn: 1010 ** Script 3 - Enumerate (separate image) ** | path s set reader | path _ 'c:\temp\bigger'. s _ MagmaSession hostAddress: #(127 0 0 1) asByteArray port: 1010. s connectAs: 'chris'. set _ s root. reader := (s root read: #asString36). 1 to: reader size do: [ :i | | thing | thing := reader at: i. "do other things with thing here..." "materialize a lot of objects but no transactions" i\\1000=0 ifTrue: [ Transcript show: '.' ]. ]. s I let this run but the server hovered at 30MB constant for several minutes before I interrupted it. My guess is you probably had the commit memory-leak affecting you. The attached change-set seems to fix it. I hope it fixes your memory-leak issue.. - Chris

1 0

Magma update
by Chris Muller 26 Jul '05

26 Jul '05

I just thought I'd describe where Magma is these days. Besides significant improvement in stability, the last year has seen several features added: == near-Universal object support == Almost any kind of object is supported. The only ones I still need to fix are subclasses of Set and Dictionary (such as PluggableSet and PluggableDictionary). However, Blocks and variableWord and even Morphs and Projects seem to work. == Collaborative development == You can run your image "out-of-sync" with respect to class definitions in repository's you're connected to. Instance variables, like standard Smalltalk, are mapped by name. Magma will even warn you if you are about to truncate an instance variable on account of your in-image definition not having a particular instance variable. You can also use Magma as a code-repository. Classes, ChangeSets and Monticello packages can be saved to a special area that does not require a connection in order to download and install the code. This way repositories can/should be built to contain all the objects AND code needed to access those objects (self contained repository). The Monticello GUI can be used to directly load/save packages. == Remote proxies == Magma is intended to run client/server AND peer-to-peer simultaneously. You can have your own personal database in which you may wish to loosely reference an object in another repository. Just send #maAsForwardingProxy to the remote object and commit the result to your repository. Then, any messages you send to that will cause automatic connection to the remote repository and send the message. == Large-collection enhancements == A lot of work has gone into large-collections. Thanks to Brent Pinkney, we now has MagmaSet and MagmaDictionary. These are two classes that act like their Smalltalk brothers, except can hold millions of objects and endure high-concurrency. We also have MagmaArray, which provides fast-access to a very-large Array structure. All large-collections now work before they are persistent, as well as seamlessly integrating uncommitted objects added during a transaction. Despite these major improvements, add-speed of MagmaCollections has been improved about 400% in the last year. Read-speed was always fast and remains fast. == Disaster/recovery support == It is now (believed to be) impossible* for the repository to become corrupted should a power-outage occur, however the last few seconds (dictated by #cacheFlushFrequency) of commits could be lost unless you set flushAggressively: true, which will affect performance. == leaner and faster == Between WriteBarrier support, a smaller buffer format, large-collection improvements, Magma has maintained a peak commit rate of 24 per second on my machine. MagmaBenchmarker was developed as a basis for assessing performance limitations. Magma certainly does not have light code-base, but it is better-factored than its ever been. At least two users who have delved into the code have been able to understand it well. == * (no warranty see http://minnow.cc.gatech.edu/squeak/5725, evaluate the design and decide for yourself at http://minnow.cc.gatech.edu/squeak/5724)

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Magma