Magma

magma@lists.squeakfoundation.org

722 discussions

[ANN] New indexes for DateAndTime and UUID instances
by Brent Pinkney 22 Dec '05

22 Dec '05

Hi, There is some chatter on this list regarding new projects using Magma. To prevent duplication and despair I have attached the extensions to Magma that I use to index UUID and DateAndTime instances. Indexing UUIDs ----------------- MaUUIDIndex will index any UUID and has a keySize of 128 bits. This it because a UUID is 16 bytes long. Indexing DateAndTimes -------------------------- A DateAndTime instance includes the offset from UTC and has nanoSecond precision. It can also store any date since 24 November -4713. Indexing a full DateAndTime would require a LOT of bits, which is probably why Chris never added an index for them. Fortunately, we do not need a) to store dates all the way back to the dawn of time b) nano second precision c) to store dates infinitely far into the future. MaDateAndTimeIndex allows the developer to specify: a) the start epoch (dates on or before this are not allowed) b) the clock precision (one of 1 second. 1 milliSecond, 1 nanoSecond) c) the duration into the furture (ie. epoc + duration = last date which is allowed) The default has a keySize of 72 bits and gives nanoSecond precision from 1 January 1900 for 1000 years! I usually use: (MaDateAndTimeIndex attribute: #dateOfBirth) epoch: 2000 asYear; duration: (100 * 365.24) days; precision: 1 second; yourself which gives a keySize of 32 bits. Feedback welcomed. Chris, if you like this work, feel free to modify it and add it to Magma. Regards Brent

2 1

Re: Q: Complex Magma Indexed queries - draft syntax
by Brent Pinkney 22 Dec '05

22 Dec '05

Hi Chris, [Could you remember to send your replies to the magma mailing list] I must confess that I have no solution yet - just early thoughts. Problem --------- Let us construct a hypothetical collection with indexes on attributes #a, #b, #c. Now let use query this collection using these clauses: a = 'A' b between 'Ba' and 'Bz' c = 'C' We therefore have 4 possibilities using just the AND and OR operators: i) a & b & c ii) a | b | c iii) (a ^ b) | c iv) a | (b ^ c) Syntax -------- Let us add convenience methods to Symbol, e.g. #at: which answers a MaClause. Then using Smalltalks cascade operator, the MaCollectionReader can build up a tree of clauses with the appropriate prescedence: i) a & b & c collection reader where: (#a at: 'A'); and: (#b from: 'Ba' to 'Bz'); and: (#c at: 'C'); ii) a | b | c collection reader where: (#a at: 'A'); or: (#b from: 'Ba' to 'Bz'); or: (#c at: 'C'); iii) (a ^ b) | c collection reader where: ( (#a at: 'A') or: (#b from: 'Ba' to 'Bz') ); or: (#c at: 'C') iv) a | (b ^ c) collection reader where: (#a at: 'A') or: ( (#b from: 'Ba' to 'Bz') and: (#c at: 'C') ) Client Side MaCollection Reader Work ------------------------------------------- As with the existing infrastructure, the reader askes each clause to compute the minimum and maximum hash values. A modified tree of clauses is then serialised and submitted to the magma server. Server Side Request Work ------------------------------ Here is where I am still very vague, but the idea is that just as with the existing queries, each clause will be able to find the oids between the minimum and maximum hashes. I have no idea on performance, but could each clause not modify a bitmap index appropraitely ? The server will then have a 2^48 (?) bit bitmap with the oids satisfying the query. This bitmap will be very sparse and may compress nicely. The server can return this bitmap and the first page of objects. Paging -------- Subsequent requests from the client for pages of objects, will submit the bitmap with the request. Comments welcome Brent

3 2

Q: Complex Magma Indexed queries
by Brent Pinkney 21 Dec '05

21 Dec '05

Hi, Concider a MagmaCollection with Person instances. The collection is indexed on #familyName and #dateOfBirth. Magma currently supported very efficient indexed queries on these such indices, but only one index may be queried at a time. Would it be possible to support complex queries on multiple indices ? e.g. a) Find all people named 'Smith' and who were born in 1980 ? b) Find all people named 'Smith' or named 'Jones' To my mind, this should be possible and quite efficient as the current MaCollectionReader code calculates the minimum and maximum hash values for each index. The Magma server then walks the index from the minimum value to the maximum value extracting the oids of each object. I am not sure of the impact the readSize would have on this idea - at first glance it may be necessary to extract all the oids from each clause and cache the appropriate intersaction or union of the oids. The query could then return the pages from this cache. Comments welcomed Brent

2 3

Free text search
by goran＠krampe.se 20 Dec '05

20 Dec '05

Hi fellas! I may very well end up using Magma in a very interesting large scale project soon, and yes, I am also interested in querying capabilities that is being discussed in the other thread. This is a related question: What free text search capabilities does Magma have today? I admit I haven't looked, but wanted to ask anyway since I might get some info on the "plan". And also - is this used/usable/interesting? http://map.squeak.org/packagebyname/fulltext regards, Göran

2 3

Re: Index updating
by Chris Muller 10 Dec '05

10 Dec '05

> Ive got a MagmaCollection with objects that contain attribute #date > with assigned index. How to update index in case I change a date in an > object? Do I have to remove this object from the collection and put it > back again or is there any other way? Hi Pavel, try MagmaSession>>#noteOldKeysFor:. Just supply the object that has the date you are about to change. You have to do it *before* you change the date in the object. Magma will update the date index as part of that commit. Cheers, Chris

1 0

Index updating
by Pavel Krivanek 10 Dec '05

10 Dec '05

Hi all I’ve got a MagmaCollection with objects that contain attribute #date with assigned index. How to update index in case I change a date in an object? Do I have to remove this object from the collection and put it back again or is there any other way? -- Pavel

1 0

Re: [ANN] Magma Server image with web management console
by Lukas Renggli 09 Dec '05

09 Dec '05

> Well, then I fear you'll have to do some debugging :P. The code is > simple: create the database, start a listener. The listener bit is > missing in your image - could it be that between my snapshot and your > startup, some network code didn't pick up changes? It does work from a workspace. As far as I understand KmsManager it doesn't properly encode/parse the database-names, sometimes fileName is empty, sometimes it is the full-path, sometimes only the file-name, ... Lukas -- Lukas Renggli http://www.lukas-renggli.ch

2 2

Re: [ANN] Magma Server image with web management console
by Cees De Groot 08 Dec '05

08 Dec '05

[please remove squeak-dev on reply, taking this to the magma list] On 12/8/05, Lukas Renggli <renggli(a)gmail.com> wrote: > When creating a new database and clicking on 'stats' or 'shutdown' I > get 'Not a running Magma instance'. When I try to it using another > image I get the exception 'MaNetworkError: Cannot connect to > 192.168.1.10:21001'. Both seem to indicate that the create database didn't work. No walkbacks if you run the image headful and try to create a db? Has the directory been created?

2 1

Indexing attributes that return collections?
by Cees De Groot 08 Dec '05

08 Dec '05

Hi, I'm busy cooking up something that'll index my mail archive. The first step is just to throw all mails in a MagmaCollection, and index it on date, message-id, sender/from/to/..., cross-reference it on references, etcetera. My easiest idea would be to have my mailmessage class (a simple subclass of Celeste's mailmessage class) have, say: references

3 6

Re: Indexing attributes that return collections?
by Cees De Groot 08 Dec '05

08 Dec '05

On 12/8/05, Chris Muller <afunkyobject(a)yahoo.com> wrote: > After a couple of days running the load, I realized that common prepositions, > 'THE' 'TO' 'A' 'I' 'AND' 'OF' 'IS' 'IN' 'IT' 'THAT' 'FOR' 'ON' 'THIS' 'BE' > 'YOU' 'WITH' 'BUT' 'HAVE' 'NOT' 'IF' 'AT', etc., etc., was really wasting time > and space. When you have 150,000 occurrences of the same word, it costs a lot > to keep inserting more and, yet, the value of that entry continues to dilute.. > Well, I'm going to tackle this two-step - first a browsable mail archive (by date, senders, references), then full-text. About the full-text I'm still not too sure what to do - quality FTX is hard, and there are lots of good libraries out there (Lucene, mnogosearch, manazu, ...). So I'll probably reuse what's there rather than implement something that doesn't work. > Separate #subject indexing might be a good idea.. > > I'd probably skip all > those lines that begin with ">" (in replies to other posts) simply because > they're so darn many of them. > What you want, of course, is weighted indexing. Words in #subject should have a higher weight that words in the message body, which should have a higher weight than words in attachments, which should have a higher weight than words in quoted messages (">" lines). Probably the thing to do is build a index that holds two integers - weight and object - instead of a single one, or build an index per weight (probably easier to do in Magma). Then, of course, after retrieval, you need to score every hit, based on weight, number of occurrences (too many and the score could presumably go down again), proximity if you're searching on multiple keywords, etcetera. Finally, you present the whole hitlist in scored order. As I said, there's lotsacode out there and there's a reason most of it is ugly and complicated :-)

2 2

← Newer
1
...
66
67
68
69
70
71
72
73
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Magma