>
> Norberto, i suspect that you doing something wrong. Scanning 76000
> records by iterating though entire collection
> when looking for duplicates is indeed slow. And sure thing, time to
> add new item will grow exponentially once collection grows.
> And its not really matters what kind of DB you will use. It will be
> slow everywhere, if you need to scan entire dataset before adding new
> item to it.
>
Hey!!!! I'm not scanning the entire data base. What about indexes?
I tried to add all  objects before  and perform the comparision after, and
it takes even more time.
I made this before using traditional technology, SQL and all that, and I
never have to wait several days for the result.

>
> So, i suspect that there is something wrong with the way how you using
> MagmaCollection,
> and because of that, performance degrades from O(log(n)) down to O(n),
> for each operation.
> Sure, its hard to say anything without looking at actual code.
>

Ok. Let's see the code

"this method creates the repository"
createMagmaRepository
| thePath theSession |
thePath := FileDirectory default pathName assurePath,'\magma'.
((FileDirectory on: thePath) fileExists:'objects')
        ifFalse:[     (Smalltalk at: #MagmaSession)
disconnectAndCloseAllConnectedSessions.
                (Smalltalk at: #MagmaRepositoryController)   delete:
thePath.
                (Smalltalk at: #MagmaRepositoryController)
                    create: thePath
                    root: (IdentityDictionary new
                            at: #works put: self workCatalog;
                            at: #expressions put: self expressionCatalog ;
                            at: #manifestations  put: self
manifestationCatalog ;
                            yourself).
                theSession := (Smalltalk at: #MagmaSession)  openLocal:
thePath.
                theSession  connectAs: 'nor'.].
            ^theSession

"these method creates the collections (workCatalog, expressionCatalog,
manifestationCatalog"

initializeCollection
collection:=  MagmaCollection new.
collection addIndex:
      ((MaSearchStringIndex
         attribute: #authorizedName)
         keySize: 64;
    beAscii)

"these two methods add and find the objects"
add: anObject
| found |
found:= self find: anObject.
found isNil
    ifTrue:  [self collection add: anObject ]
    ifFalse: [ self merger merge: anObject  on: found ].
^anObject

find: anAccessPoint
| found |
found:= (self collection  where:
        [:reader
        reader read: #authorizedName
                at: anAccessPoint authorizedName]).

"authorizedNamed is a string"

As I allready said, merging is not the problem, so I omit the methods for
merging.

Transaction is made using an interator which let you change the transaction
step. So, that's not a problem. I'm making commits every 300 addings.

"This is the main method"

BibHuma instance useMagmaCollections.
ferberizator iterator: (MagmaIterator newForIsis list: (1 to: 2000)).
"records to migrate to Magma"
ferberizator iterator transactionStep: 300.
session := MagmaSession openLocal: FileDirectory default pathName
assurePath,'magma'.
session connectAs: 'nm'.
ferberizator iterator session: session.
session readStrategy: (MaReadStrategy minimumDepth: 0).
ferberizator build




> > Norberto
> >
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>
>


-- 
Norberto Manzanos
Instituto de Investigaciones en Humanidades y Ciencias Sociales (IdIHCS)
FaHCE/UNLP - CONICET
Calle 48 e/ 6 y 7 s/Nº - 8º piso - oficina 803
Tel: +54-221-4230125 interno 262
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20101102/62d2b1c9/attachment.htm