<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div><div class="h5">
<br>
</div></div>Norberto, i suspect that you doing something wrong. Scanning 76000<br>
records by iterating though entire collection<br>
when looking for duplicates is indeed slow. And sure thing, time to<br>
add new item will grow exponentially once collection grows.<br>
And its not really matters what kind of DB you will use. It will be<br>
slow everywhere, if you need to scan entire dataset before adding new<br>
item to it.<br></blockquote><div>Hey!!!! I'm not scanning the entire data base. What about indexes? <br></div><div>I tried to add all objects before and perform the comparision after, and it takes even more time.<br>
I made this before using traditional technology, SQL and all that, and I never have to wait several days for the result.<br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
So, i suspect that there is something wrong with the way how you using<br>
MagmaCollection,<br>
and because of that, performance degrades from O(log(n)) down to O(n),<br>
for each operation.<br>
Sure, its hard to say anything without looking at actual code.<br></blockquote><div><br>Ok. Let's see the code<br><br>"this method creates the repository"<br>createMagmaRepository<br>| thePath theSession |<br>
thePath := FileDirectory default pathName assurePath,'\magma'.<br>((FileDirectory on: thePath) fileExists:'objects')<br> ifFalse:[ (Smalltalk at: #MagmaSession) disconnectAndCloseAllConnectedSessions.<br>
(Smalltalk at: #MagmaRepositoryController) delete: thePath.<br> (Smalltalk at: #MagmaRepositoryController) <br> create: thePath<br> root: (IdentityDictionary new <br>
at: #works put: self workCatalog;<br> at: #expressions put: self expressionCatalog ;<br> at: #manifestations put: self manifestationCatalog ;<br>
yourself).<br> theSession := (Smalltalk at: #MagmaSession) openLocal: thePath.<br> theSession connectAs: 'nor'.].<br> ^theSession<br><br>"these method creates the collections (workCatalog, expressionCatalog, manifestationCatalog"<br>
<br>initializeCollection<br>collection:= MagmaCollection new.<br>collection addIndex:<br> ((MaSearchStringIndex<br> attribute: #authorizedName)<br> keySize: 64;<br> beAscii) <br><br>"these two methods add and find the objects" <br>
add: anObject<br>| found |<br>found:= self find: anObject.<br>found isNil<br> ifTrue: [self collection add: anObject ]<br> ifFalse: [ self merger merge: anObject on: found ].<br>^anObject<br><br>find: anAccessPoint<br>
| found |<br>found:= (self collection where:<br> [:reader <br> reader read: #authorizedName<br> at: anAccessPoint authorizedName]).<br><br>"authorizedNamed is a string"<br><br>As I allready said, merging is not the problem, so I omit the methods for merging.<br>
<br>Transaction is made using an interator which let you change the transaction step. So, that's not a problem. I'm making commits every 300 addings. <br><br>"This is the main method"<br><br>BibHuma instance useMagmaCollections.<br>
ferberizator iterator: (MagmaIterator newForIsis list: (1 to: 2000)). "records to migrate to Magma"<br>ferberizator iterator transactionStep: 300.<br>session := MagmaSession openLocal: FileDirectory default pathName assurePath,'magma'.<br>
session connectAs: 'nm'.<br>ferberizator iterator session: session.<br>session readStrategy: (MaReadStrategy minimumDepth: 0).<br>ferberizator build <br><br><br><br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
> Norberto<br>
><br>
<font color="#888888"><br>
<br>
--<br>
</font><div><div></div><div class="h5">Best regards,<br>
Igor Stasenko AKA sig.<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><font face="Arial" size="2">Norberto Manzanos<br>Instituto de Investigaciones en
Humanidades y Ciencias Sociales (IdIHCS)<br>FaHCE/UNLP - CONICET<br>Calle 48 e/
6 y 7 s/Nº - 8º piso - oficina 803<br>Tel: +54-221-4230125 interno
262</font><div style="display: inline;"></div><div style="display: inline;"></div><div style="display: inline;"></div><br>