On 3 November 2010 00:02, Norberto Manzanos <nmanzanos at gmail.com> wrote:
>
>>
>> Norberto, i suspect that you doing something wrong. Scanning 76000
>> records by iterating though entire collection
>> when looking for duplicates is indeed slow. And sure thing, time to
>> add new item will grow exponentially once collection grows.
>> And its not really matters what kind of DB you will use. It will be
>> slow everywhere, if you need to scan entire dataset before adding new
>> item to it.
>
> Hey!!!! I'm not scanning the entire data base. What about indexes?
> I tried to add all  objects before  and perform the comparision after, and
> it takes even more time.
> I made this before using traditional technology, SQL and all that, and I
> never have to wait several days for the result.

Okay.. here what i have tried on bleeding edge magma-tester suite and Squeak VM:

MagmaRepositoryController
    create: FileDirectory default pathName , '\foo'
    root: OrderedCollection new


| session |
session := MagmaSession openLocal: FileDirectory default pathName , '\foo'.
session connectAs: 'nm'.
session commit: [
	session root add: 1->1
	].
session closeRepository.


| session temp time |
session := MagmaSession openLocal: FileDirectory default pathName , '\foo'.
session connectAs: 'nm'.
session commit: [
temp := session root at: 1 ].
time := [ 100 timesRepeat: [
	session commit: [
		1000 timesRepeat: [ temp value: ( 1->1). temp := temp value. ]
	]]] timeToRun.
session closeRepository.
time

53704

~54 seconds for populating 100000 unique objects.

Now, with magma collection:

| session |
session := MagmaSession openLocal: FileDirectory default pathName , '\foo'.
session connectAs: 'nm'.
session commit: [
	session root at: 1 put: MagmaCollection new
	].
session closeRepository.

| session temp time |
session := MagmaSession openLocal: FileDirectory default pathName , '\foo'.
session connectAs: 'nm'.
session commit: [ temp := session root at: 1 ].
time := [ 100 timesRepeat: [
	session commit: [
		1000 timesRepeat: [ temp add: ( 1->1) ]
	]]] timeToRun.
session closeRepository.
time

116307

Again, 116 seconds.

Okay, what about magma collection with index?

| session coll |
session := MagmaSession openLocal: FileDirectory default pathName , '\foo'.
session connectAs: 'nm'.
session commit: [
	coll := MagmaCollection new.
	coll addIndex: ((MaIntegerIndex attribute: #key) keySize: 64).
	session root at: 1 put: coll
	].
session closeRepository.

| session temp time i |
session := MagmaSession openLocal: FileDirectory default pathName , '\foo'.
session connectAs: 'nm'.
session commit: [ temp := session root at: 1 ].
i := 0.
time := [ 100 timesRepeat: [
	session commit: [
		1000 timesRepeat: [ temp add: ( i -> 1). i := i +1. ]
	]]] timeToRun.
session closeRepository.
time
 263306

263 seconds. Which is what i expected. Maintaining extra index doubles
population time.

Still, i'd prefer to see it 10-50 times faster.

But as you can see, it doesn't takes hours. Later i will try to
complicate the test with adding a 'uniqueness' check,
using magma reader, in same way as you do.

Meanwhile, can you run this code and tell what is your numbers?


-- 
Best regards,
Igor Stasenko AKA sig.