Slow performance

[Sebastian Sastre]

Hi Todd,

	beside what you said, have you made some profiling?

	The test runner has this option of running a test profiled. Maybe
you can make a test of one of those insertions and see exactly what is
eating so much time.

	If you confirm that is that ordered collection (I would suspect so)
then use something that when changes do not have to be entirely serialized
each time, like a MagmaCollection or a Btree.

	I also think that having a Dictionary for those 1100 securities is
not helping you. I bet that if you use a Btree instead of it things will get
better. Also you can try a MagmaCollection.

	Don't think in MagmaCollections just for large collections but also
for efficient writes when they are changed. The oposite is most probably
true with common collection, dicts, etc.

	I hope you can improve that,

	cheers,

Sebastian Sastre
 

> -----Mensaje original-----
> De: magma-bounces at lists.squeakfoundation.org 
> [mailto:magma-bounces at lists.squeakfoundation.org] En nombre 
> de Todd Blanchard
> Enviado el: Lunes, 05 de Noviembre de 2007 05:39
> Para: magma at lists.squeakfoundation.org
> Asunto: Slow performance
> 
> I may be doing this wrong so I'll share my approach - I'm 
> finding it unusably slow though.
> 
> I'm working on a stock market analysis tool.  There are about 
> 1100 securities on the AMEX, NASDAQ, and NYSE.  So I create a 
> Dictionary in the root dict called securities.  This is keyed 
> by stock ticker symbol and the value is an object of type 
> security which contains fields tickerSymbol, exchange, 
> issueName, and historicalData.
> 
> The historicalData is an OrderedCollection of quotes going 
> back as far as the 80's.  A quote is a timestamp, hi, low, 
> close, and trade volume.  There are around 300 of these per 
> year - going back as much as 25 years.  Believe it or not, I 
> can fetch these from yahoo as a csv and process them into 
> objects in about 5-10 seconds per security.  Saving this data 
> structure into magma seems to take many times that.  
> Something like 3 hours in Magma I think.  If this were just 
> the initial load, it would be tolerable.  However fetching 
> the last 5 days quotes and splicing them onto the tail of the 
> historicalData collection takes as long as the initial load.  
> So this approach isn't working for me.
> 
> I'm open to ideas on better ways to structure this.  I'll 
> also be adding some charts to keep in the database - the idea 
> being the charts are mostly up to date and I only have to 
> replot the last day's worth of data everytime it is fetched.  
> When I say chart - I mean a data structure containing a 2D 
> array of values - not a visual representation.  I will always 
> draw the visual form on the fly.
> 
> This chart will also potentially be a big block of data that 
> will reference the historicalData points.
> 
> Ideas?  I'm close to just going to image segments - one per security.
> 
> -Todd
> _______________________________________________
> Magma mailing list
> Magma at lists.squeakfoundation.org
> http://lists.squeakfoundation.org/mailman/listinfo/magma