Slow performance

[Todd Blanchard]

I may be doing this wrong so I'll share my approach - I'm finding it  
unusably slow though.

I'm working on a stock market analysis tool.  There are about 1100  
securities on the AMEX, NASDAQ, and NYSE.  So I create a Dictionary  
in the root dict called securities.  This is keyed by stock ticker  
symbol and the value is an object of type security which contains  
fields tickerSymbol, exchange, issueName, and historicalData.

The historicalData is an OrderedCollection of quotes going back as  
far as the 80's.  A quote is a timestamp, hi, low, close, and trade  
volume.  There are around 300 of these per year - going back as much  
as 25 years.  Believe it or not, I can fetch these from yahoo as a  
csv and process them into objects in about 5-10 seconds per  
security.  Saving this data structure into magma seems to take many  
times that.  Something like 3 hours in Magma I think.  If this were  
just the initial load, it would be tolerable.  However fetching the  
last 5 days quotes and splicing them onto the tail of the  
historicalData collection takes as long as the initial load.  So this  
approach isn't working for me.

I'm open to ideas on better ways to structure this.  I'll also be  
adding some charts to keep in the database - the idea being the  
charts are mostly up to date and I only have to replot the last day's  
worth of data everytime it is fetched.  When I say chart - I mean a  
data structure containing a 2D array of values - not a visual  
representation.  I will always draw the visual form on the fly.

This chart will also potentially be a big block of data that will  
reference the historicalData points.

Ideas?  I'm close to just going to image segments - one per security.

-Todd