Slow performance

[Todd Blanchard]

Thanks, I'm busy updating it to use MagmaSet as we speak (where set  
key is the timestamp of the data).
The most common strategy with historical data is to ask for a range.
IOW, gimme all the quotes from 1/1/1995 to today.

The other item that is maybe more problematic is storing the chart  
data - big chunk of points - but maybe its not so big as it is an  
OrderedCollection of OrderedCollections on the order of about 100 per  
side.

I assume sending commitAndBegin is a checkpoint save?

Also, after creating a MagmaSet that I stick in a BTree (which is the  
root) I'm getting this trying to read it back:

'Unable to realize Orphaned MagmaMutatingProxy'

If there's an error, an exception would be nicer than just returning  
an error string for the collection I'm trying to read.

On Nov 7, 2007, at 7:28 PM, Chris Muller wrote:

> There is another aspect to this type of application that I would like
> to leverage to improve Magma performance in the future.  That is the
> notion that the Quote objects are never ever going to change again.
> Therefore Magma could be told to ignore them for change-detection and
> this could improve performance considerably.
>
> There is already space in the object-buffer for the bit, I just need
> to think of a good way for the end-user program to tell Magma a
> particular object is "read-only"..  Probably something like:
>
>   mySession beReadOnly: anObject
>
> ??
>
> Something for me think about for a future version, if anyone has any
> suggestions please let me know.
>
> On Nov 5, 2007 2:38 AM, Todd Blanchard <tblanchard at mac.com> wrote:
>> I may be doing this wrong so I'll share my approach - I'm finding it
>> unusably slow though.
>>
>> I'm working on a stock market analysis tool.  There are about 1100
>> securities on the AMEX, NASDAQ, and NYSE.  So I create a Dictionary
>> in the root dict called securities.  This is keyed by stock ticker
>> symbol and the value is an object of type security which contains
>> fields tickerSymbol, exchange, issueName, and historicalData.
>>
>> The historicalData is an OrderedCollection of quotes going back as
>> far as the 80's.  A quote is a timestamp, hi, low, close, and trade
>> volume.  There are around 300 of these per year - going back as much
>> as 25 years.  Believe it or not, I can fetch these from yahoo as a
>> csv and process them into objects in about 5-10 seconds per
>> security.  Saving this data structure into magma seems to take many
>> times that.  Something like 3 hours in Magma I think.  If this were
>> just the initial load, it would be tolerable.  However fetching the
>> last 5 days quotes and splicing them onto the tail of the
>> historicalData collection takes as long as the initial load.  So this
>> approach isn't working for me.
>>
>> I'm open to ideas on better ways to structure this.  I'll also be
>> adding some charts to keep in the database - the idea being the
>> charts are mostly up to date and I only have to replot the last day's
>> worth of data everytime it is fetched.  When I say chart - I mean a
>> data structure containing a 2D array of values - not a visual
>> representation.  I will always draw the visual form on the fly.
>>
>> This chart will also potentially be a big block of data that will
>> reference the historicalData points.
>>
>> Ideas?  I'm close to just going to image segments - one per security.
>>
>> -Todd
>> _______________________________________________
>> Magma mailing list
>> Magma at lists.squeakfoundation.org
>> http://lists.squeakfoundation.org/mailman/listinfo/magma
>>