On Wed, 23 Jan 2002, Ralph Johnson wrote:

> 	From: Scott A Crosby <crosby at qwes.math.cmu.edu>
> 
> 	If you think I'm in error in this case, I'm happy to be convinced
> 	otherwise.
> 
> For what it is worth, I am 100% on your side.  

Hmm. I'm something like 99% :)

Of course, it's always possible to drop back to the old, very slow methods
for machines with less RAM. But those tend to be the very machines that
need the improvements the *most*.

Older machines tend to have harder limits on how much RAM you can put in,
and how cheap that RAM is. (I'm not talking 1980s hardware, either!)

So, for such folks, upgrading the RAM means upgrading the whole
machine.

Of course, on a lot of those machines, you're working with very small
datasets :) Although, having fast search on my iPaq would be sweet!

Hmm. For the kind of stuff one'd be using these searches for, it's prolly
not going to be a problem. If you're running a serious server, add the
RAM. If you're a personal user, a coarse grain persistent solution should
work just dandy (e.g., close celeste and serialize the index :)).

For what it's worth, I've started on a MailMessageIndex and
IndexAdaptor. Thus far, it's working spiffy. I was able, *easily* to
substitute in and IndexFileEntry for the fulltext in the Index. Which
means that this 1) can be used directly by Celeste with fairly minimal
modifications and 2) it's not going to take a *hell* of a lot more memory
than the current Celeste situation and 3) you get blazing full text
searchs for that extra memory.

Since I rarely want to search my entire 96 meg message file on full text,
this can be judiciously managed, I think. I rather suspect that indexes
for each key header term will be a big win and pare things down
considerable.

Heck, even partitioning the index by date could help a lot. Only load the
older indexes upon request. A good chunk of the time I don't want to find
email from *last year* :)

Finally, what about the SCAN db code? Seems like a reasonable fit for some
of these persistence issues.

Cheers,
Bijan Parsia.