I'd like to point to the "Managing Gigabytes" book.
Warning:  their software is price=zero but not Open Source,
and didn't work on 64-bit machines (and wasn't expected to)
last time I tried it.
The book is a great source of ideas, though,
and the techniques described _do_ let you search reasonably
fast on stock hardware through _huge_ amounts of text.