[Newbies] Re: Binary file I/O performance problems

[Zulq Alam]

Hi David,

You could try using stream next: 4 to read the 4 bytes in one go:

[StandardFileStream readOnlyFileNamed: 'Base.image' do:
	[:stream |
	[stream atEnd] whileFalse:
		[stream next.
		stream next.
		stream next.
		stream next.]]] timeToRun
" 328505 "

[StandardFileStream readOnlyFileNamed: 'Base.image' do:
	[:stream |
	stream binary.
	[stream atEnd] whileFalse:
		[stream next: 4]]] timeToRun
" 144469 "

If you can, read larger chunks:

[StandardFileStream readOnlyFileNamed: 'Base.image' do:
	[:stream |
	stream binary.
	[stream atEnd] whileFalse:
		[stream next: 2048]]] timeToRun
" 343 "


[StandardFileStream readOnlyFileNamed: 'Base.image' do:
	[:stream |
	stream binary.
	[stream atEnd] whileFalse:
		[stream next: 2048]]] timeToRun
" 197 "

I'm surprised there isn't a generic class for this, like 
java.io.BufferedInputStream. Perhaps I haven't discovered it yet. Anyone?

Regards,
Zulq.


David Finlayson wrote:

> There are at 4 calls to stream next for each integer and sure enough,
> a profile of the code (attached below) shows that most of the time is
> being lost in the StandardFileStream basicNext and next methods. There
> must be a better way to do this. Scaled up to operational code, I will
> need to process about 40 Gb of data per day. My C code currently takes
> about 16 cpu hours to do this work (including number crunching). In
> Squeak, just reading the data would take 3 cpu months!