[Newbies] Re: Binary file I/O performance problems

[David Finlayson]

I have implemented a number of signal processing programs in both C99
and Python (with psyco jit). I have an 8-core Mac Pro workstation
which I can use as for parallel processing by launching multiple
instances of the code using Make scripts. An interesting thing
happened when I compared the performance of the C code to the Python
code:

The C code became I/O bound at 4 cores saturating either the disks or
the memory bus (I am not sure exactly where the bottleneck is). While
the Python version never became I/O bound at 8 cores, it did however
close to within a factor of 10 of the performance of the C code. This
suggested to me that If I had enough processors to saturate the I/O
there was no speed advantage of writing the code in C.

The next generation of workstations we buy will probably have dozens
of cores but hard drives and memory will only be marginally faster (if
history is any indication). So, if I/O is the rate limiting factor,
not cpu speed, why not look for the most productive programing
environment possible? I've always read that Smalltalk is often
considered the most productive programing environment ever invented.
So I wanted to give it a try. But I am discovering (from the point of
view of a scientist programmer like myself) it lacks a lot in
comparison to Matlab or Python (both high-level) and especially C and
C++ (lots and lots of library code).

I am going to have to weigh the pros and cons of whether it makes
since to push on with this.

David