[Newbies] Re: Binary file I/O performance problems

[David Finlayson]

Unfortunately, the data is not a simple block of floats. For example,
in C here is how I read a "ping" header block from one of our vendors
formats:

/* read_xyza_ping: read ping block, returns 1 if successful, EOF if
 * end of file  */
int read_xyza_ping(FILE *fin, XYZA_Ping *pp) {
    int8_t byte[4];

    fread(&pp->linename, sizeof(int8_t), MAX_LINENAME_LEN, fin);
    fread(&pp->pingnum, sizeof(uint32_t), 1, fin);
    fread(&byte, sizeof(int8_t), 4, fin);
    fread(&pp->time, sizeof(double), 1, fin);
    fread(&pp->notxers, sizeof(int32_t), 1, fin);
    fread(&byte, sizeof(int8_t), 4, fin);
    read_posn(fin, &pp->posn);
    fread(&pp->roll, sizeof(double), 1, fin);
    fread(&pp->pitch, sizeof(double), 1, fin);
    fread(&pp->heading, sizeof(double), 1, fin);
    fread(&pp->height, sizeof(double), 1, fin);
    fread(&pp->tide, sizeof(double), 1, fin);
    fread(&pp->sos, sizeof(double), 1, fin);

    if (ferror(fin) != 0) {
        perror("sxpfile: error: (read_xyza_ping)");
        abort();
    }

    // time between 1995 - 2020?
    assert(788936400 < pp->time && pp->time < 1577865600);
    assert(0 < pp->notxers && pp->notxers <= MAX_TXERS);
    assert(-90.0 < pp->roll && pp->roll < 90.0);
    assert(-90.0 < pp->pitch && pp->pitch < 90.0);
    assert(0.0 <= pp->heading && pp->heading <= 360.0);

    // heave values
    assert(-10.0 < pp->height && pp->height < 10.0);
    assert(-100 < pp->tide && pp->tide < 100.0);

    // speed of sound reasonable? (freshwater too)
    assert(1000 <= pp->sos && pp->sos < 1600);

    return feof(fin) ? EOF : 1;
}

Note how there are various sized integers and floating point numbers
mixed together along with padding space put into the file during the
write (the original engineer must have just used fwrite on the
structs).

The notxers variable above indicates the number of XYZA_Txer structs
to follow, each XYZA_Txer struct indicates the number of XYZA_Point
structs to follow and so on until the entire structure is read into
memory. Then you start over again and read the next ping.

It is painful, but I don't know how to read any other way except to
read them in one structure at a time.





-- 
David Finlayson, Ph.D.
Operational Geologist

U.S. Geological Survey
Pacific Science Center
400 Natural Bridges Drive
Santa Cruz, CA 95060, USA

Tel: 831-427-4757, Fax: 831-427-4748, E-mail: dfinlayson at usgs.gov