Hi David,

On Fri, Jan 14, 2022 at 10:03:46PM -0500, David O'Toole wrote:
> I forgot to include the output method. Adding "flush" seems to have helped,
> but output still stops coming well before it is complete.
> 
> On Fri, Jan 14, 2022 at 9:49 PM David O'Toole <deeteeoh1138 at gmail.com>
> wrote:
> 
> > I have a question relating to PipeableOSProcess. I'm trying to read the
> > standard output from the process and (optionally) print it to the
> > Transcript. Several large chunks consisting of dozens of lines are
> > produced, but then output halts always after a certain number of characters
> > (stopping in the middle of a line) even though I can verify that the Scheme
> > process completed its entire job successfully and should have produced more
> > output. Am I setting up / accessing the PipeableOSProcess incorrectly in my
> > snippet here? I would greatly appreciate any tips you might be able to give
> > on my investigating this further. I am running Squeak 5.3 on Linux.
> >

A likely source of the problem relates to reading available data from
an OS pipe, particularly if a large amount of data is involved. When
you send #upToEnd to the reader, you will get all of the data that is
currently available in the pipe. Meanwhile, pipe itself may be waiting
for someone to read from it before it allows the process that is writing
to the pipe (the Scheme interpreter in this case) to complete the write.

The end result is that when you read upToEnd from a pipe, you may not
receive the entire chunk of data that is being written by Scheme, and
you may need to wait for a few milliseconds and do another read to
get the rest.

For purposes of debugging, try putting a short delay before the #upToEnd
(Delay forMilliseconds: 200). If that makes it work, it will confirm
my guess above.

A couple of related notes for your information:

Take a look at the #upToEndOfFile method, as compared to #upToEnd.
You will not be able to use it for your Scheme interpreter because
it would wait for the Scheme interpreter to close its end of the
pipe, which is not what you want for in interactive connection. But
it may clarify the issues of reading from a pipe.

Also look at class BufferedAsyncFileReadStream, which deals with
the issue by populating its buffer based on AIO events (data available
on the pipe) to minimize the need for polling loops to get data
from a pipe. This is how you are reading the pipe data, and depending
on whether you have the AIOPlugin in your VM, it may be falling back
on its polling loop instead of being event driven. This should still
work fine but it affects timing of the data available from the pipe.

Regarding the #flush, I'm not sure I understand what you are doing
in the #step method, but when you write to the external Scheme
process on pipeToInput you do want to flush it. That ensures that
your data gets pushed out to the OS pipe, and makes it available
for Scheme to read from its end of the pipe.

HTH,
Dave