[squeak-dev] Printing standard output from PipeableOSProcess on the Transcript

David T. Lewis lewis at mail.msen.com
Sat Jan 15 18:21:45 UTC 2022


Aha, that's coming from the buffered reader. See BufferedAsyncFileReadStream>>#readAheadChunkSize

The buffer is populated in chunks of (up to) 1000 by #moveAvailableDataFrom:
which is called both by the AIO event handler (or polling loop if the plugin
is not running), and also by by #upToEndOfFile method that you are calling.
I would not rule out the possibility of a concurrency problem here, since
I did not synchronize the method around a semaphore. You would think that I
(or somebody) would have noticed by now if this was a problem, but you never
know.

Just for debugging, try changing the chunk size from 1000 to say 1000000
and see if it does anything different.

Dave


On Sat, Jan 15, 2022 at 12:48:08PM -0500, David O'Toole wrote:
> Thank you. This did not change the behavior. I did discover that output is
> always read in exact 1000 byte multiples, first it reads 1000 then in a
> later invocation of #step, it reads 4000 and then output stalls.
> 
> On Sat, Jan 15, 2022 at 12:34 PM David T. Lewis <lewis at mail.msen.com> wrote:
> 
> > Hi David,
> >
> > Hmmm, I'm not sure where the missing output is going.
> >
> > I do have another suggestion though. I think you can simplify your
> > initialization by using just the #prepareOutputForInternalReader method.
> > This will set things up properly, with non-blocking applied only to
> > the Scheme output reader pipe (the writer to Scheme should remain in
> > blocking mode).
> >
> > So the code can be reduced to this:
> >
> >
> > startScheme
> >
> >         sourcePath ifNil: [self error: 'You must set sourcePath to the
> > directory where Mosaic''s Scheme source files are stored.'].
> >         self schemeSourceFileExists ifFalse: [self error: 'Cannot find
> > Scheme source in sourcePath.'].
> >         schemeProcess ifNil: [
> >                 schemeProcess := PipeableOSProcess
> >                         new: schemeExecutable
> >                         arguments: nil
> >                         environment: nil
> >                         descriptors: nil
> >                         workingDir: nil
> >                         errorPipelineStream: nil].
> >         schemeProcess prepareOutputForInternalReader.
> >         "actually start the process now"
> >         schemeProcess value.
> >         self sendInitialize.
> >         self sendSchemeExtensions.
> >         self sendScheme: '(provide ''squeak-mosaic)'
> >
> >
> >
> > On Sat, Jan 15, 2022 at 11:40:33AM -0500, David O'Toole wrote:
> > > Hi David, thanks for your help. I indeed have the following in my About
> > > Squeak dialog: AioPlugin VMConstruction-Plugins-AioPlugin-eem.22 (i)
> > >
> > > The Delay suggestion did not work. I notice that I can start a new Scheme
> > > job, and the standard output begins flowing again, without ever producing
> > > the remaining output from the previous run, even though the run completed
> > > successfully. So the output is getting lost somewhere.
> > >
> > > Should I be doing [schemeOutput atEnd] whileFalse: [ | output | output :=
> > > schemeOutput upToEnd ] ? That seemed to prevent output entirely.
> > >
> > > Thanks for the references to those other places I could look. I will
> > > endeavor to debug this.
> > >
> > > On Sat, Jan 15, 2022 at 10:53 AM David T. Lewis <lewis at mail.msen.com>
> > wrote:
> > >
> > > > Hi David,
> > > >
> > > > On Fri, Jan 14, 2022 at 10:03:46PM -0500, David O'Toole wrote:
> > > > > I forgot to include the output method. Adding "flush" seems to have
> > > > helped,
> > > > > but output still stops coming well before it is complete.
> > > > >
> > > > > On Fri, Jan 14, 2022 at 9:49 PM David O'Toole <
> > deeteeoh1138 at gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I have a question relating to PipeableOSProcess. I'm trying to
> > read the
> > > > > > standard output from the process and (optionally) print it to the
> > > > > > Transcript. Several large chunks consisting of dozens of lines are
> > > > > > produced, but then output halts always after a certain number of
> > > > characters
> > > > > > (stopping in the middle of a line) even though I can verify that
> > the
> > > > Scheme
> > > > > > process completed its entire job successfully and should have
> > produced
> > > > more
> > > > > > output. Am I setting up / accessing the PipeableOSProcess
> > incorrectly
> > > > in my
> > > > > > snippet here? I would greatly appreciate any tips you might be
> > able to
> > > > give
> > > > > > on my investigating this further. I am running Squeak 5.3 on Linux.
> > > > > >
> > > >
> > > > A likely source of the problem relates to reading available data from
> > > > an OS pipe, particularly if a large amount of data is involved. When
> > > > you send #upToEnd to the reader, you will get all of the data that is
> > > > currently available in the pipe. Meanwhile, pipe itself may be waiting
> > > > for someone to read from it before it allows the process that is
> > writing
> > > > to the pipe (the Scheme interpreter in this case) to complete the
> > write.
> > > >
> > > > The end result is that when you read upToEnd from a pipe, you may not
> > > > receive the entire chunk of data that is being written by Scheme, and
> > > > you may need to wait for a few milliseconds and do another read to
> > > > get the rest.
> > > >
> > > > For purposes of debugging, try putting a short delay before the
> > #upToEnd
> > > > (Delay forMilliseconds: 200). If that makes it work, it will confirm
> > > > my guess above.
> > > >
> > > > A couple of related notes for your information:
> > > >
> > > > Take a look at the #upToEndOfFile method, as compared to #upToEnd.
> > > > You will not be able to use it for your Scheme interpreter because
> > > > it would wait for the Scheme interpreter to close its end of the
> > > > pipe, which is not what you want for in interactive connection. But
> > > > it may clarify the issues of reading from a pipe.
> > > >
> > > > Also look at class BufferedAsyncFileReadStream, which deals with
> > > > the issue by populating its buffer based on AIO events (data available
> > > > on the pipe) to minimize the need for polling loops to get data
> > > > from a pipe. This is how you are reading the pipe data, and depending
> > > > on whether you have the AIOPlugin in your VM, it may be falling back
> > > > on its polling loop instead of being event driven. This should still
> > > > work fine but it affects timing of the data available from the pipe.
> > > >
> > > > Regarding the #flush, I'm not sure I understand what you are doing
> > > > in the #step method, but when you write to the external Scheme
> > > > process on pipeToInput you do want to flush it. That ensures that
> > > > your data gets pushed out to the OS pipe, and makes it available
> > > > for Scheme to read from its end of the pipe.
> > > >
> > > > HTH,
> > > > Dave
> > > >
> > > >
> > > >
> >
> > >
> >
> >
> >

> 



More information about the Squeak-dev mailing list