On Fri, Dec 11, 2009 at 07:17:50PM -0800, Andreas Raab wrote:
> Gerardo Richarte wrote:
> >without going to see anything in squeak (so, maybe totally wrong),
> >usually when you call pipe() in unix, you then fork() and pass one of
> >the two descriptors to the child process. The common practice is to
> >close() the file descriptors passed to the child process. This has to be
> >done in the parent process after forking. Otherwise the parent keeps
> >open the end of the pipe() corresponding to the child.

Yes, that's right. OSProcess does exactly this, it closes the descriptors
that will be used by the child process and keeps open the descriptors
that will be used for writing to the stdin of the child process, and
reading from the stdout and stderr of the child process.

> Thanks. We're using PipeableOSProcess to be able to get to the output of 
> various commands.
> 
> >Of course, after finishing you also hahve to close() the other end of
> >the pipe() in the parent. The child process usually dies, hence closing
> >all file descriptors. But, if the child doesn't die, then you have to be
> >careful to also close the file descriptors belonging to the parent, and
> >the proper way to do it is to close them just after forking, in the
> >child process... something like:
> 
> Hm ... not sure what OSProcess does in that regard. Anyone know?
> 
> Thanks for the info!

Hi Andreas,

If you are using PipeableOSProcess class>>command:, then the most
likely source of the problem is that you need to explicitly close the
pipeFromOutput after reading the output data. PipeableOSProcess closes
as many of the pipe handles as possible, but the handle for the
pipe output (your reader end for the stdout from the external process)
is left open to permit you to read the output data after the external
process has exited. 

To confirm this, use and object explorer on the PipeableOSProcess and
drill down to the pipe endpoints and check to make sure they are all
#closed.

You might consider using ProxyPipeline class>>command: rather than
PipeableOSProcess class>>command. In that case you can use #upToEndOfFile
rather than #upToEnd to obtain the output. This will read all available
output, waiting as needed to allow the external process to exit. A side
effect of the end of file detection is that the pipe reader will be
closed, hence no left over file handles.

ProxyPipeline is more complex than PipeableOSProcess, but the automatic
cleanup may make it more reliable for you overall.

A couple other notes that might be relevant:

- Pipe handles are not cleaned up by finalization. The reason for
this is that I explicitly disabled the handle registration in
AttachableFileStream class>>register: in order to prevent performance
issues associated with the weak registry.  Basically I found that running
the CommandShell unit tests was enough to cause performance problems
for the entire image, so I turned off the registration for file streams
associated with OS pipes. If you are not using OSProcess too heavily,
you might consider turning the registration back on as an extra measure
of safety. But do keep a close eye on the size of the weak registry,
because PipeableOSProcess uses a lot of file streams to connect to
all those pipes.

- Aside from failing to explicitly close pipes, the other possible (but
very unlikely) source of handle leaks would be a failure in the child 
process cleanup mechanism, which results in zombie child processes and
lots of open handles. Fortunately, as long as you do not use a Pharo
image you are not likely to encounter this problem ;-)

HTH,
Dave