I am competing in a friendly Code Smackdown at work. Smalltalk vs. All on-comers (Haskell, Erlang, LISP, Java, C and Python). The challenge is to write a client for a Chinese Poker game.
Everything is going well except communicating with the Python-based server. I am having trouble reading and writing from STDIN and STDOUT. I've installed OSProcess which should give me access to these streams, but I don't see how to read one line at a time from STDIN and then get a handle on STDOUT to write a line. Anyone have pointers or sample code?
Also, the image needs to be launched by Python with code like...
subprocess.Popen(["/Application/Squeak/Contents/MacOS/Squeak VM Opt", "/Users/jrogers/Squeak/images/chinese_poker.image", "Squeak Client"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
This will obviously need to change for a server running on another OS. The question I have here is how to start a process reading from STDIN as soon as the VM is fired up. Should I just create an object that is waiting on STDIN, reading a line at a time, before saving the image? That way it would just keep trying to read when the image was fired up again.
Thanks.
On Tue, May 12, 2009 at 04:59:27PM -0400, Jason Rogers wrote:
I am competing in a friendly Code Smackdown at work. Smalltalk vs. All on-comers (Haskell, Erlang, LISP, Java, C and Python). The challenge is to write a client for a Chinese Poker game.
Everything is going well except communicating with the Python-based server. I am having trouble reading and writing from STDIN and STDOUT. I've installed OSProcess which should give me access to these streams, but I don't see how to read one line at a time from STDIN and then get a handle on STDOUT to write a line. Anyone have pointers or sample code?
Also, the image needs to be launched by Python with code like...
subprocess.Popen(["/Application/Squeak/Contents/MacOS/Squeak VM Opt", "/Users/jrogers/Squeak/images/chinese_poker.image", "Squeak Client"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
This will obviously need to change for a server running on another OS. The question I have here is how to start a process reading from STDIN as soon as the VM is fired up. Should I just create an object that is waiting on STDIN, reading a line at a time, before saving the image? That way it would just keep trying to read when the image was fired up again.
Hi Jason,
I'm assuming that you are using OS X or other unix based OS (note that OSProcess does not do pipes correctly on Windows, so you won't be able to do that).
Please install CommandShell in addition to OSProcess, as that will give you better ways to control external processes connected by pipes. I'm not clear from your question as to whether Python or Squeak is the "client" in this scenario, but it sounds like you will want to have Squeak making requests to a Python server, but you want Python to set up the pipes and invoke Squeak as a subprocess. I don't have an example of this scenario exactly, but the following may help.
For an example of Squeak as the client, but where Squeak (not Python) does all the work of setting up the pipes connections to the server, see PipeableOSProcess class>>tkExample. This shows Squeak running a tk/tcl interpreter and putting up a trivial UI with buttons connected to Squeak.
For an example of Squeak as the server, communicating through its STDIN and STDOUT streams (which might be pipes set up by Python in your scenario), see class ExternalCommandShell. After evaluating "ExternalCommandShell start" your Squeak image will read command lines from STDIN and respond on STDOUT. The ExternalCommandShell will evaluate input commands and Smalltalk expressions, returning its output to the STDOUT stream. This does not sound like what you want for your project, but it should give you an idea of how these things can be done.
In general, the STDIN steam is "OSProcess thisOSProcess stdIn" and the STDOUT and STDERR streams are "OSProcess thisOSProcess stdOut" and "OSProcess thisOSProcess stdErr". Do not save these in variables, but evaluate them at runtime so your Squeak image can find the right input/output streams even after an image restart. That means you can start a Process in Squeak that reads from stdIn and writes to stdOut. Leave the process running when you save the image, and it will carry on where it left off whenever your Python program restarts the Squeak image.
Note that the stdIn stream for Squeak is (and must be) non-blocking in order to prevent locking up the Squeak VM on a blocked read. That means that your process will need to read available data from stdIn, possibly looping until you get a complete string of whatever you are looking for. For example, if ten characters are currently available in your STDIN input stream, then "OSProcess thisOSProcess stdIn next: 10000" will answer a string of length ten.
ExternalCommandShell does this all in an event-driven manner with no polling, so you can copy that if you want to get fancy. Otherwise a simple polling loop in your process will probably work well enough. Put a 100 ms delay in whatever loop is reading from stdIn an you will get tolerable performance.
Dave
Dave,
Thanks for the help. I will try what you mention in the last two sections. They sound closer to what I need.
On Tue, May 12, 2009 at 8:59 PM, David T. Lewis lewis@mail.msen.com wrote:
On Tue, May 12, 2009 at 04:59:27PM -0400, Jason Rogers wrote:
I am competing in a friendly Code Smackdown at work. Smalltalk vs. All on-comers (Haskell, Erlang, LISP, Java, C and Python). The challenge is to write a client for a Chinese Poker game.
Everything is going well except communicating with the Python-based server. I am having trouble reading and writing from STDIN and STDOUT. I've installed OSProcess which should give me access to these streams, but I don't see how to read one line at a time from STDIN and then get a handle on STDOUT to write a line. Anyone have pointers or sample code?
Also, the image needs to be launched by Python with code like...
subprocess.Popen(["/Application/Squeak/Contents/MacOS/Squeak VM Opt", "/Users/jrogers/Squeak/images/chinese_poker.image", "Squeak Client"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
This will obviously need to change for a server running on another OS. The question I have here is how to start a process reading from STDIN as soon as the VM is fired up. Should I just create an object that is waiting on STDIN, reading a line at a time, before saving the image? That way it would just keep trying to read when the image was fired up again.
Hi Jason,
I'm assuming that you are using OS X or other unix based OS (note that OSProcess does not do pipes correctly on Windows, so you won't be able to do that).
Please install CommandShell in addition to OSProcess, as that will give you better ways to control external processes connected by pipes. I'm not clear from your question as to whether Python or Squeak is the "client" in this scenario, but it sounds like you will want to have Squeak making requests to a Python server, but you want Python to set up the pipes and invoke Squeak as a subprocess. I don't have an example of this scenario exactly, but the following may help.
For an example of Squeak as the client, but where Squeak (not Python) does all the work of setting up the pipes connections to the server, see PipeableOSProcess class>>tkExample. This shows Squeak running a tk/tcl interpreter and putting up a trivial UI with buttons connected to Squeak.
For an example of Squeak as the server, communicating through its STDIN and STDOUT streams (which might be pipes set up by Python in your scenario), see class ExternalCommandShell. After evaluating "ExternalCommandShell start" your Squeak image will read command lines from STDIN and respond on STDOUT. The ExternalCommandShell will evaluate input commands and Smalltalk expressions, returning its output to the STDOUT stream. This does not sound like what you want for your project, but it should give you an idea of how these things can be done.
In general, the STDIN steam is "OSProcess thisOSProcess stdIn" and the STDOUT and STDERR streams are "OSProcess thisOSProcess stdOut" and "OSProcess thisOSProcess stdErr". Do not save these in variables, but evaluate them at runtime so your Squeak image can find the right input/output streams even after an image restart. That means you can start a Process in Squeak that reads from stdIn and writes to stdOut. Leave the process running when you save the image, and it will carry on where it left off whenever your Python program restarts the Squeak image.
Note that the stdIn stream for Squeak is (and must be) non-blocking in order to prevent locking up the Squeak VM on a blocked read. That means that your process will need to read available data from stdIn, possibly looping until you get a complete string of whatever you are looking for. For example, if ten characters are currently available in your STDIN input stream, then "OSProcess thisOSProcess stdIn next: 10000" will answer a string of length ten.
ExternalCommandShell does this all in an event-driven manner with no polling, so you can copy that if you want to get fancy. Otherwise a simple polling loop in your process will probably work well enough. Put a 100 ms delay in whatever loop is reading from stdIn an you will get tolerable performance.
Dave
On Tue, May 12, 2009 at 8:59 PM, David T. Lewis lewis@mail.msen.com wrote:
On Tue, May 12, 2009 at 04:59:27PM -0400, Jason Rogers wrote:
I am competing in a friendly Code Smackdown at work. Smalltalk vs. All on-comers (Haskell, Erlang, LISP, Java, C and Python). The challenge is to write a client for a Chinese Poker game.
Everything is going well except communicating with the Python-based server. I am having trouble reading and writing from STDIN and STDOUT. I've installed OSProcess which should give me access to these streams, but I don't see how to read one line at a time from STDIN and then get a handle on STDOUT to write a line. Anyone have pointers or sample code?
Also, the image needs to be launched by Python with code like...
subprocess.Popen(["/Application/Squeak/Contents/MacOS/Squeak VM Opt", "/Users/jrogers/Squeak/images/chinese_poker.image", "Squeak Client"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
This will obviously need to change for a server running on another OS. The question I have here is how to start a process reading from STDIN as soon as the VM is fired up. Should I just create an object that is waiting on STDIN, reading a line at a time, before saving the image? That way it would just keep trying to read when the image was fired up again.
Hi Jason,
I'm assuming that you are using OS X or other unix based OS (note that OSProcess does not do pipes correctly on Windows, so you won't be able to do that).
Please install CommandShell in addition to OSProcess, as that will give you better ways to control external processes connected by pipes. I'm not clear from your question as to whether Python or Squeak is the "client" in this scenario, but it sounds like you will want to have Squeak making requests to a Python server, but you want Python to set up the pipes and invoke Squeak as a subprocess. I don't have an example of this scenario exactly, but the following may help.
For an example of Squeak as the client, but where Squeak (not Python) does all the work of setting up the pipes connections to the server, see PipeableOSProcess class>>tkExample. This shows Squeak running a tk/tcl interpreter and putting up a trivial UI with buttons connected to Squeak.
For an example of Squeak as the server, communicating through its STDIN and STDOUT streams (which might be pipes set up by Python in your scenario), see class ExternalCommandShell. After evaluating "ExternalCommandShell start" your Squeak image will read command lines from STDIN and respond on STDOUT. The ExternalCommandShell will evaluate input commands and Smalltalk expressions, returning its output to the STDOUT stream. This does not sound like what you want for your project, but it should give you an idea of how these things can be done.
In general, the STDIN steam is "OSProcess thisOSProcess stdIn" and the STDOUT and STDERR streams are "OSProcess thisOSProcess stdOut" and "OSProcess thisOSProcess stdErr". Do not save these in variables, but evaluate them at runtime so your Squeak image can find the right input/output streams even after an image restart. That means you can start a Process in Squeak that reads from stdIn and writes to stdOut. Leave the process running when you save the image, and it will carry on where it left off whenever your Python program restarts the Squeak image.
Note that the stdIn stream for Squeak is (and must be) non-blocking in order to prevent locking up the Squeak VM on a blocked read. That means that your process will need to read available data from stdIn, possibly looping until you get a complete string of whatever you are looking for. For example, if ten characters are currently available in your STDIN input stream, then "OSProcess thisOSProcess stdIn next: 10000" will answer a string of length ten.
ExternalCommandShell does this all in an event-driven manner with no polling, so you can copy that if you want to get fancy. Otherwise a simple polling loop in your process will probably work well enough. Put a 100 ms delay in whatever loop is reading from stdIn an you will get tolerable performance.
Dave
Thanks again Dave for the help. My tack was to "copy" ExternalCommandShell's pattern. StdIn is working fine, but when I write to StdOut nothing seems to actually "get out". Any ideas?
I'm running on Mac, I don't have the AioPlugin installed (I've commented out the inform: that tells about the plugin not being there and therefore polling will be used). I'm not verbose enough at building plugins.
Any ideas why? I can post my Monticello file(s) somewhere if that would help.
On Fri, May 15, 2009 at 1:31 AM, Jason Rogers jacaetevha@gmail.com wrote:
On Tue, May 12, 2009 at 8:59 PM, David T. Lewis lewis@mail.msen.com wrote:
On Tue, May 12, 2009 at 04:59:27PM -0400, Jason Rogers wrote:
I am competing in a friendly Code Smackdown at work. Smalltalk vs. All on-comers (Haskell, Erlang, LISP, Java, C and Python). The challenge is to write a client for a Chinese Poker game.
Everything is going well except communicating with the Python-based server. I am having trouble reading and writing from STDIN and STDOUT. I've installed OSProcess which should give me access to these streams, but I don't see how to read one line at a time from STDIN and then get a handle on STDOUT to write a line. Anyone have pointers or sample code?
Also, the image needs to be launched by Python with code like...
subprocess.Popen(["/Application/Squeak/Contents/MacOS/Squeak VM Opt", "/Users/jrogers/Squeak/images/chinese_poker.image", "Squeak Client"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
This will obviously need to change for a server running on another OS. The question I have here is how to start a process reading from STDIN as soon as the VM is fired up. Should I just create an object that is waiting on STDIN, reading a line at a time, before saving the image? That way it would just keep trying to read when the image was fired up again.
Hi Jason,
I'm assuming that you are using OS X or other unix based OS (note that OSProcess does not do pipes correctly on Windows, so you won't be able to do that).
Please install CommandShell in addition to OSProcess, as that will give you better ways to control external processes connected by pipes. I'm not clear from your question as to whether Python or Squeak is the "client" in this scenario, but it sounds like you will want to have Squeak making requests to a Python server, but you want Python to set up the pipes and invoke Squeak as a subprocess. I don't have an example of this scenario exactly, but the following may help.
For an example of Squeak as the client, but where Squeak (not Python) does all the work of setting up the pipes connections to the server, see PipeableOSProcess class>>tkExample. This shows Squeak running a tk/tcl interpreter and putting up a trivial UI with buttons connected to Squeak.
For an example of Squeak as the server, communicating through its STDIN and STDOUT streams (which might be pipes set up by Python in your scenario), see class ExternalCommandShell. After evaluating "ExternalCommandShell start" your Squeak image will read command lines from STDIN and respond on STDOUT. The ExternalCommandShell will evaluate input commands and Smalltalk expressions, returning its output to the STDOUT stream. This does not sound like what you want for your project, but it should give you an idea of how these things can be done.
In general, the STDIN steam is "OSProcess thisOSProcess stdIn" and the STDOUT and STDERR streams are "OSProcess thisOSProcess stdOut" and "OSProcess thisOSProcess stdErr". Do not save these in variables, but evaluate them at runtime so your Squeak image can find the right input/output streams even after an image restart. That means you can start a Process in Squeak that reads from stdIn and writes to stdOut. Leave the process running when you save the image, and it will carry on where it left off whenever your Python program restarts the Squeak image.
Note that the stdIn stream for Squeak is (and must be) non-blocking in order to prevent locking up the Squeak VM on a blocked read. That means that your process will need to read available data from stdIn, possibly looping until you get a complete string of whatever you are looking for. For example, if ten characters are currently available in your STDIN input stream, then "OSProcess thisOSProcess stdIn next: 10000" will answer a string of length ten.
ExternalCommandShell does this all in an event-driven manner with no polling, so you can copy that if you want to get fancy. Otherwise a simple polling loop in your process will probably work well enough. Put a 100 ms delay in whatever loop is reading from stdIn an you will get tolerable performance.
Dave
Thanks again Dave for the help. My tack was to "copy" ExternalCommandShell's pattern. StdIn is working fine, but when I write to StdOut nothing seems to actually "get out". Any ideas?
I'm running on Mac, I don't have the AioPlugin installed (I've commented out the inform: that tells about the plugin not being there and therefore polling will be used). I'm not verbose enough at building plugins.
Any ideas why? I can post my Monticello file(s) somewhere if that would help.
I hate to respond to my own post, but the problem was not in the Smalltalk code, it was in the Python server that spawned the image. It was looking for a line feed in the stdout stream.
On Fri, May 15, 2009 at 01:31:57AM -0400, Jason Rogers wrote:
On Tue, May 12, 2009 at 8:59 PM, David T. Lewis lewis@mail.msen.com wrote:
ExternalCommandShell does this all in an event-driven manner with no polling, so you can copy that if you want to get fancy. Otherwise a simple polling loop in your process will probably work well enough. Put a 100 ms delay in whatever loop is reading from stdIn an you will get tolerable performance.
Dave
Thanks again Dave for the help. My tack was to "copy" ExternalCommandShell's pattern. StdIn is working fine, but when I write to StdOut nothing seems to actually "get out". Any ideas?
Try doing a #flush after you write to stdOut, like this:
OSProcess thisOSProcess stdOut nextPutAll: 'hello world'; nextPut: Character lf; flush.
On unix (OS X), the I/O is done through the C stdio library. There is normally some buffering going on at that level, so if you flush the stream it should push the bytes out the tube.
I'm running on Mac, I don't have the AioPlugin installed (I've commented out the inform: that tells about the plugin not being there and therefore polling will be used). I'm not verbose enough at building plugins.
Don't worry about it. AioPlugin is used by CommandShell to do the event-driven I/O. If the plugin is not present, it gives you a warning and falls back on polling loops instead of using the "data ready" events from AioPlugin. For your application, polling loops should be good enough.
Dave
On Fri, May 15, 2009 at 11:29:15AM -0300, Casimiro de Almeida Barreto wrote:
I'm experiencing a similar problem: I have a method that calls an external function via FFI. This function reports its progress via stdout. I'd like such output could be redirected either to Transcript or to a squeak shell (worse). But that's not happening. The routine prints on the shell where squeak was started. When squeak is started in the form of something like: nohup squeak > /dev/null 2> /dev/null & the routine redirects to the "device" (meaning /dev/tty...). One consequence is that how there's no IO perceived by squeak, everything freezes until routine is done (about 12 hours). What would be the best way of redirecting stdio (file descriptor) to Transcript?
Do you mean that the external library is writing to stdout, and you want to be able to see that output from within the Squeak image? In that case you want Squeak to be reading its own standard output, and redirecting it to the Transcript?
There is probably some way to do that, although I can't think of anything right now. If you don't need to be too fancy about it, consider just directing stdout to a file and periodically reading the file contents from Squeak.
Or direct stdout to a named pipe (unix mkfifo command) and read from the named pipe. If you use a named pipe, you will need to open it as a file stream in Squeak, then set the file stream to non-blocking mode before you use it, otherwise you will lock up your Squeak VM. Search for methods with 'setnonblocking' in the name. It's somewhere in OSProcess, but I don't have Squeak with me to check right now.
Dave
David T. Lewis escreveu:
(...) Do you mean that the external library is writing to stdout, and you want to be able to see that output from within the Squeak image? In that case you want Squeak to be reading its own standard output, and redirecting it to the Transcript?
Exact. It is writing to stdout. I want to capture that and be able to transcribe into the Transcript.
There is probably some way to do that, although I can't think of anything right now. If you don't need to be too fancy about it, consider just directing stdout to a file and periodically reading the file contents from Squeak.
Yes... it must be a way of doing that...
Or direct stdout to a named pipe (unix mkfifo command) and read from the named pipe. If you use a named pipe, you will need to open it as a file stream in Squeak, then set the file stream to non-blocking mode before you use it, otherwise you will lock up your Squeak VM. Search for methods with 'setnonblocking' in the name. It's somewhere in OSProcess, but I don't have Squeak with me to check right now.
Dave
On Sat, May 16, 2009 at 10:08:46PM -0300, Casimiro de Almeida Barreto wrote:
David T. Lewis escreveu:
Do you mean that the external library is writing to stdout, and you want to be able to see that output from within the Squeak image? In that case you want Squeak to be reading its own standard output, and redirecting it to the Transcript?
(...)
Yes, I have something like (the most time consuming routine): fInterface apiFannTrain: theFANN onFile: trainingDataFileName times: theMaxEpochs reporting: numberOfEpochs toPrecision: theDesiredError. Where apiFannTrain is: apiFannTrain: aFann onData: aTrainData times: aNumber1 reporting: aNumber2 toPrecision: aFloat < cdecl: void 'fann_train_on_data' ( Fann* FannTrainData* ulong ulong float ) module: 'floatfann' > ^ self externalCallFailed. And that routine outputs progress to stdout. Problems are: 1. As I cannot redirect stdout from inside squeak, I'm not able to watch progress inside squeak (Transcript, morph... you name it) 2. As this kind of output doesn't create context change within squeak VM, apiFannTrain freezes VM until its completion (which may take more than 12 hours). Geez, it's a long time with frozen squeak...
I just figured out that if it was possible to redirect stdout to something inside squeak it would allow me to fix the two problems.
By the way, as the apiFannTrain is not preemptive from inside squeak, there's no use in "fork" it.
Actually, I think that you *do* want to fork it. If the procedure that you are calling from FFI will run for a long time, then you would probably want to have it run in a separate process or thread such that it would not block the Squeak VM in the FFI call.
This is an interesting problem. I do not have an answer for you now, but there must be a way to do it. I will reply back later if I think of a solution.
(By the way, it is hard for me to read your posting because it is in an html format. If you can change your mail to send plain text to the squeak-dev list, that would be helpful.)
Dave
Yes, doing that already. The real problem was that the Python server was expecting a line feed, and so was continuing to wait on the stream's contents, and thus didn't move on with its execution stack.
Thanks for your help!!
On Fri, May 15, 2009 at 7:29 AM, David T. Lewis lewis@mail.msen.com wrote:
On Fri, May 15, 2009 at 01:31:57AM -0400, Jason Rogers wrote:
On Tue, May 12, 2009 at 8:59 PM, David T. Lewis lewis@mail.msen.com wrote:
ExternalCommandShell does this all in an event-driven manner with no polling, so you can copy that if you want to get fancy. Otherwise a simple polling loop in your process will probably work well enough. Put a 100 ms delay in whatever loop is reading from stdIn an you will get tolerable performance.
Dave
Thanks again Dave for the help. My tack was to "copy" ExternalCommandShell's pattern. StdIn is working fine, but when I write to StdOut nothing seems to actually "get out". Any ideas?
Try doing a #flush after you write to stdOut, like this:
OSProcess thisOSProcess stdOut nextPutAll: 'hello world'; nextPut: Character lf; flush.
On unix (OS X), the I/O is done through the C stdio library. There is normally some buffering going on at that level, so if you flush the stream it should push the bytes out the tube.
I'm running on Mac, I don't have the AioPlugin installed (I've commented out the inform: that tells about the plugin not being there and therefore polling will be used). I'm not verbose enough at building plugins.
Don't worry about it. AioPlugin is used by CommandShell to do the event-driven I/O. If the plugin is not present, it gives you a warning and falls back on polling loops instead of using the "data ready" events from AioPlugin. For your application, polling loops should be good enough.
Dave
squeak-dev@lists.squeakfoundation.org
-
Casimiro de Almeida Barreto -
David T. Lewis -
Jason Rogers