Hello,
I've been unable to find good documentation / examples of an OSProcess use case. How can I start an external process that remains running while I pipe data into its stdin and read data from its stdout?
To be specific, I am starting to toy around with the JS/Typescript LSP server https://github.com/typescript-language-server/typescript-language-server, which seems to expect input on stdin (rather than using sockets or whatnot)
Thanks!
Hey Eric,
probably not following the best practices of OSProcess either but you may find some code useful for copying here regardless: https://github.com/hpi-swa-lab/sb-tree-sitter/blob/master/packages/Sandblock...
Best, Tom
On Thu, Aug 25, 2022 at 2:10 PM Eric Gade eric.gade@gmail.com wrote:
Hello,
I've been unable to find good documentation / examples of an OSProcess use case. How can I start an external process that remains running while I pipe data into its stdin and read data from its stdout?
To be specific, I am starting to toy around with the JS/Typescript LSP server https://github.com/typescript-language-server/typescript-language-server, which seems to expect input on stdin (rather than using sockets or whatnot)
Thanks!
-- Eric
Oh wow, this is excellent, thanks! And I'll have to check out this project of course.
I was hoping to work on a general-purpose LSP client using NeoJSON, but I'm going to poke around at this first. Sweet!
On Thu, Aug 25, 2022 at 8:17 AM Tom Beckmann tomjonabc@gmail.com wrote:
Hey Eric,
probably not following the best practices of OSProcess either but you may find some code useful for copying here regardless:
https://github.com/hpi-swa-lab/sb-tree-sitter/blob/master/packages/Sandblock...
Hi Eric,
On Thu, Aug 25, 2022 at 08:10:35AM -0400, Eric Gade wrote:
Hello,
I've been unable to find good documentation / examples of an OSProcess use case. How can I start an external process that remains running while I pipe data into its stdin and read data from its stdout?
For this you will want to use a PipeableOSProcess, which allows you to start and external process, write to its stdin, and read from its stdout and stderr. The class comment gives an overview.
PipeableOSProcess is part of the CommandShell package, so be sure to load CommandShell along with OSProcess.
To be specific, I am starting to toy around with the JS/Typescript LSP server https://github.com/typescript-language-server/typescript-language-server, which seems to expect input on stdin (rather than using sockets or whatnot)
If you happen to have tk/wish installed on your computer, try evaluating "PipeableOSProcess tkExample" to see something that may be similar to what you have in mind. It starts an external process running /usr/bin/wish with pipes connected to stdin/stdout/stderr to send commands to the wish shell and to read its output.
HTH,
Dave
I am having a problem (I think) getting even basic external process to run. For example, if I execute this code and inspect it: ``` proc := ExternalUnixOSProcess forkAndExec: '/opt/homebrew/bin/typescript-language-server' arguments: { '--stdio' } environment: nil ```
The process status is complete immediately (if it works the process should run until explicitly terminated). This is also the case if I append `/bin/zsh ` or `/bin/sh ` to the command.
What am I missing?
On Thu, Aug 25, 2022 at 8:41 AM David T. Lewis lewis@mail.msen.com wrote:
Hi Eric,
On Thu, Aug 25, 2022 at 08:10:35AM -0400, Eric Gade wrote:
Hello,
I've been unable to find good documentation / examples of an OSProcess
use
case. How can I start an external process that remains running while I
pipe
data into its stdin and read data from its stdout?
For this you will want to use a PipeableOSProcess, which allows you to start and external process, write to its stdin, and read from its stdout and stderr. The class comment gives an overview.
PipeableOSProcess is part of the CommandShell package, so be sure to load CommandShell along with OSProcess.
To be specific, I am starting to toy around with the JS/Typescript LSP server <
https://github.com/typescript-language-server/typescript-language-server%3E,
which seems to expect input on stdin (rather than using sockets or
whatnot)
If you happen to have tk/wish installed on your computer, try evaluating "PipeableOSProcess tkExample" to see something that may be similar to what you have in mind. It starts an external process running /usr/bin/wish with pipes connected to stdin/stdout/stderr to send commands to the wish shell and to read its output.
HTH,
Dave
Some things you could try:
* What's the return code in the inspector? * Does it work if you run the lsp from a terminal? * Does e.g. /bin/ls work? * If you start squeak from a terminal, is something printed to its stdout/stderr?
Best, Tom
On Thu, Aug 25, 2022, 16:14 Eric Gade eric.gade@gmail.com wrote:
I am having a problem (I think) getting even basic external process to run. For example, if I execute this code and inspect it:
proc := ExternalUnixOSProcess forkAndExec: '/opt/homebrew/bin/typescript-language-server' arguments: { '--stdio' } environment: nil
The process status is complete immediately (if it works the process should run until explicitly terminated). This is also the case if I append `/bin/zsh ` or `/bin/sh ` to the command.
What am I missing?
On Thu, Aug 25, 2022 at 8:41 AM David T. Lewis lewis@mail.msen.com wrote:
Hi Eric,
On Thu, Aug 25, 2022 at 08:10:35AM -0400, Eric Gade wrote:
Hello,
I've been unable to find good documentation / examples of an OSProcess
use
case. How can I start an external process that remains running while I
pipe
data into its stdin and read data from its stdout?
For this you will want to use a PipeableOSProcess, which allows you to start and external process, write to its stdin, and read from its stdout and stderr. The class comment gives an overview.
PipeableOSProcess is part of the CommandShell package, so be sure to load CommandShell along with OSProcess.
To be specific, I am starting to toy around with the JS/Typescript LSP server <
https://github.com/typescript-language-server/typescript-language-server
, which seems to expect input on stdin (rather than using sockets or
whatnot)
If you happen to have tk/wish installed on your computer, try evaluating "PipeableOSProcess tkExample" to see something that may be similar to what you have in mind. It starts an external process running /usr/bin/wish with pipes connected to stdin/stdout/stderr to send commands to the wish shell and to read its output.
HTH,
Dave
-- Eric
I am a little out of my depth here so please bear with me! Answers inline:
On Thu, Aug 25, 2022 at 10:55 AM Tom Beckmann tomjonabc@gmail.com wrote:
Some things you could try:
- What's the return code in the inspector?
The exitStatus is 32512
- Does it work if you run the lsp from a terminal?
It works in my terminal if I just execute `typescript-language-server` or `/opt/homebrew/bin/typescript-language-server`. However, this is just a javascript file that the shell executes in a node environment using `#!/usr/bin/env node` at the top.
- Does e.g. /bin/ls work?
The exitStatus is 0, so I think so. However, when I try to inspect initialStdOut, it doesn't appear to contain any data anywhere.
- If you start squeak from a terminal, is something printed to its
stdout/stderr?
In the case of /bin/ls, yes, it outputs the result to the terminal if I run Squeak from a terminal (on this Mac I don't usually do so).
Thanks!
Also just guessing here :) A brief google search for the return code seemed to suggest that it's sometimes used for "file not found" -- not sure from which component that would come from though. If you are using nvm, maybe nodejs only gets added to your PATH if you're starting from your zsh?
In addition, you could try adding these params to the language server invocation: {'--stdio'. '--log-level'. '4'. '--tsserver-log-file'. '/tmp/ts.log'. '--tsserver-log-verbosity'. 'verbose'} and see if the log file is created.
Best, Tom
On Fri, Aug 26, 2022 at 12:28 AM Eric Gade eric.gade@gmail.com wrote:
I am a little out of my depth here so please bear with me! Answers inline:
On Thu, Aug 25, 2022 at 10:55 AM Tom Beckmann tomjonabc@gmail.com wrote:
Some things you could try:
- What's the return code in the inspector?
The exitStatus is 32512
- Does it work if you run the lsp from a terminal?
It works in my terminal if I just execute `typescript-language-server` or `/opt/homebrew/bin/typescript-language-server`. However, this is just a javascript file that the shell executes in a node environment using `#!/usr/bin/env node` at the top.
- Does e.g. /bin/ls work?
The exitStatus is 0, so I think so. However, when I try to inspect initialStdOut, it doesn't appear to contain any data anywhere.
- If you start squeak from a terminal, is something printed to its
stdout/stderr?
In the case of /bin/ls, yes, it outputs the result to the terminal if I run Squeak from a terminal (on this Mac I don't usually do so).
Thanks!
-- Eric
I believe what is happening is that the various #command: messages in OSProcess classes are executing without searching my PATH. Even if I add a copy of the current running process environment and set that as the environment of a new PipeableOSProcess, for example, I can't even get a result for `which node`. I can see that /usr/bin and /usr/sbin are on the path from within Squeak.
Another example is that `PipeableOSProcess class >> #unixCommandPipeLine` responds with an empty string when I inspect it.
Might this all be related to newer macOS settings somehow?
On Fri, Aug 26, 2022 at 1:08 AM Tom Beckmann tomjonabc@gmail.com wrote:
Also just guessing here :) A brief google search for the return code seemed to suggest that it's sometimes used for "file not found" -- not sure from which component that would come from though. If you are using nvm, maybe nodejs only gets added to your PATH if you're starting from your zsh?
In addition, you could try adding these params to the language server invocation: {'--stdio'. '--log-level'. '4'. '--tsserver-log-file'. '/tmp/ts.log'. '--tsserver-log-verbosity'. 'verbose'} and see if the log file is created.
Best, Tom
On Fri, Aug 26, 2022 at 12:28 AM Eric Gade eric.gade@gmail.com wrote:
I am a little out of my depth here so please bear with me! Answers inline:
On Thu, Aug 25, 2022 at 10:55 AM Tom Beckmann tomjonabc@gmail.com wrote:
Some things you could try:
- What's the return code in the inspector?
The exitStatus is 32512
- Does it work if you run the lsp from a terminal?
It works in my terminal if I just execute `typescript-language-server` or `/opt/homebrew/bin/typescript-language-server`. However, this is just a javascript file that the shell executes in a node environment using `#!/usr/bin/env node` at the top.
- Does e.g. /bin/ls work?
The exitStatus is 0, so I think so. However, when I try to inspect initialStdOut, it doesn't appear to contain any data anywhere.
- If you start squeak from a terminal, is something printed to its
stdout/stderr?
In the case of /bin/ls, yes, it outputs the result to the terminal if I run Squeak from a terminal (on this Mac I don't usually do so).
Thanks!
-- Eric
Hi Eric and Tom,
On Fri, Aug 26, 2022 at 04:27:56PM -0400, Eric Gade wrote:
I believe what is happening is that the various #command: messages in OSProcess classes are executing without searching my PATH. Even if I add a copy of the current running process environment and set that as the environment of a new PipeableOSProcess, for example, I can't even get a result for `which node`. I can see that /usr/bin and /usr/sbin are on the path from within Squeak.
Yes that is correct. On a unix system, the interpretation of PATH (and lots of other things) are done by the shell, which is typically an executable program such as /bin/bash. The actual programs and commands that you run on a unix system are subprocesses of that shell program. So the shell interprets the PATH and uses it to find the program that you want to run.
In Squeak with OSProcess, your Squeak virtual machine is the "shell" and you can directly run programs by creating subprocesses of the VM process. Of course, you can chose to run the executable program /bin/bash, in which case you have a real unix shell, and you can let that shell run other commands in the normal way.
You can also run any other executable directly (and I think this is what you have been doing). In that case there is no unix shell program involved, and you would need to specify the fully qualified path to the program, and you would need to do any necessary command parameter expansion that would otherwise have been done by the unix shell.
In OSProcess/CommandShell there is also a pure Smalltalk emulation of a simple unix shell. This is by no means a complete emulation of /bin/bash but it is sufficient to handle things like PATH expansion and command pipeline composition without requiring the actual /bin/bash shell.
So to tie the pieces together, if you want to run the command "which node" you can do this:
OSProcess outputOf: 'which node'
This uses a CommandShell in Squeak to do the shell processing to locate and run the program /usr/bin/which (so no need for /bin/bash). It takes some patience but if you step through this it a debugger you will be able to see what is going on. The Smalltalk command shell emulation sets everything up so that OSProcess can run the /usr/bin/which program as a direct subprocess of the Squeak VM.
Another example is that `PipeableOSProcess class >> #unixCommandPipeLine` responds with an empty string when I inspect it.
Might this all be related to newer macOS settings somehow?
This method is just an example to show how to compose a command pipeline with some programs that would be found on a typical Unix/Linux system. I'm not sure what it does on Mac. It uses the programs 'ps', 'grep', and 'cut', so if any of these is not found on your system, it probably will just give an empty string in the output (plus some error output that you can probably find if you dig through it in a debugger).
Dave
Hi Eric,
Maybe it will help if I also give an example of Squeak interacting with an external program through stdio pipes Here I will use /bin/bash as the exteral program. You would want to substitute that with the full path to whatever program you are actually trying to run.
You can experiment with the example below by copying it into a workspace:
"Create an external OS process running the /bin/bash program. The program will be connected to Squeak through stdio pipes." bashProxy := PipeableOSProcess command: '/bin/bash'.
"Send a command line to the bash program, followed by a line terminator, and flush the stream so the command is available to the bash program." bashProxy nextPutAll: 'who'; nextPut: Character lf; flush.
"Wait a short time to give the bash program time to run, and to send the result back to Squeak." (Delay forMilliseconds: 20) wait.
"Read all available data from the stdout stream of the bash process. Note also methods #errorUpToEndOfFile, #upToEndOfFile." bashProxy upToEnd.
"When you are done interacting with your bash program, close it and allow the external process to exit." bashProxy close
Dave
On Fri, Aug 26, 2022 at 11:19:04PM -0400, David T. Lewis wrote:
Hi Eric and Tom,
On Fri, Aug 26, 2022 at 04:27:56PM -0400, Eric Gade wrote:
I believe what is happening is that the various #command: messages in OSProcess classes are executing without searching my PATH. Even if I add a copy of the current running process environment and set that as the environment of a new PipeableOSProcess, for example, I can't even get a result for `which node`. I can see that /usr/bin and /usr/sbin are on the path from within Squeak.
Yes that is correct. On a unix system, the interpretation of PATH (and lots of other things) are done by the shell, which is typically an executable program such as /bin/bash. The actual programs and commands that you run on a unix system are subprocesses of that shell program. So the shell interprets the PATH and uses it to find the program that you want to run.
In Squeak with OSProcess, your Squeak virtual machine is the "shell" and you can directly run programs by creating subprocesses of the VM process. Of course, you can chose to run the executable program /bin/bash, in which case you have a real unix shell, and you can let that shell run other commands in the normal way.
You can also run any other executable directly (and I think this is what you have been doing). In that case there is no unix shell program involved, and you would need to specify the fully qualified path to the program, and you would need to do any necessary command parameter expansion that would otherwise have been done by the unix shell.
In OSProcess/CommandShell there is also a pure Smalltalk emulation of a simple unix shell. This is by no means a complete emulation of /bin/bash but it is sufficient to handle things like PATH expansion and command pipeline composition without requiring the actual /bin/bash shell.
So to tie the pieces together, if you want to run the command "which node" you can do this:
OSProcess outputOf: 'which node'
This uses a CommandShell in Squeak to do the shell processing to locate and run the program /usr/bin/which (so no need for /bin/bash). It takes some patience but if you step through this it a debugger you will be able to see what is going on. The Smalltalk command shell emulation sets everything up so that OSProcess can run the /usr/bin/which program as a direct subprocess of the Squeak VM.
Another example is that `PipeableOSProcess class >> #unixCommandPipeLine` responds with an empty string when I inspect it.
Might this all be related to newer macOS settings somehow?
This method is just an example to show how to compose a command pipeline with some programs that would be found on a typical Unix/Linux system. I'm not sure what it does on Mac. It uses the programs 'ps', 'grep', and 'cut', so if any of these is not found on your system, it probably will just give an empty string in the output (plus some error output that you can probably find if you dig through it in a debugger).
Dave
Hi Dave,
Thanks for the comprehensive explanation and examples. I'm using Squeak 6.1 here, which might be an issue, but the following message responds with a DNU:
OSProcess outputOf: 'which node'
I don't see any method in the system called #outputOf:
I installed OSProcess and CommandShell via the optional additions during the setup wizard.
On Sat, Aug 27, 2022 at 10:36 AM David T. Lewis lewis@mail.msen.com wrote:
Hi Eric,
Maybe it will help if I also give an example of Squeak interacting with an external program through stdio pipes Here I will use /bin/bash as the exteral program. You would want to substitute that with the full path to whatever program you are actually trying to run.
You can experiment with the example below by copying it into a workspace:
"Create an external OS process running the /bin/bash program. The program will be connected to Squeak through stdio pipes." bashProxy := PipeableOSProcess command: '/bin/bash'. "Send a command line to the bash program, followed by a line terminator, and flush the stream so the command is available to the bash program." bashProxy nextPutAll: 'who'; nextPut: Character lf; flush. "Wait a short time to give the bash program time to run, and to send the result back to Squeak." (Delay forMilliseconds: 20) wait. "Read all available data from the stdout stream of the bash process. Note also methods #errorUpToEndOfFile, #upToEndOfFile." bashProxy upToEnd. "When you are done interacting with your bash program, close it and allow the external process to exit." bashProxy close
Dave
On Fri, Aug 26, 2022 at 11:19:04PM -0400, David T. Lewis wrote:
Hi Eric and Tom,
On Fri, Aug 26, 2022 at 04:27:56PM -0400, Eric Gade wrote:
I believe what is happening is that the various #command: messages in OSProcess classes are executing without searching my PATH. Even if I
add a
copy of the current running process environment and set that as the environment of a new PipeableOSProcess, for example, I can't even get a result for `which node`. I can see that /usr/bin and /usr/sbin are on
the
path from within Squeak.
Yes that is correct. On a unix system, the interpretation of PATH (and lots of other things) are done by the shell, which is typically an executable program such as /bin/bash. The actual programs and commands that you run on a unix system are subprocesses of that shell program. So the shell interprets the PATH and uses it to find the program that you want to run.
In Squeak with OSProcess, your Squeak virtual machine is the "shell" and you can directly run programs by creating subprocesses of the VM process. Of course, you can chose to run the executable program /bin/bash, in which case you have a real unix shell, and you can let that shell run other commands in the normal way.
You can also run any other executable directly (and I think this is what you have been doing). In that case there is no unix shell program involved, and you would need to specify the fully qualified path to the program, and you would need to do any necessary command parameter expansion that would otherwise have been done by the unix shell.
In OSProcess/CommandShell there is also a pure Smalltalk emulation of a simple unix shell. This is by no means a complete emulation of /bin/bash but it is sufficient to handle things like PATH expansion and command pipeline composition without requiring the actual /bin/bash shell.
So to tie the pieces together, if you want to run the command "which
node"
you can do this:
OSProcess outputOf: 'which node'
This uses a CommandShell in Squeak to do the shell processing to locate and run the program /usr/bin/which (so no need for /bin/bash). It takes some patience but if you step through this it a debugger you will be able to see what is going on. The Smalltalk command shell emulation sets everything up so that OSProcess can run the /usr/bin/which program as a direct subprocess of the Squeak VM.
Another example is that `PipeableOSProcess class >>
#unixCommandPipeLine`
responds with an empty string when I inspect it.
Might this all be related to newer macOS settings somehow?
This method is just an example to show how to compose a command pipeline with some programs that would be found on a typical Unix/Linux system. I'm not sure what it does on Mac. It uses the programs 'ps', 'grep', and 'cut', so if any of these is not found on your system, it probably will just give an empty string in the output (plus some error output that you can probably find if you dig through it in a debugger).
Dave
Hi Eric,
On Sat, Aug 27, 2022 at 10:44:01AM -0400, Eric Gade wrote:
Hi Dave,
Thanks for the comprehensive explanation and examples. I'm using Squeak 6.1 here, which might be an issue, but the following message responds with a DNU:
OSProcess outputOf: 'which node'
I don't see any method in the system called #outputOf:
I installed OSProcess and CommandShell via the optional additions during the setup wizard.
I cannot explain why, but it sounds like you may have an older version of OSProcess/CommandShell. The setup wizard seems to be working for me (I just tried it again on a fresh Squeak 6.0 release image). The wizard seems to be using Metacello to load the packages, which does not seem right to me, but it nevertheless seems to work and load the right packages.
All that I can suggest is to use a regular Monticello browser and open up the OSProcess and CommandShell repositories to make sure that the latest versions are installed. The repositories are at https://www.squeaksource.com/OSProcess and https://www.squeaksource.com/CommandShell.
If anything does not seem up to date, then load OSProcess-dtl.130.mcz and CommandShell-dtl.111.mcz.
Dave
squeak-dev@lists.squeakfoundation.org