Hi,
I managed to make Raspbian unwilling to open any new process. Neither in the X session, nor from Squeak nor from a SSH session. It seems I create Zombie processes (what is that, in the SSH session I ran top and it reports thousands of zombies).
What I do: I poll a GPIO pin every 100ms. When it is high I query I2C via: (PipeableOSProcess command: 'i2cget -y ', busNo, ' ', address) output
and when I find a certain bit set, I switch on some LEDs via: current := ExternalUnixOSProcess command: 'i2cset -y ', busNo, ' ', address, ' ', aString. OSProcess thisOSProcess unregisterChildProcess: current. "This doesn't unregister the process?????" current := nil.
I found 2700++ instances of ExternalOSProcess, emptied the Dictionary via UnixProcess>>unregisterChildProcess and it stopped when it accumulated to another 900+ instances.
This ran for half a day until It can't start another process. First I thought this was the Raspbian update for the PasPi B2 but now I reproduced it on a Pi A+ last updated mid January.
Having read Wikipedia on Zombie Process it seems, that I should try to kill each process I started? What do I do to the PipeableOSProcess? closePipes? release?
Thanks,
Herbert
Hi,
now that I noticed the zombies I can experiment much faster so here some further information:
Stopping Squeak removes all Zombies. Each sensor event creates 3 Zombies, two for the I2Csends, one for the I2Cread. Sending the PipeableOSProcess instances #closePipes then #release doesn't help. Sending the ExternalUnixProcess instances #terminate doesn't help either. Sending an additional PipeableOSProcess>>unixCommandPipeLine doesn't add a Zombie but returns ''. PipeableOSProcess>>tkExample works fine and doesn't add Zombies.
So is it a problem of I2Ctools? Should I try other Linux commands?
Thanks,
Herbert
Am 07.02.2015 um 18:11 schrieb Herbert König:
Hi,
I managed to make Raspbian unwilling to open any new process. Neither in the X session, nor from Squeak nor from a SSH session. It seems I create Zombie processes (what is that, in the SSH session I ran top and it reports thousands of zombies).
What I do: I poll a GPIO pin every 100ms. When it is high I query I2C via: (PipeableOSProcess command: 'i2cget -y ', busNo, ' ', address) output
and when I find a certain bit set, I switch on some LEDs via: current := ExternalUnixOSProcess command: 'i2cset -y ', busNo, ' ', address, ' ', aString. OSProcess thisOSProcess unregisterChildProcess: current. "This doesn't unregister the process?????" current := nil.
I found 2700++ instances of ExternalOSProcess, emptied the Dictionary via UnixProcess>>unregisterChildProcess and it stopped when it accumulated to another 900+ instances.
This ran for half a day until It can't start another process. First I thought this was the Raspbian update for the PasPi B2 but now I reproduced it on a Pi A+ last updated mid January.
Having read Wikipedia on Zombie Process it seems, that I should try to kill each process I started? What do I do to the PipeableOSProcess? closePipes? release?
Thanks,
Herbert
Hi Herbert,
On Sat, Feb 07, 2015 at 07:46:43PM +0100, Herbert K??nig wrote:
Hi,
now that I noticed the zombies I can experiment much faster so here some further information:
Stopping Squeak removes all Zombies. Each sensor event creates 3 Zombies, two for the I2Csends, one for the I2Cread. Sending the PipeableOSProcess instances #closePipes then #release doesn't help. Sending the ExternalUnixProcess instances #terminate doesn't help either. Sending an additional PipeableOSProcess>>unixCommandPipeLine doesn't add a Zombie but returns ''. PipeableOSProcess>>tkExample works fine and doesn't add Zombies.
So is it a problem of I2Ctools? Should I try other Linux commands?
This looks like some kind of problem in OSProcess. I am not sure what is wrong, so let me try to explain how it works.
In Unix, a zombie process is a process that has exited, but its parent process has not yet acknowledged that it has exited. When you run a command with OSProcess, it starts a new process that is a child of the VM process. When that external process exits, it is the responsibility of OSProcess (running in the VM process) to notice this, and to acknowledge the exit by calling the Unix waitpid() system call.
When things are working right, you should be able to open a process browser in Squeak, and see an active process labeled "the child OSProcess watcher". That process is normally blocking on a Semaphore, and it wake up whenever the semaphore is signalled (which occurs when the OSProcess plugin detects a SIGCHLD signal from the operating system). Each time it wakes up, it calls a primitive (UnixOSProcessPlugin>>primitiveReapChildProcess). That primitive uses a call to waitpid() to obtain the exit status of the child process, and as soon as that happens the process is no longer in a zombie state.
I am not sure where this is going wrong, but one thing you can try is to restart the child watcher process, and see if that makes the zombies go away:
"OSProcess accessor restartChildWatcherProcess"
The actual child watcher process is UnixOSProcessAccessor>>grimReaperProcess, so you may want to look at that and see if it gives you any other ideas.
Please let us know if you figure out the problem. Whatever is going wrong here, I definitely want to get it fixed.
Thanks, Dave
Thanks,
Herbert
Am 07.02.2015 um 18:11 schrieb Herbert K??nig:
Hi,
I managed to make Raspbian unwilling to open any new process. Neither in the X session, nor from Squeak nor from a SSH session. It seems I create Zombie processes (what is that, in the SSH session I ran top and it reports thousands of zombies).
What I do: I poll a GPIO pin every 100ms. When it is high I query I2C via: (PipeableOSProcess command: 'i2cget -y ', busNo, ' ', address) output
and when I find a certain bit set, I switch on some LEDs via: current := ExternalUnixOSProcess command: 'i2cset -y ', busNo, ' ', address, ' ', aString. OSProcess thisOSProcess unregisterChildProcess: current. "This doesn't unregister the process?????" current := nil.
I found 2700++ instances of ExternalOSProcess, emptied the Dictionary via UnixProcess>>unregisterChildProcess and it stopped when it accumulated to another 900+ instances.
This ran for half a day until It can't start another process. First I thought this was the Raspbian update for the PasPi B2 but now I reproduced it on a Pi A+ last updated mid January.
Having read Wikipedia on Zombie Process it seems, that I should try to kill each process I started? What do I do to the PipeableOSProcess? closePipes? release?
Thanks,
Herbert
Hi David, Am 08.02.2015 um 21:04 schrieb David T. Lewis:
This looks like some kind of problem in OSProcess. I am not sure what is wrong, so let me try to explain how it works. In Unix, a zombie process is a process that has exited, but its parent process has not yet acknowledged that it has exited. When you run a command with OSProcess, it starts a new process that is a child of the VM process. When that external process exits, it is the responsibility of OSProcess (running in the VM process) to notice this, and to acknowledge the exit by calling the Unix waitpid() system call. When things are working right, you should be able to open a process browser in Squeak, and see an active process labeled "the child OSProcess watcher".
I see that process while producing Zombies
That process is normally blocking on a Semaphore, and it wake up whenever the semaphore is signalled (which occurs when the OSProcess plugin detects a SIGCHLD signal from the operating system). Each time it wakes up, it calls a primitive (UnixOSProcessPlugin>>primitiveReapChildProcess). That primitive uses a call to waitpid() to obtain the exit status of the child process, and as soon as that happens the process is no longer in a zombie state. I am not sure where this is going wrong, but one thing you can try is to restart the child watcher process, and see if that makes the zombies go away: "OSProcess accessor restartChildWatcherProcess"
leaves the number of Zombies constant and continues to produce new ones on each call of ExternalUnixOSProcess>>command
The actual child watcher process is UnixOSProcessAccessor>>grimReaperProcess, so you may want to look at that
That looked like more than I can handle before I'm off again so I tried something different. On Saturday I had bookmarked: http://en.wikipedia.org/wiki/Zombie_process where the fourth paragraph of the overview section talks about sending SIGCHLD so I tried that. It works but only adds to the mystery.
For each event in the real world (me moving in front of a PIR sensor) I send one PipeableOSProcess command: 'i2cget -y 1 0x20) and two ExternalUnixOSProcess command: 'i2cset -y 1 0x20', commandByte
This gave me three new Zombies per event. PipeableOSProcess does not understand sigchld so I only sent sigchld to the two ExternalUnixOSProcess end expected a reduction to one Zombie per event. But no more Zombies are produced. We called that "Fehler unerkannt entkommen" (roughly "the error escaped incognito") and I'm very interested what happened here.
and see if it gives you any other ideas. Please let us know if you figure out the problem. Whatever is going wrong here, I definitely want to get it fixed. Thanks, Dave
Later I'll try to step through grimReaperProcess to see what I can learn. And then I'll also look at the version of OSProcess you recently uploaded. Also the other mystery: When I overflowed the process table I couldn't from a terminal do 'ls | grep something' but I could do that via ExternalUnixOSProcess without creating additional Zombies. So which commands create Zombies and which don't?
Cheers,
Herbert
Hi,
just a short follow up.
Am 09.02.2015 um 15:38 schrieb Herbert König:
Hi David, Am 08.02.2015 um 21:04 schrieb David T. Lewis:
This looks like some kind of problem in OSProcess. I am not sure what is wrong, so let me try to explain how it works.
several updates of Raspbian later I do not need to send sigchld any more. So I assume it was a bug in Raspbian introduced by the transition to the Pi 2. Nothing to do with OSProcess.
Also I now use the version of OSProcess which limits the number of children it holds on to.
This has run uninterrupted now for two weeks without a single more Zombie and Squeak's memory consumption being constant.
Thanks,
Herbert
On Mon, Feb 23, 2015 at 08:27:42PM +0100, Herbert K?nig wrote:
Hi,
just a short follow up.
Hi Herbert,
Thank you for the follow up message, much appreciated!
Dave
Am 09.02.2015 um 15:38 schrieb Herbert K?nig:
Hi David, Am 08.02.2015 um 21:04 schrieb David T. Lewis:
This looks like some kind of problem in OSProcess. I am not sure what is wrong, so let me try to explain how it works.
several updates of Raspbian later I do not need to send sigchld any more. So I assume it was a bug in Raspbian introduced by the transition to the Pi 2. Nothing to do with OSProcess.
Also I now use the version of OSProcess which limits the number of children it holds on to.
This has run uninterrupted now for two weeks without a single more Zombie and Squeak's memory consumption being constant.
Thanks,
Herbert
squeak-dev@lists.squeakfoundation.org