Hi All,
I tried to update my Squeak image today, and found that source.squeak.org is down. On box2 I found that it's using all the cpu. Based on the apachelogs it went down 7-8 hours ago. I decided to restart it. svc -t didn't kill the image (nor did -k), instead it spawned another image, so I stopped svc, killed the process manually, and restarted it with svc. The image is up now, but I wonder if there's anything else to do when a SqueakSource image goes down.
Levente
Just on the topic, I find daemontools confounding. The svc tool is supposed to start and stop processes, but I also find it doesn’t work. As I understand it, svc is supposed to control a supervise process, which overlooks a process like source.squeak.org http://source.squeak.org/. But how does that start exactly? I think there’s more than one way. I think creating a symlink to to the /service directory is supposed to do it. I’ve also used something like “supervise foo&”, which I figure is a hack.
daemontools is great because it restarts processes automatically when rebooting. But the svc and svstat haven’t proven that useful in my experience. And daemontools is state-of-the-art 2004. I’m looking at systemd (which netstyle.ch uses to start and stop its hundreds of client images from requests sent to the box), which I think supersedes the daemontools feature set. And I understand, it will be PID1 default on some Linuxes in the near future. It is possible that sometime the use of daemontools on Squeak boxes might be reconsidered. Not my decision, of course, but as the topic came out, I thought I’d brain dump. I want to like daemontools more than I find I’m able to.
And on the topic of systemd, I’d say this, it’s a big leap. Back up everything and review what you know about GRUB2. I’ve found that out the hard way. I’m learning on a dedicated laptop and I lost my UI. Couldn’t get it back. Could only get a terminal on tty1. So I created an install stick and re-installed Ubuntu 14.04 from USB. Now I can try, fail, and have a back up plan, learn more about GRUB2, systemd, /etc/default/grub, systemctl, unit config files, etc. I don’t think anybody would want that experience on a production box. It’s a dramatic leap. Oh yea.
Chris
On Oct 21, 2014, at 7:27 AM, Levente Uzonyi leves@elte.hu wrote:
Hi All,
I tried to update my Squeak image today, and found that source.squeak.org is down. On box2 I found that it's using all the cpu. Based on the apachelogs it went down 7-8 hours ago. I decided to restart it. svc -t didn't kill the image (nor did -k), instead it spawned another image, so I stopped svc, killed the process manually, and restarted it with svc. The image is up now, but I wonder if there's anything else to do when a SqueakSource image goes down.
Levente
On 21.10.2014, at 08:30, Chris Cunnington brasspen@gmail.com wrote:
Just on the topic, I find daemontools confounding. The svc tool is supposed to start and stop processes, but I also find it doesn’t work. As I understand it, svc is supposed to control a supervise process, which overlooks a process like source.squeak.org. But how does that start exactly? I think there’s more than one way. I think creating a symlink to to the /service directory is supposed to do it. I’ve also used something like “supervise foo&”, which I figure is a hack.
this is a hack indeed.
daemontools is great because it restarts processes automatically when rebooting. But the svc and svstat haven’t proven that useful in my experience. And daemontools is state-of-the-art 2004. I’m looking at systemd (which netstyle.ch uses to start and stop its hundreds of client images from requests sent to the box), which I think supersedes the daemontools feature set. And I understand, it will be PID1 default on some Linuxes in the near future. It is possible that sometime the use of daemontools on Squeak boxes might be reconsidered. Not my decision, of course, but as the topic came out, I thought I’d brain dump. I want to like daemontools more than I find I’m able to.
And on the topic of systemd, I’d say this, it’s a big leap. Back up everything and review what you know about GRUB2. I’ve found that out the hard way. I’m learning on a dedicated laptop and I lost my UI. Couldn’t get it back. Could only get a terminal on tty1. So I created an install stick and re-installed Ubuntu 14.04 from USB. Now I can try, fail, and have a back up plan, learn more about GRUB2, systemd, /etc/default/grub, systemctl, unit config files, etc. I don’t think anybody would want that experience on a production box. It’s a dramatic leap. Oh yea.
Perhaps I can help out here and there? I use daemontools for some years now without any noticeable problems, really. Yes, the mindset is different from other process supervision systems, but then again, each of them has its downsides. I haven't heard too much good stuff about systemd; it allegedly does “too much” (including being a DHCP client and gui interaction…). I'd really like to help to keep daemontools as it is.
Best -Tobias
On Tue, Oct 21, 2014 at 08:36:55AM -0700, Tobias Pape wrote:
On 21.10.2014, at 08:30, Chris Cunnington brasspen@gmail.com wrote:
Just on the topic, I find daemontools confounding. The svc tool is supposed to start and stop processes, but I also find it doesn?t work. As I understand it, svc is supposed to control a supervise process, which overlooks a process like source.squeak.org. But how does that start exactly? I think there?s more than one way. I think creating a symlink to to the /service directory is supposed to do it. I?ve also used something like ?supervise foo&?, which I figure is a hack.
this is a hack indeed.
daemontools is great because it restarts processes automatically when rebooting. But the svc and svstat haven?t proven that useful in my experience. And daemontools is state-of-the-art 2004. I?m looking at systemd (which netstyle.ch uses to start and stop its hundreds of client images from requests sent to the box), which I think supersedes the daemontools feature set. And I understand, it will be PID1 default on some Linuxes in the near future. It is possible that sometime the use of daemontools on Squeak boxes might be reconsidered. Not my decision, of course, but as the topic came out, I thought I?d brain dump. I want to like daemontools more than I find I?m able to.
And on the topic of systemd, I?d say this, it?s a big leap. Back up everything and review what you know about GRUB2. I?ve found that out the hard way. I?m learning on a dedicated laptop and I lost my UI. Couldn?t get it back. Could only get a terminal on tty1. So I created an install stick and re-installed Ubuntu 14.04 from USB. Now I can try, fail, and have a back up plan, learn more about GRUB2, systemd, /etc/default/grub, systemctl, unit config files, etc. I don?t think anybody would want that experience on a production box. It?s a dramatic leap. Oh yea.
Perhaps I can help out here and there? I use daemontools for some years now without any noticeable problems, really. Yes, the mindset is different from other process supervision systems, but then again, each of them has its downsides. I haven't heard too much good stuff about systemd; it allegedly does ?too much? (including being a DHCP client and gui interaction?). I'd really like to help to keep daemontools as it is.
+1000
Ken forced me to learn daemontools in order to set up squeaksource.com on box3, and I am glad that he did. It is the perfect tool for running Squeak based services on a system like this. Once I figured out how it worked, it has been no problem at all, and it eliminated the usual hackish scripts that I might otherwise have been tempted to invent. It is also a good way to ensure that our various services are managed in similar ways, so that people other than the original author can figure out how they are set up, and what to do when they stop working.
FWIW, I don't think I have had to touch anything related to squeaksource.com since at least last February, and even that was just an image update that I did through VNC.
Dave
Just to add to the confusion/discussion - at work I’m very happy playing with Docker and (under Ubuntu 14.04) upstart for supervision, which looks quite a bit like daemontools. The advantage of wrapping Squeak processes in Docker containers would be that you can test locally, etcetera.
Just my 2 cents,
Cees
On 21Oct, 2014, at 17:34, David T. Lewis lewis@mail.msen.com wrote:
On Tue, Oct 21, 2014 at 08:36:55AM -0700, Tobias Pape wrote:
On 21.10.2014, at 08:30, Chris Cunnington brasspen@gmail.com wrote:
Just on the topic, I find daemontools confounding. The svc tool is supposed to start and stop processes, but I also find it doesn?t work. As I understand it, svc is supposed to control a supervise process, which overlooks a process like source.squeak.org. But how does that start exactly? I think there?s more than one way. I think creating a symlink to to the /service directory is supposed to do it. I?ve also used something like ?supervise foo&?, which I figure is a hack.
this is a hack indeed.
daemontools is great because it restarts processes automatically when rebooting. But the svc and svstat haven?t proven that useful in my experience. And daemontools is state-of-the-art 2004. I?m looking at systemd (which netstyle.ch uses to start and stop its hundreds of client images from requests sent to the box), which I think supersedes the daemontools feature set. And I understand, it will be PID1 default on some Linuxes in the near future. It is possible that sometime the use of daemontools on Squeak boxes might be reconsidered. Not my decision, of course, but as the topic came out, I thought I?d brain dump. I want to like daemontools more than I find I?m able to.
And on the topic of systemd, I?d say this, it?s a big leap. Back up everything and review what you know about GRUB2. I?ve found that out the hard way. I?m learning on a dedicated laptop and I lost my UI. Couldn?t get it back. Could only get a terminal on tty1. So I created an install stick and re-installed Ubuntu 14.04 from USB. Now I can try, fail, and have a back up plan, learn more about GRUB2, systemd, /etc/default/grub, systemctl, unit config files, etc. I don?t think anybody would want that experience on a production box. It?s a dramatic leap. Oh yea.
Perhaps I can help out here and there? I use daemontools for some years now without any noticeable problems, really. Yes, the mindset is different from other process supervision systems, but then again, each of them has its downsides. I haven't heard too much good stuff about systemd; it allegedly does ?too much? (including being a DHCP client and gui interaction?). I'd really like to help to keep daemontools as it is.
+1000
Ken forced me to learn daemontools in order to set up squeaksource.com on box3, and I am glad that he did. It is the perfect tool for running Squeak based services on a system like this. Once I figured out how it worked, it has been no problem at all, and it eliminated the usual hackish scripts that I might otherwise have been tempted to invent. It is also a good way to ensure that our various services are managed in similar ways, so that people other than the original author can figure out how they are set up, and what to do when they stop working.
FWIW, I don't think I have had to touch anything related to squeaksource.com since at least last February, and even that was just an image update that I did through VNC.
Dave
IMHO daemontools does what it offers, it supervises processes. In case of the source.squeak.org the image was up and running, but it was non-functional. Daemontools is not a system monitoring tool; it won't check the health of the services, it'll simply ensure that the processes are running. On the new boxes we can consider using monitoring tools.
Levente
On Tue, 21 Oct 2014, Chris Cunnington wrote:
Just on the topic, I find daemontools confounding. The svc tool is supposed to start and stop processes, but I also find it doesn’t work. As I understand it, svc is supposed to control a supervise process, which overlooks a process like source.squeak.org. But how does that start exactly? I think there’s more than one way. I think creating a symlink to to the /service directory is supposed to do it. I’ve also used something like “supervise foo&”, which I figure is a hack. daemontools is great because it restarts processes automatically when rebooting. But the svc and svstat haven’t proven that useful in my experience. And daemontools is state-of-the-art 2004. I’m looking at systemd (which netstyle.ch uses to start and stop its hundreds of client images from requests sent to the box), which I think supersedes the daemontools feature set. And I understand, it will be PID1 default on some Linuxes in the near future. It is possible that sometime the use of daemontools on Squeak boxes might be reconsidered. Not my decision, of course, but as the topic came out, I thought I’d brain dump. I want to like daemontools more than I find I’m able to.
And on the topic of systemd, I’d say this, it’s a big leap. Back up everything and review what you know about GRUB2. I’ve found that out the hard way. I’m learning on a dedicated laptop and I lost my UI. Couldn’t get it back. Could only get a terminal on tty1. So I created an install stick and re-installed Ubuntu 14.04 from USB. Now I can try, fail, and have a back up plan, learn more about GRUB2, systemd, /etc/default/grub, systemctl, unit config files, etc. I don’t think anybody would want that experience on a production box. It’s a dramatic leap. Oh yea.
Chris
On Oct 21, 2014, at 7:27 AM, Levente Uzonyi <leves@elte.hu> wrote:
Hi All,
I tried to update my Squeak image today, and found that source.squeak.org is down. On box2 I found that it's using all the cpu. Based on the apachelogs it went down 7-8 hours ago. I decided to restart it. svc -t didn't kill the image (nor did -k), instead it spawned another image, so I stopped svc, killed the process manually, and restarted it with svc. The image is up now, but I wonder if there's anything else to do when a SqueakSource image goes down.
Levente
I'm glad this topic came up because I have "learn daemontools" coming up on my list. I'm a total newbie with it, but I think I want try using it with my own servers soon.
But one thing I noticed when I google for it wants to bring me to http://www.daemon-tools.cc/home, but the one Ken referred me to a while back was http://cr.yp.to/daemontools.html.
Are these two the same thing or is the latter what I want?
Thanks.
On Tue, Oct 21, 2014 at 10:57 AM, Levente Uzonyi leves@elte.hu wrote:
IMHO daemontools does what it offers, it supervises processes. In case of the source.squeak.org the image was up and running, but it was non-functional. Daemontools is not a system monitoring tool; it won't check the health of the services, it'll simply ensure that the processes are running. On the new boxes we can consider using monitoring tools.
Levente
On Tue, 21 Oct 2014, Chris Cunnington wrote:
Just on the topic, I find daemontools confounding. The svc tool is supposed to start and stop processes, but I also find it doesn’t work. As I understand it, svc is supposed to control a supervise process, which overlooks a process like source.squeak.org. But how does that start exactly? I think there’s more than one way. I think creating a symlink to to the /service directory is supposed to do it. I’ve also used something like “supervise foo&”, which I figure is a hack. daemontools is great because it restarts processes automatically when rebooting. But the svc and svstat haven’t proven that useful in my experience. And daemontools is state-of-the-art 2004. I’m looking at systemd (which netstyle.ch uses to start and stop its hundreds of client images from requests sent to the box), which I think supersedes the daemontools feature set. And I understand, it will be PID1 default on some Linuxes in the near future. It is possible that sometime the use of daemontools on Squeak boxes might be reconsidered. Not my decision, of course, but as the topic came out, I thought I’d brain dump. I want to like daemontools more than I find I’m able to.
And on the topic of systemd, I’d say this, it’s a big leap. Back up everything and review what you know about GRUB2. I’ve found that out the hard way. I’m learning on a dedicated laptop and I lost my UI. Couldn’t get it back. Could only get a terminal on tty1. So I created an install stick and re-installed Ubuntu 14.04 from USB. Now I can try, fail, and have a back up plan, learn more about GRUB2, systemd, /etc/default/grub, systemctl, unit config files, etc. I don’t think anybody would want that experience on a production box. It’s a dramatic leap. Oh yea.
Chris
On Oct 21, 2014, at 7:27 AM, Levente Uzonyi <leves@elte.hu> wrote:
Hi All,
I tried to update my Squeak image today, and found that source.squeak.org is down. On box2 I found that it's using all the cpu. Based on the apachelogs it went down 7-8 hours ago. I decided to restart it. svc -t didn't kill the image (nor did -k), instead it spawned another image, so I stopped svc, killed the process manually, and restarted it with svc. The image is up now, but I wonder if there's anything else to do when a SqueakSource image goes down.
Levente
On 21.10.2014, at 11:29, Chris Muller asqueaker@gmail.com wrote:
I'm glad this topic came up because I have "learn daemontools" coming up on my list. I'm a total newbie with it, but I think I want try using it with my own servers soon.
But one thing I noticed when I google for it wants to bring me to http://www.daemon-tools.cc/home, but the one Ken referred me to a while back was http://cr.yp.to/daemontools.html.
Are these two the same thing or is the latter what I want?
These things are entirely different. You want the second one for process supervision.
Best -Tobias
On Tue, Oct 21, 2014 at 01:27:30PM +0200, Levente Uzonyi wrote:
Hi All,
I tried to update my Squeak image today, and found that source.squeak.org is down. On box2 I found that it's using all the cpu. Based on the apachelogs it went down 7-8 hours ago. I decided to restart it. svc -t didn't kill the image (nor did -k), instead it spawned another image, so I stopped svc, killed the process manually, and restarted it with svc. The image is up now, but I wonder if there's anything else to do when a SqueakSource image goes down.
I don't know all the details on the source.squeak.org configuration, but based on my experience with squeaksource.com on box3, the only real concern is that recent updates to user accounts and other image resident information may get lost. The image should be saving itself regularly, and file uploads are resident on disk and will be recovered. Most likely nothing was lost at all, although a note to squeak-dev announcing the outage is a good idea.
Whenever we move source.squeak.org to box4 (hopefully soon, if Chris Muller and I manage to get ourselves motivated), I will try to get better educated on this before the next outage.
Thank for restarting it,
Dave
box-admins@lists.squeakfoundation.org