I just want to sum up some things that happened with the box and Jenkins this week:
- We have Jenkins running behind its domain squeakci.org. It comes up reliably now, but there is around a twenty second wait, which is too long. - The /var/log/apache2/other_vhosts_access.log is around 2.5G now. That means its growing at .3G a day. - We were being used as an open proxy and have done everything conceivable to stop it. The log now says that we still get lots of requests to do this and that they all get 403 responses. I'm unfamiliar with this as it didn't happen on my ServerBeach server (the one I'm about to shutter). I can't help feel it's a kind of low level DDOS and that even with 403 responses it likely affects the ability of people to reach our Jenkins server - I've launched a 4.4 image with Altitude and RFB in the server. Using RFB I can see that the problem of Altitude not finding libsys is there. All the Linux FFI tests run green so that's not the problem. I figure I'll create a symlink between where FFI looks for things and /usr/lib/libsys*
Some things we want to do next:
- Dave has a vm build script that returns a tarball of VMM-generated sources. We want to make that a daily build on the Jenkins server and produced a downloadable artifact - Levente has suggested making Jenkins listen only on the local interface, and to - add a firewall to the server while moving port22 requests to another port.
An observation:
- Between the Jenkins server, compiling vms, and deploying an Altitude website, I think we are starting to crowd this server which only has 1G of RAM.
Chris
On 26 October 2012 17:10, Chris Cunnington smalltalktelevision@gmail.com wrote:
I just want to sum up some things that happened with the box and Jenkins this week:
- We have Jenkins running behind its domain squeakci.org. It comes up
reliably now, but there is around a twenty second wait, which is too long.
- The /var/log/apache2/other_vhosts_access.log is around 2.5G now. That
means its growing at .3G a day.
- We were being used as an open proxy and have done everything conceivable
to stop it. The log now says that we still get lots of requests to do this and that they all get 403 responses. I'm unfamiliar with this as it didn't happen on my ServerBeach server (the one I'm about to shutter). I can't help feel it's a kind of low level DDOS and that even with 403 responses it likely affects the ability of people to reach our Jenkins server
- I've launched a 4.4 image with Altitude and RFB in the server. Using RFB I
can see that the problem of Altitude not finding libsys is there. All the Linux FFI tests run green so that's not the problem. I figure I'll create a symlink between where FFI looks for things and /usr/lib/libsys*
Some things we want to do next:
- Dave has a vm build script that returns a tarball of VMM-generated
sources. We want to make that a daily build on the Jenkins server and produced a downloadable artifact
Wouldn't it be better to trigger on commits to svn and/or vm-dev notifications? (The former's just a plugin, the latter's just reusing Levente's polling script.)
frank
- Levente has suggested making Jenkins listen only on the local interface,
and to
- add a firewall to the server while moving port22 requests to another port.
An observation:
- Between the Jenkins server, compiling vms, and deploying an Altitude
website, I think we are starting to crowd this server which only has 1G of RAM.
The solution here is to use other machines as the build agents. I'm trying to find some kind of offline system where Jenkins can say "there's a build due for this project" and other machines can watch for this without, say, being reachable from Jenkins (being always-on, in other words). This would let volunteers donate their machines without exposing same. But this doesn't solve the issue at hand.
frank
Chris
On Fri, Oct 26, 2012 at 05:32:08PM +0100, Frank Shearar wrote:
- Dave has a vm build script that returns a tarball of VMM-generated
sources. We want to make that a daily build on the Jenkins server and produced a downloadable artifact
Wouldn't it be better to trigger on commits to svn and/or vm-dev notifications? (The former's just a plugin, the latter's just reusing Levente's polling script.)
The VM actually includes various plugins from repositories outside of the VMMaker repositories. Evaluating "VMMaker updateFromServer" attempts to load the latest of everything, and that is part of the build script. I'm not sure how reliable it is, but at least that's what it's attempting to do.
- Levente has suggested making Jenkins listen only on the local interface,
and to
- add a firewall to the server while moving port22 requests to another port.
An observation:
- Between the Jenkins server, compiling vms, and deploying an Altitude
website, I think we are starting to crowd this server which only has 1G of RAM.
The solution here is to use other machines as the build agents. I'm trying to find some kind of offline system where Jenkins can say "there's a build due for this project" and other machines can watch for this without, say, being reachable from Jenkins (being always-on, in other words). This would let volunteers donate their machines without exposing same. But this doesn't solve the issue at hand.
A much better place to do VM builds is on squeakvm.org, and Ian Piumarta has indicated to me that he would prefer to have things done there anyway. So if we need to move the processing off of box3, I'd be inclined to just see if I can get Ian to give me a hand setting it up on the squeakvm box (which after all he has managed for the benefit of the community for many years now).
Right now I'm just working on getting the VM tarball build to be reliable and repeatable, and I don't have any experience with Jenkins at all. But maybe once we get something up and running we can take a look at moving the processing off to another box.
Dave
box-admins@lists.squeakfoundation.org
-
Chris Cunnington -
David T. Lewis -
Frank Shearar