On Thu, 25 Oct 2012, Chris Cunnington wrote:
On 2012-10-25 12:50 PM, Levente Uzonyi wrote:
On Thu, 25 Oct 2012, Chris Cunnington wrote:
OK, I made some changes. We now have a stanza that looks like this. [1] I played with the <Proxy></Proxy> directive a bit. The result was to have requests return as 404. With the RewriteRule they all return 403. [2] Our log file is growing at about ~20M an hour. There are LogFormat directives in apache2.conf, but no CustomLog directory. It has rolled over onto another file once in the past (i.e. other_vhosts_access.log.1 from other_vhosts_access.log), but I'm not sure from where.
I take it that with the 403 requests shown in the log [2] that the pressure is off the Jenkins server but not off our logging apparatus. I think it is clear that the apache2.conf file we received is shorter than usual, shorter than the httpd.conf I'm used to in CentOS. And that with the high amount of traffic we are experiencing, we seem to be in a shipping lane.
I don't know how is it done on CentOS, but on Debian/Ubuntu the apache configuration file is split up into several parts (separate files/directories). The apache.conf only has server specific settings and shouldn't include anything else. Each site has it's own config file (in /etc/apache2/sites-available/) and optionally log files (usually in /var/log/apache2/). The config should include the following lines for separate log files:
CustomLog ${APACHE_LOG_DIR}/jenkins-access.log combined ErrorLog ${APACHE_LOG_DIR}/jenkins-error.log
Log files are rotated via logrotate, once a day by default.
Several files instead of one big file. OK. The /sites-available/default does have a CustomLog directive.
I doubt logging is a bottleneck, but using a separate log file is useful. It would be good to check the error.log to see if apache is low on resources (or not). Also "top -d 1" can give you hints about what's eating up CPU/memory, or what's waiting for the disk for too long.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15277 lewis 20 0 1028m 89m 1300 R 2.0 8.9 18:15.60 squeakvm 1 root 20 0 2032 652 556 S 0.0 0.1 0:15.83 init
lewis 15277 2.1 8.8 1053112 91572 pts/0 S 04:40 18:15 /usr/local/lib/squeak/4.10.5-2619/squeakvm -nodisplay /home/lewis/VMUnixBuild/Squeak4.3.image /home/lewis/VMUnixBuild/VMUnixBuild.st
That is consistently at the top of top -d 1. This VPS has 1G of RAM.
It would also be good to reconfigure jenkins to listen on only the local interface (see the link in my previous mail) and add a firewall to the server. When I set up a server, I never leave ssh on port 22, but move it to a random port and drop all packets which are not intended to be received via iptables. This reduces the number of attack attempts to almost 0.
These are good ideas and greater system administration ideas. But for today, I think the greater issue of Jenkins has been addressed. (As in, I need a bit of a break.)
I can do all this stuff if I get access to the box. ;)
Since I didn't find any easy to use firewall script, therefore I wrote my own init.d script for that. If there's interest in it, then I can make it available for download.
Yes. I would like to see that, please.
Chris
I uploaded the firewall script to http://leves.web.elte.hu/squeak/firewall . This basic version keeps port 22 and 80 open and drops all packets arriving to other ports. If you want something different, then you have to change the rules() function. When the file is copied to /etc/init.d , then
$ /etc/init.d/firewall start
will "start" the firewall. If you want to start it whenever the server is restarting, then just do
$ /etc/init.d/firewall install
I'd suggest commenting out the line
iptables -A INPUT -j DROP
the first time, so you won't lose the connection to the server if anything goes wrong. :)
Levente