[Box-Admins] The story from the log files

[Chris Cunnington]

On 2012-10-25 12:50 PM, Levente Uzonyi wrote:
> On Thu, 25 Oct 2012, Chris Cunnington wrote:
>
>> OK, I made some changes. We now have a stanza that looks like this. 
>> [1] I played with the <Proxy></Proxy> directive a bit. The result was 
>> to have requests return as 404. With the RewriteRule they all return 
>> 403. [2] Our log file is growing at about ~20M an hour. There are 
>> LogFormat directives in apache2.conf, but no CustomLog directory. It 
>> has rolled over onto another file once in the past (i.e. 
>> other_vhosts_access.log.1 from other_vhosts_access.log), but I'm not 
>> sure from where.
>>
>> I take it that with the 403 requests shown in the log [2] that the 
>> pressure is off the Jenkins server but not off our logging apparatus. 
>> I think it is clear that the apache2.conf file we received is shorter 
>> than usual, shorter than the httpd.conf I'm used to in CentOS. And 
>> that with the high amount of traffic we are experiencing, we seem to 
>> be in a shipping lane.
>
> I don't know how is it done on CentOS, but on Debian/Ubuntu the apache
> configuration file is split up into several parts (separate 
> files/directories). The apache.conf only has server specific settings 
> and shouldn't include anything else. Each site has it's own config 
> file (in /etc/apache2/sites-available/) and optionally log files 
> (usually in /var/log/apache2/). The config should include the 
> following lines for separate log files:
>
>     CustomLog ${APACHE_LOG_DIR}/jenkins-access.log combined
>     ErrorLog ${APACHE_LOG_DIR}/jenkins-error.log
>
> Log files are rotated via logrotate, once a day by default.
Several files instead of one big file. OK. The /sites-available/default 
does have a CustomLog directive.
>
> I doubt logging is a bottleneck, but using a separate log file is 
> useful. It would be good to check the error.log to see if apache is 
> low on resources (or not). Also "top -d 1" can give you hints about 
> what's eating up CPU/memory, or what's waiting for the disk for too long.

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
15277 lewis     20   0 1028m  89m 1300 R  2.0  8.9  18:15.60 squeakvm
     1 root      20   0  2032  652  556 S  0.0  0.1   0:15.83 init

lewis    15277  2.1  8.8 1053112 91572 pts/0   S    04:40  18:15 
/usr/local/lib/squeak/4.10.5-2619/squeakvm -nodisplay 
/home/lewis/VMUnixBuild/Squeak4.3.image 
/home/lewis/VMUnixBuild/VMUnixBuild.st

That is consistently at the top of top -d 1. This VPS has 1G of RAM.
>
> It would also be good to reconfigure jenkins to listen on only the 
> local interface (see the link in my previous mail) and add a firewall 
> to the server. When I set up a server, I never leave ssh on port 22, 
> but move it to a random port and drop all packets which are not 
> intended to be received via iptables. This reduces the number of 
> attack attempts to almost 0.
>
These are good ideas and greater system administration ideas. But for 
today, I think the greater issue of Jenkins has been addressed. (As in, 
I need a bit of a break.)
> Since I didn't find any easy to use firewall script, therefore I wrote 
> my own init.d script for that. If there's interest in it, then I can 
> make it available for download.
>
Yes. I would like to see that, please.

Chris