[Box-Admins] The story from the log files

[Levente Uzonyi]

On Thu, 25 Oct 2012, Chris Cunnington wrote:

> On 2012-10-25 12:50 PM, Levente Uzonyi wrote:
>> On Thu, 25 Oct 2012, Chris Cunnington wrote:
>> 
>>> OK, I made some changes. We now have a stanza that looks like this. [1] I 
>>> played with the <Proxy></Proxy> directive a bit. The result was to have 
>>> requests return as 404. With the RewriteRule they all return 403. [2] Our 
>>> log file is growing at about ~20M an hour. There are LogFormat directives 
>>> in apache2.conf, but no CustomLog directory. It has rolled over onto 
>>> another file once in the past (i.e. other_vhosts_access.log.1 from 
>>> other_vhosts_access.log), but I'm not sure from where.
>>> 
>>> I take it that with the 403 requests shown in the log [2] that the 
>>> pressure is off the Jenkins server but not off our logging apparatus. I 
>>> think it is clear that the apache2.conf file we received is shorter than 
>>> usual, shorter than the httpd.conf I'm used to in CentOS. And that with 
>>> the high amount of traffic we are experiencing, we seem to be in a 
>>> shipping lane.
>> 
>> I don't know how is it done on CentOS, but on Debian/Ubuntu the apache
>> configuration file is split up into several parts (separate 
>> files/directories). The apache.conf only has server specific settings and 
>> shouldn't include anything else. Each site has it's own config file (in 
>> /etc/apache2/sites-available/) and optionally log files (usually in 
>> /var/log/apache2/). The config should include the following lines for 
>> separate log files:
>>
>>     CustomLog ${APACHE_LOG_DIR}/jenkins-access.log combined
>>     ErrorLog ${APACHE_LOG_DIR}/jenkins-error.log
>> 
>> Log files are rotated via logrotate, once a day by default.
> Several files instead of one big file. OK. The /sites-available/default does 
> have a CustomLog directive.
>> 
>> I doubt logging is a bottleneck, but using a separate log file is useful. 
>> It would be good to check the error.log to see if apache is low on 
>> resources (or not). Also "top -d 1" can give you hints about what's eating 
>> up CPU/memory, or what's waiting for the disk for too long.
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
> 15277 lewis     20   0 1028m  89m 1300 R  2.0  8.9  18:15.60 squeakvm
>    1 root      20   0  2032  652  556 S  0.0  0.1   0:15.83 init
>
> lewis    15277  2.1  8.8 1053112 91572 pts/0   S    04:40  18:15 
> /usr/local/lib/squeak/4.10.5-2619/squeakvm -nodisplay 
> /home/lewis/VMUnixBuild/Squeak4.3.image 
> /home/lewis/VMUnixBuild/VMUnixBuild.st
>
> That is consistently at the top of top -d 1. This VPS has 1G of RAM.
>> 
>> It would also be good to reconfigure jenkins to listen on only the local 
>> interface (see the link in my previous mail) and add a firewall to the 
>> server. When I set up a server, I never leave ssh on port 22, but move it 
>> to a random port and drop all packets which are not intended to be received 
>> via iptables. This reduces the number of attack attempts to almost 0.
>> 
> These are good ideas and greater system administration ideas. But for today, 
> I think the greater issue of Jenkins has been addressed. (As in, I need a bit 
> of a break.)

I can do all this stuff if I get access to the box. ;)

>> Since I didn't find any easy to use firewall script, therefore I wrote my 
>> own init.d script for that. If there's interest in it, then I can make it 
>> available for download.
>> 
> Yes. I would like to see that, please.
>
> Chris
>

I uploaded the firewall script to http://leves.web.elte.hu/squeak/firewall 
. This basic version keeps port 22 and 80 open and drops all packets 
arriving to other ports. If you want something different, then you have to 
change the rules() function. When the file is copied to /etc/init.d , then

$ /etc/init.d/firewall start

will "start" the firewall. If you want to start it whenever the server is 
restarting, then just do

$ /etc/init.d/firewall install

I'd suggest commenting out the line

   iptables -A INPUT -j DROP

the first time, so you won't lose the connection to the server if anything 
goes wrong. :)


Levente