[Box-Admins] The story from the log files

Thu Oct 25 16:50:21 UTC 2012

On Thu, 25 Oct 2012, Chris Cunnington wrote:

> OK, I made some changes. We now have a stanza that looks like this. [1] I 
> played with the <Proxy></Proxy> directive a bit. The result was to have 
> requests return as 404. With the RewriteRule they all return 403. [2] Our log 
> file is growing at about ~20M an hour. There are LogFormat directives in 
> apache2.conf, but no CustomLog directory. It has rolled over onto another 
> file once in the past (i.e. other_vhosts_access.log.1 from 
> other_vhosts_access.log), but I'm not sure from where.
>
> I take it that with the 403 requests shown in the log [2] that the pressure 
> is off the Jenkins server but not off our logging apparatus. I think it is 
> clear that the apache2.conf file we received is shorter than usual, shorter 
> than the httpd.conf I'm used to in CentOS. And that with the high amount of 
> traffic we are experiencing, we seem to be in a shipping lane.

I don't know how is it done on CentOS, but on Debian/Ubuntu the apache
configuration file is split up into several parts (separate 
files/directories). The apache.conf only has server specific settings and 
shouldn't include anything else. Each site has it's own config file (in 
/etc/apache2/sites-available/) and optionally log files (usually in 
/var/log/apache2/). The config should include the following lines for 
separate log files:

 	CustomLog ${APACHE_LOG_DIR}/jenkins-access.log combined
 	ErrorLog ${APACHE_LOG_DIR}/jenkins-error.log

Log files are rotated via logrotate, once a day by default.

I doubt logging is a bottleneck, but using a separate log file is useful. 
It would be good to check the error.log to see if apache is low on 
resources (or not). Also "top -d 1" can give you hints about what's eating 
up CPU/memory, or what's waiting for the disk for too long.

It would also be good to reconfigure jenkins to listen on only the local 
interface (see the link in my previous mail) and add a firewall to the 
server. When I set up a server, I never leave ssh on port 22, but move it 
to a random port and drop all packets which are not intended to be 
received via iptables. This reduces the number of attack attempts to 
almost 0.

Since I didn't find any easy to use firewall script, therefore I wrote my 
own init.d script for that. If there's interest in it, then I can make it 
available for download.

Levente

>
> Chris
>
> [1]
>
> <VirtualHost *:80>
>        ServerName www.squeakci.org
>        ServerAlias squeakci.org
>        ProxyRequests Off
>        ProxyPreserveHost On
>        ProxyPass / http://127.0.0.1:8080/
>        ProxyPassReverse / http://127.0.0.1:8080/
>        RewriteEngine On
>        RewriteCond %{THE_REQUEST} ^GET\ http(s?)://
>        RewriteRule .* - [F]
> </VirtualHost>
>
>
> [2]
>
> www.squeakci.org:80 142.91.217.213 - - [25/Oct/2012:18:06:29 +0200] "GET 
> http://ad.globe7.com/st?ad_type=pop&ad_size=0x0&section=3512133&banned_pop_types=29&pop_times=1&pop_frequency=0&pop_nofreqcap=1&pub_url=${PUB_URL} 
> HTTP/1.0" 403 524 
> "http://moonhealthylive.com/index.php?view=article&catid=34%3Abeauty-and-style&id=415%3A2011-07-16-12-14-20&format=pdf&option=com_content&Itemid=63" 
> "Mozilla/5.0 (X11; U; Linux i586; de; rv:5.0) Gecko/20100101 Firefox/5.0"
> www.squeakci.org:80 108.177.168.108 - - [25/Oct/2012:18:06:29 +0200] "GET 
> http://ad.tagjunction.com/st?ad_type=iframe&ad_size=160x600&section=3146202&pub_url=${PUB_URL} 
> HTTP/1.0" 403 529 
> "http://www.entertainmentangle.com/index.php?option=com_content&view=frontpage&Itemid=90" 
> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/535.1 (KHTML, 
> like Gecko) Chrome/14.0.790.0 Safari/535.1"
> www.squeakci.org:80 108.62.75.104 - - [25/Oct/2012:18:06:29 +0200] "GET 
> http://ad.adserverplus.com/st?ad_type=iframe&ad_size=728x90&section=2903043 
> HTTP/1.0" 403 530 
> "http://fashionlifestreet.com/index.php?view=article&catid=44%3Awholesale-fashion-dresses&id=28252%3A2011-12-18-22-26-35&format=pdf&option=com_content&Itemid=100" 
> "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.1.4322)"
> www.squeakci.org:80 142.91.217.167 - - [25/Oct/2012:18:06:29 +0200] "GET 
> http://ad.globe7.com/st?ad_type=iframe&ad_size=728x90&section=3011420&pub_url=${PUB_URL} 
> HTTP/1.0" 403 524 
> "http://www.knowledgelighthouse.com/index.php?view=article&catid=42%3Aeducational-games&id=9752%3A2011-09-30-14-40-35&tmpl=component&print=1&layout=default&page=&option=com_content&Itemid=98" 
> "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, 
> like Gecko) Version/3.1 Safari/525.13"
> www.squeakci.org:80 108.62.185.146 - - [25/Oct/2012:18:06:29 +0200] "GET 
> http://ad.globe7.com/st?ad_type=iframe&ad_size=300x250&section=3667021&pub_url=${PUB_URL} 
> HTTP/1.0" 403 524 
> "http://likecatpink.com/index.php?view=article&catid=43%3Afashion-jewellery&id=10097%3A2012-01-07-14-12-10&format=pdf&option=com_content&Itemid=99" 
> "Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 
> Opera 10.70"
> www.squeakci.org:80 23.19.195.254 - - [25/Oct/2012:18:06:29 +0200] "GET 
> http://ads.creafi-online-media.com/st?ad_type=ad&ad_size=300x250&section=3699322&pub_url=${PUB_URL} 
> HTTP/1.0" 403 538 
> "http://www.webgamesclub.com/index.php/play-games-online/1348-play-arcade-gamesonline-play-classic-arcade-games-online" 
> "Mozilla/4.76 [en] (X11; U; HP-UX B.10.20 9000/782)"
> www.squeakci.org:80 50.93.207.108 - - [25/Oct/2012:18:06:29 +0200] "GET 
> http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=728x90&section=2666175 
> HTTP/1.0" 403 530 "http://www.newfindcar.com/2011/01/13/audi-tt-gt4-concept/" 
> "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; YPC 3.2.0; SLCC1; .NET 
> CLR 2.0.50727; .NET CLR 3.0.04506)"
> www.squeakci.org:80 142.91.189.220 - - [25/Oct/2012:18:06:29 +0200] "GET 
> http://ad.globaltakeoff.net/st?ad_type=iframe&ad_size=728x90&section=2077929&pub_url=${PUB_URL} 
> HTTP/1.0" 403 531 
> "http://www.qtsfinancial.com/index.php?option=com_mailto&tmpl=component&link=aHR0cDovL3d3dy5xdHNmaW5hbmNpYWwuY29tL2luZGV4LnBocD9vcHRpb249Y29tX2NvbnRlbnQmdmlldz1hcnRpY2xlJmlkPTMyOTE6MjAxMS0wNy0wNi0xMy0yNS0xNyZjYXRpZD00MDpmaW5hbmNpYWwtaW5mbyZJdGVtaWQ9OTY=" 
> "Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.9.1) Gecko/20090702 
> Firefox/3.5"
> www.squeakci.org:80 23.19.67.42 - - [25/Oct/2012:18:06:29 +0200] "GET 
> http://ad.globe7.com/st?ad_type=iframe&ad_size=160x600&section=3011410&pub_url=${PUB_URL} 
> HTTP/1.0" 403 524 
> "http://www.femaleapple.com/index.php?option=com_mailto&tmpl=component&link=aHR0cDovL3d3dy5mZW1hbGVhcHBsZS5jb20vaW5kZXgucGhwP29wdGlvbj1jb21fY29udGVudCZ2aWV3PWFydGljbGUmaWQ9Njc0MToyMDEyLTAxLTE1LTAyLTI0LTM4JmNhdGlkPTQzOndvbWVucy1oZWFsdGgtc3Vic2NyaXB0aW9uJkl0ZW1pZD05OQ==" 
> "Opera/10.50 (Windows NT 6.1; U; en-GB) Presto/2.2.2"
> www.squeakci.org:80 108.62.178.116 - - [25/Oct/2012:18:06:30 +0200] "GET 
> http://ad.adserverplus.com/st?ad_type=pop&ad_size=0x0&section=3256403&banned_pop_types=29&pop_times=1&pop_frequency=0&pub_url=${PUB_URL} 
> HTTP/1.0" 403 530 
> "http://www.loseweightwomen.com/index.php?view=article&catid=34%3Ahealth-advice&id=791%3Avaricose-veins-in-vaginal-area-any-advice&format=pdf&option=com_content&Itemid=53" 
> "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar)"
>