Levente was right about the open proxy exploitation. [1] It has stopped now. [Editor's Note: No it hasn't.] The last one was at 7:52 on 23 Oct. This server is on CEST time, so subtracting six hours that would be 1:52 here in eastern North America. The GET requests display explotation when they are asking for a server that is not ours. The request for http://ad.yieldmanager.com is an example. I don't suppose there's any real damage, but it is my mistake.
The open proxy exploitation was followed by many POST requests. [2] Notice the size of this log file:
-rw-r----- 1 root adm 2173022665 Oct 25 14:20 other_vhosts_access.log
What is that? To my eyes that's 2.02 Gigs of data collected over maybe ~72 hours. Many [2] are POST requests. I can't say what ajaxExecutors or ajaxBuildQueue is. It is definitely part of Jenkins, I'm just not sure what part. I'll look into it.
Actually, I'm wrong. [3]. We're still being exploited as an open proxy. Those are the latest results from the log file.
I've changed the stanza to and restarted:
<VirtualHost *:80> ServerName www.squeakci.org ServerAlias squeakci.org ProxyRequests Off ProxyPreserveHost On ProxyPass / http://127.0.0.1:8080/ ProxyPassReverse / http://127.0.0.1:8080/ <Proxy *> Order deny,allow Allow from all </Proxy> </VirtualHost>
And will check the log file again in two hours.
Chris
[1]
92.17.231.188 - - [23/Oct/2012:07:52:54 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:07:52:55 +0200] "POST /ajaxBuildQueue HTTP/1.1" 200 415 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:55 +0200] "GET http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=30... HTTP/1.0" 200 4982 "http://www.file4dvd.com" "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:56 +0200] "GET http://ad.yieldmanager.com/imp?Z=300x250&s=3007994&T=3&_salt=191... HTTP/1.0" 302 712 "http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=30..." "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:57 +0200] "GET http://cookex.amp.yahoo.com/v2/cexposer/SIG=13rmsj29b/*http%3A//ad.yieldmana... HTTP/1.0" 302 751 "http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=30..." "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:07:52:59 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:59 +0200] "GET http://ad.yieldmanager.com/imp?Z=300x250&s=3007994&T=3&_salt=191... HTTP/1.0" 200 1806 "http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=30..." "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)"
[2]
92.17.231.188 - - [23/Oct/2012:04:41:01 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:04:41:03 +0200] "POST /ajaxBuildQueue HTTP/1.1" 200 415 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:04:41:06 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14"
[3]
108.62.111.169 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.scanmedios.com/st?ad_type=iframe&ad_size=160x600%C2%A7ion=3522... HTTP/1.0" 404 558 "http://classidressing.com/index.php?option=com_mailto&tmpl=component&..." "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Win64; x64; .NET CLR 2.0.50727; SLCC1; Media Center PC 5.0; .NET CLR 3.0.04506)" www.squeakci.org:80 50.93.195.16 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=19... HTTP/1.0" 404 558 "http://www.suddengame.com/index.html" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; SLCC1; Media Center PC 5.0; .NET CLR 2.0.50727)" www.squeakci.org:80 23.19.67.38 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.adserverplus.com/st?ad_type=iframe&ad_size=728x90%C2%A7ion=289... HTTP/1.0" 404 558 "http://femaleapple.com/index.php?option=com_content&view=article&id=..." "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5" www.squeakci.org:80 108.62.178.236 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.tagjunction.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=293... HTTP/1.0" 404 558 "http://bestmylive.com/index.php?option=com_mailto&tmpl=component&lin..." "Mozilla/4.0 (compatible; MSIE 6.0; Windows CE; IEMobile 7.11)" www.squeakci.org:80 108.62.75.188 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.globe7.com/st?ad_type=iframe&ad_size=160x600%C2%A7ion=3542181&... HTTP/1.0" 404 558 "http://fashionarrow.com/index.php?option=com_mailto&tmpl=component&l..." "Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.53" www.squeakci.org:80 173.208.94.17 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.scanmedios.com/st?ad_type=iframe&ad_size=160x600%C2%A7ion=3522... HTTP/1.0" 404 558 "http://classidressing.com/index.php?option=com_mailto&tmpl=component&..." "Mozilla/4.0 (compatible; MSIE 6.0; Update a; Win32)" www.squeakci.org:80 142.91.189.9 - - [25/Oct/2012:14:30:30 +0200] "GET http://ads1.ministerial5.com/creative/2-002134057-00001i;size=4 HTTP/1.0" 404 558 "http://travellingonroad.com/index.php?view=article&catid=34%3Acheap-trav..." "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.35 (KHTML, like Gecko) Ubuntu/10.10 Chromium/13.0.764.0 Chrome/13.0.764.0 Safari/534.35" www.squeakci.org:80 142.91.217.190 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.globaltakeoff.net/st?ad_type=iframe&ad_size=300x250%C2%A7ion=2... HTTP/1.0" 404 558 "http://www.ttfemalehealth.com/index.php?option=com_content&view=article&..." "Opera/9.80 (Windows NT 6.0; U; en) Presto/2.8.99 Version/11.10" www.squeakci.org:80 142.91.189.47 - - [25/Oct/2012:14:30:31 +0200] "GET http://ad.adserverplus.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=32... HTTP/1.0" 404 558 "http://newsja.com/index.php?view=article&catid=35%3Acelebrity&id=845..." "Mozilla/4.76 [en] (X11; U; SunOS 5.7 sun4u)"
The ProxyRequests Off line stops apache working as a forward proxy. The <proxy> block is only necessary to allow proxying if other parts of the apache config deny it (default on most linuxes). More details here: https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache
Currently the server returns a 200 response for all non-local request, but it serves the jenkins page instead of what was requested. In order to get rid of this extra load we should reject all non-local requests. It can be done with RewriteEngine:
execute: sudo a2enmod rewrite
add the following to the configuration:
RewriteEngine On RewriteCond %{THE_REQUEST} ^GET\ http(s?):// RewriteRule .* - [F]
Then restart apache.
Levente
On Thu, 25 Oct 2012, Chris Cunnington wrote:
Levente was right about the open proxy exploitation. [1] It has stopped now. [Editor's Note: No it hasn't.] The last one was at 7:52 on 23 Oct. This server is on CEST time, so subtracting six hours that would be 1:52 here in eastern North America. The GET requests display explotation when they are asking for a server that is not ours. The request for http://ad.yieldmanager.com is an example. I don't suppose there's any real damage, but it is my mistake.
The open proxy exploitation was followed by many POST requests. [2] Notice the size of this log file:
-rw-r----- 1 root adm 2173022665 Oct 25 14:20 other_vhosts_access.log
What is that? To my eyes that's 2.02 Gigs of data collected over maybe ~72 hours. Many [2] are POST requests. I can't say what ajaxExecutors or ajaxBuildQueue is. It is definitely part of Jenkins, I'm just not sure what part. I'll look into it.
Actually, I'm wrong. [3]. We're still being exploited as an open proxy. Those are the latest results from the log file.
I've changed the stanza to and restarted:
<VirtualHost *:80> ServerName www.squeakci.org ServerAlias squeakci.org ProxyRequests Off ProxyPreserveHost On ProxyPass / http://127.0.0.1:8080/ ProxyPassReverse / http://127.0.0.1:8080/ <Proxy *> Order deny,allow Allow from all
</Proxy> </VirtualHost>
And will check the log file again in two hours.
Chris
[1]
92.17.231.188 - - [23/Oct/2012:07:52:54 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:07:52:55 +0200] "POST /ajaxBuildQueue HTTP/1.1" 200 415 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:55 +0200] "GET http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=30... HTTP/1.0" 200 4982 "http://www.file4dvd.com" "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:56 +0200] "GET http://ad.yieldmanager.com/imp?Z=300x250&s=3007994&T=3&_salt=191... HTTP/1.0" 302 712 "http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=30..." "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:57 +0200] "GET http://cookex.amp.yahoo.com/v2/cexposer/SIG=13rmsj29b/*http%3A//ad.yieldmana... HTTP/1.0" 302 751 "http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=30..." "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:07:52:59 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:59 +0200] "GET http://ad.yieldmanager.com/imp?Z=300x250&s=3007994&T=3&_salt=191... HTTP/1.0" 200 1806 "http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=30..." "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)"
[2]
92.17.231.188 - - [23/Oct/2012:04:41:01 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:04:41:03 +0200] "POST /ajaxBuildQueue HTTP/1.1" 200 415 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:04:41:06 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14"
[3]
108.62.111.169 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.scanmedios.com/st?ad_type=iframe&ad_size=160x600%C2%A7ion=3522... HTTP/1.0" 404 558 "http://classidressing.com/index.php?option=com_mailto&tmpl=component&..." "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Win64; x64; .NET CLR 2.0.50727; SLCC1; Media Center PC 5.0; .NET CLR 3.0.04506)" www.squeakci.org:80 50.93.195.16 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=19... HTTP/1.0" 404 558 "http://www.suddengame.com/index.html" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; SLCC1; Media Center PC 5.0; .NET CLR 2.0.50727)" www.squeakci.org:80 23.19.67.38 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.adserverplus.com/st?ad_type=iframe&ad_size=728x90%C2%A7ion=289... HTTP/1.0" 404 558 "http://femaleapple.com/index.php?option=com_content&view=article&id=..." "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5" www.squeakci.org:80 108.62.178.236 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.tagjunction.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=293... HTTP/1.0" 404 558 "http://bestmylive.com/index.php?option=com_mailto&tmpl=component&lin..." "Mozilla/4.0 (compatible; MSIE 6.0; Windows CE; IEMobile 7.11)" www.squeakci.org:80 108.62.75.188 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.globe7.com/st?ad_type=iframe&ad_size=160x600%C2%A7ion=3542181&... HTTP/1.0" 404 558 "http://fashionarrow.com/index.php?option=com_mailto&tmpl=component&l..." "Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.53" www.squeakci.org:80 173.208.94.17 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.scanmedios.com/st?ad_type=iframe&ad_size=160x600%C2%A7ion=3522... HTTP/1.0" 404 558 "http://classidressing.com/index.php?option=com_mailto&tmpl=component&..." "Mozilla/4.0 (compatible; MSIE 6.0; Update a; Win32)" www.squeakci.org:80 142.91.189.9 - - [25/Oct/2012:14:30:30 +0200] "GET http://ads1.ministerial5.com/creative/2-002134057-00001i;size=4 HTTP/1.0" 404 558 "http://travellingonroad.com/index.php?view=article&catid=34%3Acheap-trav..." "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.35 (KHTML, like Gecko) Ubuntu/10.10 Chromium/13.0.764.0 Chrome/13.0.764.0 Safari/534.35" www.squeakci.org:80 142.91.217.190 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.globaltakeoff.net/st?ad_type=iframe&ad_size=300x250%C2%A7ion=2... HTTP/1.0" 404 558 "http://www.ttfemalehealth.com/index.php?option=com_content&view=article&..." "Opera/9.80 (Windows NT 6.0; U; en) Presto/2.8.99 Version/11.10" www.squeakci.org:80 142.91.189.47 - - [25/Oct/2012:14:30:31 +0200] "GET http://ad.adserverplus.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=32... HTTP/1.0" 404 558 "http://newsja.com/index.php?view=article&catid=35%3Acelebrity&id=845..." "Mozilla/4.76 [en] (X11; U; SunOS 5.7 sun4u)"
OK. You want to identify all non-local requests and nullify them. I am away from a terminal at the moment, so I'll be able to do it in an hour.
Chris
On Thu, Oct 25, 2012 at 9:19 AM, Levente Uzonyi leves@elte.hu wrote:
The ProxyRequests Off line stops apache working as a forward proxy. The <proxy> block is only necessary to allow proxying if other parts of the apache config deny it (default on most linuxes). More details here: https://wiki.jenkins-ci.org/**display/JENKINS/Running+** Jenkins+behind+Apachehttps://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache
Currently the server returns a 200 response for all non-local request, but it serves the jenkins page instead of what was requested. In order to get rid of this extra load we should reject all non-local requests. It can be done with RewriteEngine:
execute: sudo a2enmod rewrite
add the following to the configuration:
RewriteEngine On RewriteCond %{THE_REQUEST} ^GET\ http(s?):// RewriteRule .* - [F]Then restart apache.
Levente
On Thu, 25 Oct 2012, Chris Cunnington wrote:
Levente was right about the open proxy exploitation. [1] It has stopped now. [Editor's Note: No it hasn't.] The last one was at 7:52 on 23 Oct. This server is on CEST time, so subtracting six hours that would be 1:52 here in eastern North America. The GET requests display explotation when they are asking for a server that is not ours. The request for http://ad.yieldmanager.com is an example. I don't suppose there's any real damage, but it is my mistake.
The open proxy exploitation was followed by many POST requests. [2] Notice the size of this log file:
-rw-r----- 1 root adm 2173022665 Oct 25 14:20 other_vhosts_access.log
What is that? To my eyes that's 2.02 Gigs of data collected over maybe ~72 hours. Many [2] are POST requests. I can't say what ajaxExecutors or ajaxBuildQueue is. It is definitely part of Jenkins, I'm just not sure what part. I'll look into it.
Actually, I'm wrong. [3]. We're still being exploited as an open proxy. Those are the latest results from the log file.
I've changed the stanza to and restarted:
<VirtualHost *:80> ServerName www.squeakci.org ServerAlias squeakci.org ProxyRequests Off ProxyPreserveHost On ProxyPass / http://127.0.0.1:8080/ ProxyPassReverse / http://127.0.0.1:8080/ <Proxy *> Order deny,allow Allow from all
</Proxy> </VirtualHost>
And will check the log file again in two hours.
Chris
[1]
92.17.231.188 - - [23/Oct/2012:07:52:54 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:07:52:55 +0200] "POST /ajaxBuildQueue HTTP/1.1" 200 415 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:55 +0200] "GET http://ad.yieldmanager.com/st?**ad_type=iframe&ad_size=** 300x250§ion=3007994&pub_**url=${PUB_URL}http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250§ion=3007994&pub_url=$%7BPUB_URL%7DHTTP/1.0" 200 4982 " http://www.file4dvd.com" "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:56 +0200] "GET http://ad.yieldmanager.com/**imp?Z=300x250&s=3007994&T=3&_** salt=1911752854&B=12&m=2&u=**http%3A%2F%2Fwww.file4dvd.com%**2F&r=1http://ad.yieldmanager.com/imp?Z=300x250&s=3007994&T=3&_salt=1911752854&B=12&m=2&u=http%3A%2F%2Fwww.file4dvd.com%2F&r=1HTTP/1.0" 302 712 " http://ad.yieldmanager.com/**st?ad_type=iframe&ad_size=** 300x250§ion=3007994&pub_**url=${PUB_URL}http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250§ion=3007994&pub_url=$%7BPUB_URL%7D" "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:57 +0200] "GET http://cookex.amp.yahoo.com/**v2/cexposer/SIG=13rmsj29b/*** http%3A//ad.yieldmanager.com/**imp?Z=300x250&s=3007994&T=3&_** salt=1911752854&B=12&m=2&u=**http%3A%2F%2Fwww.file4dvd.com%**2F&r=1http://cookex.amp.yahoo.com/v2/cexposer/SIG=13rmsj29b/*http%3A//ad.yieldmanager.com/imp?Z=300x250&s=3007994&T=3&_salt=1911752854&B=12&m=2&u=http%3A%2F%2Fwww.file4dvd.com%2F&r=1HTTP/1.0" 302 751 " http://ad.yieldmanager.com/**st?ad_type=iframe&ad_size=** 300x250§ion=3007994&pub_**url=${PUB_URL}http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250§ion=3007994&pub_url=$%7BPUB_URL%7D" "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:07:52:59 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 184.22.82.217 - - [23/Oct/2012:07:52:59 +0200] "GET http://ad.yieldmanager.com/**imp?Z=300x250&s=3007994&T=3&_** salt=1911752854&B=12&m=2&u=**http%3A%2F%2Fwww.file4dvd.com%** 2F&r=1&SIG=10vqkkp1b;x-cookie=**2awvieq88pp7t&o=3&f=hnhttp://ad.yieldmanager.com/imp?Z=300x250&s=3007994&T=3&_salt=1911752854&B=12&m=2&u=http%3A%2F%2Fwww.file4dvd.com%2F&r=1&SIG=10vqkkp1b;x-cookie=2awvieq88pp7t&o=3&f=hnHTTP/1.0" 200 1806 " http://ad.yieldmanager.com/**st?ad_type=iframe&ad_size=** 300x250§ion=3007994&pub_**url=${PUB_URL}http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250§ion=3007994&pub_url=$%7BPUB_URL%7D" "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)"
[2]
92.17.231.188 - - [23/Oct/2012:04:41:01 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:04:41:03 +0200] "POST /ajaxBuildQueue HTTP/1.1" 200 415 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14" www.squeakci.org:80 92.17.231.188 - - [23/Oct/2012:04:41:06 +0200] "POST /ajaxExecutors HTTP/1.1" 200 545 "http://squeakci.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14"
[3]
108.62.111.169 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.scanmedios.com/st?**ad_type=iframe&ad_size=** 160x600§ion=3522623http://ad.scanmedios.com/st?ad_type=iframe&ad_size=160x600§ion=3522623HTTP/1.0" 404 558 " http://classidressing.com/**index.php?option=com_mailto&** tmpl=component&link=**aHR0cDovL2NsYXNzaWRyZXNzaW5nLm**NvbS9pbmRleC5waHA/* *b3B0aW9uPWNvbV9jb250ZW50JnZpZX**c9YXJ0aWNsZSZpZD05MzIxOjIwMTIt** MDEtMjAtMDAtMjAtNDMmY2F0aWQ9ND**U6d29tZW4tZmFzaGlvbi10cmVuZHMm** SXRlbWlkPTEwMQ==http://classidressing.com/index.php?option=com_mailto&tmpl=component&link=aHR0cDovL2NsYXNzaWRyZXNzaW5nLmNvbS9pbmRleC5waHA/b3B0aW9uPWNvbV9jb250ZW50JnZpZXc9YXJ0aWNsZSZpZD05MzIxOjIwMTItMDEtMjAtMDAtMjAtNDMmY2F0aWQ9NDU6d29tZW4tZmFzaGlvbi10cmVuZHMmSXRlbWlkPTEwMQ==" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Win64; x64; .NET CLR 2.0.50727; SLCC1; Media Center PC 5.0; .NET CLR 3.0.04506)" www.squeakci.org:80 50.93.195.16 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.yieldmanager.com/st?**ad_type=iframe&ad_size=** 300x250§ion=1949015http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=300x250§ion=1949015HTTP/1.0" 404 558 " http://www.suddengame.com/**index.htmlhttp://www.suddengame.com/index.html" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; SLCC1; Media Center PC 5.0; .NET CLR 2.0.50727)" www.squeakci.org:80 23.19.67.38 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.adserverplus.com/st?**ad_type=iframe&ad_size=728x90&** section=2898706&pub_url=${PUB_**URL}http://ad.adserverplus.com/st?ad_type=iframe&ad_size=728x90§ion=2898706&pub_url=$%7BPUB_URL%7DHTTP/1.0" 404 558 " http://femaleapple.com/index.**php?option=com_content&view=** article&id=6299:2012-01-15-02-**21-55&catid=42:health-** retreats-for-women&Itemid=98http://femaleapple.com/index.php?option=com_content&view=article&id=6299:2012-01-15-02-21-55&catid=42:health-retreats-for-women&Itemid=98" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5" www.squeakci.org:80 108.62.178.236 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.tagjunction.com/st?**ad_type=iframe&ad_size=** 300x250§ion=2933804&pub_**url=${PUB_URL}http://ad.tagjunction.com/st?ad_type=iframe&ad_size=300x250§ion=2933804&pub_url=$%7BPUB_URL%7DHTTP/1.0" 404 558 " http://bestmylive.com/index.**php?option=com_mailto&tmpl=** component&link=**73209a6d834187689d81fdf7189218**4b784d8229http://bestmylive.com/index.php?option=com_mailto&tmpl=component&link=73209a6d834187689d81fdf71892184b784d8229" "Mozilla/4.0 (compatible; MSIE 6.0; Windows CE; IEMobile 7.11)" www.squeakci.org:80 108.62.75.188 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.globe7.com/st?ad_**type=iframe&ad_size=160x600&** section=3542181&pub_url=${PUB_**URL}http://ad.globe7.com/st?ad_type=iframe&ad_size=160x600§ion=3542181&pub_url=$%7BPUB_URL%7DHTTP/1.0" 404 558 " http://fashionarrow.com/**index.php?option=com_mailto&** tmpl=component&link=**aHR0cDovL2Zhc2hpb25hcnJvdy5jb2** 0vaW5kZXgucGhwP29wdGlvbj1jb21f**Y29udGVudCZ2aWV3PWFydGljbGUmaW** Q9MjY0OTI6MjAxMS0xMi0xOS0xNi00**OS0yMSZjYXRpZD00MDpzaG9wLW9ubG** luZS1mYXNoaW9uJkl0ZW1pZD05Ng==http://fashionarrow.com/index.php?option=com_mailto&tmpl=component&link=aHR0cDovL2Zhc2hpb25hcnJvdy5jb20vaW5kZXgucGhwP29wdGlvbj1jb21fY29udGVudCZ2aWV3PWFydGljbGUmaWQ9MjY0OTI6MjAxMS0xMi0xOS0xNi00OS0yMSZjYXRpZD00MDpzaG9wLW9ubGluZS1mYXNoaW9uJkl0ZW1pZD05Ng== **" "Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.53" www.squeakci.org:80 173.208.94.17 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.scanmedios.com/st?**ad_type=iframe&ad_size=** 160x600§ion=3522623http://ad.scanmedios.com/st?ad_type=iframe&ad_size=160x600§ion=3522623HTTP/1.0" 404 558 " http://classidressing.com/**index.php?option=com_mailto&** tmpl=component&link=**aHR0cDovL2NsYXNzaWRyZXNzaW5nLm**NvbS9pbmRleC5waHA/* *b3B0aW9uPWNvbV9jb250ZW50JnZpZX**c9YXJ0aWNsZSZpZD05MzQ3OjIwMTIt** MDEtMjAtMDAtMjAtNTImY2F0aWQ9ND**U6d29tZW4tZmFzaGlvbi10cmVuZHMm** SXRlbWlkPTEwMQ==http://classidressing.com/index.php?option=com_mailto&tmpl=component&link=aHR0cDovL2NsYXNzaWRyZXNzaW5nLmNvbS9pbmRleC5waHA/b3B0aW9uPWNvbV9jb250ZW50JnZpZXc9YXJ0aWNsZSZpZD05MzQ3OjIwMTItMDEtMjAtMDAtMjAtNTImY2F0aWQ9NDU6d29tZW4tZmFzaGlvbi10cmVuZHMmSXRlbWlkPTEwMQ==" "Mozilla/4.0 (compatible; MSIE 6.0; Update a; Win32)" www.squeakci.org:80 142.91.189.9 - - [25/Oct/2012:14:30:30 +0200] "GET http://ads1.ministerial5.com/**creative/2-002134057-00001i;**size=4http://ads1.ministerial5.com/creative/2-002134057-00001i;size=4HTTP/1.0" 404 558 " http://travellingonroad.com/**index.php?view=article&catid=** 34%3Acheap-travel&id=3332%**3A2012-09-28-09-22-24&format=** pdf&option=com_content&Itemid=**53http://travellingonroad.com/index.php?view=article&catid=34%3Acheap-travel&id=3332%3A2012-09-28-09-22-24&format=pdf&option=com_content&Itemid=53" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.35 (KHTML, like Gecko) Ubuntu/10.10 Chromium/13.0.764.0 Chrome/13.0.764.0 Safari/534.35" www.squeakci.org:80 142.91.217.190 - - [25/Oct/2012:14:30:30 +0200] "GET http://ad.globaltakeoff.net/**st?ad_type=iframe&ad_size=** 300x250§ion=2186435&pub_**url=${PUB_URL}http://ad.globaltakeoff.net/st?ad_type=iframe&ad_size=300x250§ion=2186435&pub_url=$%7BPUB_URL%7DHTTP/1.0" 404 558 " http://www.ttfemalehealth.**com/index.php?option=com_** content&view=article&id=1675:**2011-07-11-01-05-13&catid=37:** mental-health&Itemid=56http://www.ttfemalehealth.com/index.php?option=com_content&view=article&id=1675:2011-07-11-01-05-13&catid=37:mental-health&Itemid=56" "Opera/9.80 (Windows NT 6.0; U; en) Presto/2.8.99 Version/11.10" www.squeakci.org:80 142.91.189.47 - - [25/Oct/2012:14:30:31 +0200] "GET http://ad.adserverplus.com/st?**ad_type=iframe&ad_size=** 300x250§ion=3256421&pub_**url=${PUB_URL}http://ad.adserverplus.com/st?ad_type=iframe&ad_size=300x250§ion=3256421&pub_url=$%7BPUB_URL%7DHTTP/1.0" 404 558 " http://newsja.com/index.php?**view=article&catid=35%** 3Acelebrity&id=8455%3A2012-05-**16-13-06-32&tmpl=component&** print=1&layout=default&page=&**option=com_content&Itemid=54http://newsja.com/index.php?view=article&catid=35%3Acelebrity&id=8455%3A2012-05-16-13-06-32&tmpl=component&print=1&layout=default&page=&option=com_content&Itemid=54" "Mozilla/4.76 [en] (X11; U; SunOS 5.7 sun4u)"
OK, I made some changes. We now have a stanza that looks like this. [1] I played with the <Proxy></Proxy> directive a bit. The result was to have requests return as 404. With the RewriteRule they all return 403. [2] Our log file is growing at about ~20M an hour. There are LogFormat directives in apache2.conf, but no CustomLog directory. It has rolled over onto another file once in the past (i.e. other_vhosts_access.log.1 from other_vhosts_access.log), but I'm not sure from where.
I take it that with the 403 requests shown in the log [2] that the pressure is off the Jenkins server but not off our logging apparatus. I think it is clear that the apache2.conf file we received is shorter than usual, shorter than the httpd.conf I'm used to in CentOS. And that with the high amount of traffic we are experiencing, we seem to be in a shipping lane.
Chris
[1]
<VirtualHost *:80> ServerName www.squeakci.org ServerAlias squeakci.org ProxyRequests Off ProxyPreserveHost On ProxyPass / http://127.0.0.1:8080/ ProxyPassReverse / http://127.0.0.1:8080/ RewriteEngine On RewriteCond %{THE_REQUEST} ^GET\ http(s?):// RewriteRule .* - [F] </VirtualHost>
[2]
www.squeakci.org:80 142.91.217.213 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.globe7.com/st?ad_type=pop&ad_size=0x0%C2%A7ion=3512133&ban... HTTP/1.0" 403 524 "http://moonhealthylive.com/index.php?view=article&catid=34%3Abeauty-and-..." "Mozilla/5.0 (X11; U; Linux i586; de; rv:5.0) Gecko/20100101 Firefox/5.0" www.squeakci.org:80 108.177.168.108 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.tagjunction.com/st?ad_type=iframe&ad_size=160x600%C2%A7ion=314... HTTP/1.0" 403 529 "http://www.entertainmentangle.com/index.php?option=com_content&view=fron..." "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.790.0 Safari/535.1" www.squeakci.org:80 108.62.75.104 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.adserverplus.com/st?ad_type=iframe&ad_size=728x90%C2%A7ion=290... HTTP/1.0" 403 530 "http://fashionlifestreet.com/index.php?view=article&catid=44%3Awholesale..." "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.1.4322)" www.squeakci.org:80 142.91.217.167 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.globe7.com/st?ad_type=iframe&ad_size=728x90%C2%A7ion=3011420&a... HTTP/1.0" 403 524 "http://www.knowledgelighthouse.com/index.php?view=article&catid=42%3Aedu..." "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Version/3.1 Safari/525.13" www.squeakci.org:80 108.62.185.146 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.globe7.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=3667021&... HTTP/1.0" 403 524 "http://likecatpink.com/index.php?view=article&catid=43%3Afashion-jewelle..." "Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.70" www.squeakci.org:80 23.19.195.254 - - [25/Oct/2012:18:06:29 +0200] "GET http://ads.creafi-online-media.com/st?ad_type=ad&ad_size=300x250%C2%A7io... HTTP/1.0" 403 538 "http://www.webgamesclub.com/index.php/play-games-online/1348-play-arcade-gam..." "Mozilla/4.76 [en] (X11; U; HP-UX B.10.20 9000/782)" www.squeakci.org:80 50.93.207.108 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=728x90%C2%A7ion=266... HTTP/1.0" 403 530 "http://www.newfindcar.com/2011/01/13/audi-tt-gt4-concept/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; YPC 3.2.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506)" www.squeakci.org:80 142.91.189.220 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.globaltakeoff.net/st?ad_type=iframe&ad_size=728x90%C2%A7ion=20... HTTP/1.0" 403 531 "http://www.qtsfinancial.com/index.php?option=com_mailto&tmpl=component&a..." "Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.9.1) Gecko/20090702 Firefox/3.5" www.squeakci.org:80 23.19.67.42 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.globe7.com/st?ad_type=iframe&ad_size=160x600%C2%A7ion=3011410&... HTTP/1.0" 403 524 "http://www.femaleapple.com/index.php?option=com_mailto&tmpl=component&am..." "Opera/10.50 (Windows NT 6.1; U; en-GB) Presto/2.2.2" www.squeakci.org:80 108.62.178.116 - - [25/Oct/2012:18:06:30 +0200] "GET http://ad.adserverplus.com/st?ad_type=pop&ad_size=0x0%C2%A7ion=3256403&a... HTTP/1.0" 403 530 "http://www.loseweightwomen.com/index.php?view=article&catid=34%3Ahealth-..." "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar)"
On Thu, 25 Oct 2012, Chris Cunnington wrote:
OK, I made some changes. We now have a stanza that looks like this. [1] I played with the <Proxy></Proxy> directive a bit. The result was to have requests return as 404. With the RewriteRule they all return 403. [2] Our log file is growing at about ~20M an hour. There are LogFormat directives in apache2.conf, but no CustomLog directory. It has rolled over onto another file once in the past (i.e. other_vhosts_access.log.1 from other_vhosts_access.log), but I'm not sure from where.
I take it that with the 403 requests shown in the log [2] that the pressure is off the Jenkins server but not off our logging apparatus. I think it is clear that the apache2.conf file we received is shorter than usual, shorter than the httpd.conf I'm used to in CentOS. And that with the high amount of traffic we are experiencing, we seem to be in a shipping lane.
I don't know how is it done on CentOS, but on Debian/Ubuntu the apache configuration file is split up into several parts (separate files/directories). The apache.conf only has server specific settings and shouldn't include anything else. Each site has it's own config file (in /etc/apache2/sites-available/) and optionally log files (usually in /var/log/apache2/). The config should include the following lines for separate log files:
CustomLog ${APACHE_LOG_DIR}/jenkins-access.log combined ErrorLog ${APACHE_LOG_DIR}/jenkins-error.log
Log files are rotated via logrotate, once a day by default.
I doubt logging is a bottleneck, but using a separate log file is useful. It would be good to check the error.log to see if apache is low on resources (or not). Also "top -d 1" can give you hints about what's eating up CPU/memory, or what's waiting for the disk for too long.
It would also be good to reconfigure jenkins to listen on only the local interface (see the link in my previous mail) and add a firewall to the server. When I set up a server, I never leave ssh on port 22, but move it to a random port and drop all packets which are not intended to be received via iptables. This reduces the number of attack attempts to almost 0.
Since I didn't find any easy to use firewall script, therefore I wrote my own init.d script for that. If there's interest in it, then I can make it available for download.
Levente
Chris
[1]
<VirtualHost *:80> ServerName www.squeakci.org ServerAlias squeakci.org ProxyRequests Off ProxyPreserveHost On ProxyPass / http://127.0.0.1:8080/ ProxyPassReverse / http://127.0.0.1:8080/ RewriteEngine On RewriteCond %{THE_REQUEST} ^GET\ http(s?):// RewriteRule .* - [F]
</VirtualHost>
[2]
www.squeakci.org:80 142.91.217.213 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.globe7.com/st?ad_type=pop&ad_size=0x0%C2%A7ion=3512133&ban... HTTP/1.0" 403 524 "http://moonhealthylive.com/index.php?view=article&catid=34%3Abeauty-and-..." "Mozilla/5.0 (X11; U; Linux i586; de; rv:5.0) Gecko/20100101 Firefox/5.0" www.squeakci.org:80 108.177.168.108 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.tagjunction.com/st?ad_type=iframe&ad_size=160x600%C2%A7ion=314... HTTP/1.0" 403 529 "http://www.entertainmentangle.com/index.php?option=com_content&view=fron..." "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.790.0 Safari/535.1" www.squeakci.org:80 108.62.75.104 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.adserverplus.com/st?ad_type=iframe&ad_size=728x90%C2%A7ion=290... HTTP/1.0" 403 530 "http://fashionlifestreet.com/index.php?view=article&catid=44%3Awholesale..." "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.1.4322)" www.squeakci.org:80 142.91.217.167 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.globe7.com/st?ad_type=iframe&ad_size=728x90%C2%A7ion=3011420&a... HTTP/1.0" 403 524 "http://www.knowledgelighthouse.com/index.php?view=article&catid=42%3Aedu..." "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Version/3.1 Safari/525.13" www.squeakci.org:80 108.62.185.146 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.globe7.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=3667021&... HTTP/1.0" 403 524 "http://likecatpink.com/index.php?view=article&catid=43%3Afashion-jewelle..." "Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.70" www.squeakci.org:80 23.19.195.254 - - [25/Oct/2012:18:06:29 +0200] "GET http://ads.creafi-online-media.com/st?ad_type=ad&ad_size=300x250%C2%A7io... HTTP/1.0" 403 538 "http://www.webgamesclub.com/index.php/play-games-online/1348-play-arcade-gam..." "Mozilla/4.76 [en] (X11; U; HP-UX B.10.20 9000/782)" www.squeakci.org:80 50.93.207.108 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.yieldmanager.com/st?ad_type=iframe&ad_size=728x90%C2%A7ion=266... HTTP/1.0" 403 530 "http://www.newfindcar.com/2011/01/13/audi-tt-gt4-concept/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; YPC 3.2.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506)" www.squeakci.org:80 142.91.189.220 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.globaltakeoff.net/st?ad_type=iframe&ad_size=728x90%C2%A7ion=20... HTTP/1.0" 403 531 "http://www.qtsfinancial.com/index.php?option=com_mailto&tmpl=component&a..." "Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.9.1) Gecko/20090702 Firefox/3.5" www.squeakci.org:80 23.19.67.42 - - [25/Oct/2012:18:06:29 +0200] "GET http://ad.globe7.com/st?ad_type=iframe&ad_size=160x600%C2%A7ion=3011410&... HTTP/1.0" 403 524 "http://www.femaleapple.com/index.php?option=com_mailto&tmpl=component&am..." "Opera/10.50 (Windows NT 6.1; U; en-GB) Presto/2.2.2" www.squeakci.org:80 108.62.178.116 - - [25/Oct/2012:18:06:30 +0200] "GET http://ad.adserverplus.com/st?ad_type=pop&ad_size=0x0%C2%A7ion=3256403&a... HTTP/1.0" 403 530 "http://www.loseweightwomen.com/index.php?view=article&catid=34%3Ahealth-..." "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar)"
On 2012-10-25 12:50 PM, Levente Uzonyi wrote:
On Thu, 25 Oct 2012, Chris Cunnington wrote:
OK, I made some changes. We now have a stanza that looks like this. [1] I played with the <Proxy></Proxy> directive a bit. The result was to have requests return as 404. With the RewriteRule they all return 403. [2] Our log file is growing at about ~20M an hour. There are LogFormat directives in apache2.conf, but no CustomLog directory. It has rolled over onto another file once in the past (i.e. other_vhosts_access.log.1 from other_vhosts_access.log), but I'm not sure from where.
I take it that with the 403 requests shown in the log [2] that the pressure is off the Jenkins server but not off our logging apparatus. I think it is clear that the apache2.conf file we received is shorter than usual, shorter than the httpd.conf I'm used to in CentOS. And that with the high amount of traffic we are experiencing, we seem to be in a shipping lane.
I don't know how is it done on CentOS, but on Debian/Ubuntu the apache configuration file is split up into several parts (separate files/directories). The apache.conf only has server specific settings and shouldn't include anything else. Each site has it's own config file (in /etc/apache2/sites-available/) and optionally log files (usually in /var/log/apache2/). The config should include the following lines for separate log files:
CustomLog ${APACHE_LOG_DIR}/jenkins-access.log combined ErrorLog ${APACHE_LOG_DIR}/jenkins-error.logLog files are rotated via logrotate, once a day by default.
Several files instead of one big file. OK. The /sites-available/default does have a CustomLog directive.
I doubt logging is a bottleneck, but using a separate log file is useful. It would be good to check the error.log to see if apache is low on resources (or not). Also "top -d 1" can give you hints about what's eating up CPU/memory, or what's waiting for the disk for too long.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15277 lewis 20 0 1028m 89m 1300 R 2.0 8.9 18:15.60 squeakvm 1 root 20 0 2032 652 556 S 0.0 0.1 0:15.83 init
lewis 15277 2.1 8.8 1053112 91572 pts/0 S 04:40 18:15 /usr/local/lib/squeak/4.10.5-2619/squeakvm -nodisplay /home/lewis/VMUnixBuild/Squeak4.3.image /home/lewis/VMUnixBuild/VMUnixBuild.st
That is consistently at the top of top -d 1. This VPS has 1G of RAM.
It would also be good to reconfigure jenkins to listen on only the local interface (see the link in my previous mail) and add a firewall to the server. When I set up a server, I never leave ssh on port 22, but move it to a random port and drop all packets which are not intended to be received via iptables. This reduces the number of attack attempts to almost 0.
These are good ideas and greater system administration ideas. But for today, I think the greater issue of Jenkins has been addressed. (As in, I need a bit of a break.)
Since I didn't find any easy to use firewall script, therefore I wrote my own init.d script for that. If there's interest in it, then I can make it available for download.
Yes. I would like to see that, please.
Chris
On Thu, Oct 25, 2012 at 01:04:25PM -0400, Chris Cunnington wrote:
On 2012-10-25 12:50 PM, Levente Uzonyi wrote:
I doubt logging is a bottleneck, but using a separate log file is useful. It would be good to check the error.log to see if apache is low on resources (or not). Also "top -d 1" can give you hints about what's eating up CPU/memory, or what's waiting for the disk for too long.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15277 lewis 20 0 1028m 89m 1300 R 2.0 8.9 18:15.60 squeakvm 1 root 20 0 2032 652 556 S 0.0 0.1 0:15.83 init
lewis 15277 2.1 8.8 1053112 91572 pts/0 S 04:40 18:15 /usr/local/lib/squeak/4.10.5-2619/squeakvm -nodisplay /home/lewis/VMUnixBuild/Squeak4.3.image /home/lewis/VMUnixBuild/VMUnixBuild.st
That is consistently at the top of top -d 1. This VPS has 1G of RAM.
Oops, sorry about that. If it happens again feel free to "sudo killall squeakvm" for me.
FYI, I do now have a reasonably safe build script in place, despite the above glitch. There is a shell script ~lewis/VMUnixBuild/makevm that runs a Squeak script ~lewis/VMUnixBuild/VMUnixBuild.st. The shell script has a watchdog loop that is supposed to kill the Squeak process if it runs longer than 30 minutes (which does in fact happen sometimes).
If everything works properly, the full script runs for about 8 minutes. The end result is a tarball containing bleeding edge Unix and VMM-generated sources. These are currently located in ~lewis/VMUnixBuild/TARBALL, and logging output from the script is in ~lewis/VMUnixBuild/LOGS.
The script does a complete VM compile before making the tarball, which ensures that each tarball contains a set of sources that is consistent enough to build a VM if you were to download it to your own Linux box and do the configure/make for yourself.
I don't have a clue how Jenkins works yet, but presumably the next step would be to wire this into the Jenkins environment so the script could run daily, and provide the latest tarball as the build artifact.
Dave
On Thu, 25 Oct 2012, Chris Cunnington wrote:
On 2012-10-25 12:50 PM, Levente Uzonyi wrote:
On Thu, 25 Oct 2012, Chris Cunnington wrote:
OK, I made some changes. We now have a stanza that looks like this. [1] I played with the <Proxy></Proxy> directive a bit. The result was to have requests return as 404. With the RewriteRule they all return 403. [2] Our log file is growing at about ~20M an hour. There are LogFormat directives in apache2.conf, but no CustomLog directory. It has rolled over onto another file once in the past (i.e. other_vhosts_access.log.1 from other_vhosts_access.log), but I'm not sure from where.
I take it that with the 403 requests shown in the log [2] that the pressure is off the Jenkins server but not off our logging apparatus. I think it is clear that the apache2.conf file we received is shorter than usual, shorter than the httpd.conf I'm used to in CentOS. And that with the high amount of traffic we are experiencing, we seem to be in a shipping lane.
I don't know how is it done on CentOS, but on Debian/Ubuntu the apache configuration file is split up into several parts (separate files/directories). The apache.conf only has server specific settings and shouldn't include anything else. Each site has it's own config file (in /etc/apache2/sites-available/) and optionally log files (usually in /var/log/apache2/). The config should include the following lines for separate log files:
CustomLog ${APACHE_LOG_DIR}/jenkins-access.log combined ErrorLog ${APACHE_LOG_DIR}/jenkins-error.logLog files are rotated via logrotate, once a day by default.
Several files instead of one big file. OK. The /sites-available/default does have a CustomLog directive.
I doubt logging is a bottleneck, but using a separate log file is useful. It would be good to check the error.log to see if apache is low on resources (or not). Also "top -d 1" can give you hints about what's eating up CPU/memory, or what's waiting for the disk for too long.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15277 lewis 20 0 1028m 89m 1300 R 2.0 8.9 18:15.60 squeakvm 1 root 20 0 2032 652 556 S 0.0 0.1 0:15.83 init
lewis 15277 2.1 8.8 1053112 91572 pts/0 S 04:40 18:15 /usr/local/lib/squeak/4.10.5-2619/squeakvm -nodisplay /home/lewis/VMUnixBuild/Squeak4.3.image /home/lewis/VMUnixBuild/VMUnixBuild.st
That is consistently at the top of top -d 1. This VPS has 1G of RAM.
It would also be good to reconfigure jenkins to listen on only the local interface (see the link in my previous mail) and add a firewall to the server. When I set up a server, I never leave ssh on port 22, but move it to a random port and drop all packets which are not intended to be received via iptables. This reduces the number of attack attempts to almost 0.
These are good ideas and greater system administration ideas. But for today, I think the greater issue of Jenkins has been addressed. (As in, I need a bit of a break.)
I can do all this stuff if I get access to the box. ;)
Since I didn't find any easy to use firewall script, therefore I wrote my own init.d script for that. If there's interest in it, then I can make it available for download.
Yes. I would like to see that, please.
Chris
I uploaded the firewall script to http://leves.web.elte.hu/squeak/firewall . This basic version keeps port 22 and 80 open and drops all packets arriving to other ports. If you want something different, then you have to change the rules() function. When the file is copied to /etc/init.d , then
$ /etc/init.d/firewall start
will "start" the firewall. If you want to start it whenever the server is restarting, then just do
$ /etc/init.d/firewall install
I'd suggest commenting out the line
iptables -A INPUT -j DROP
the first time, so you won't lose the connection to the server if anything goes wrong. :)
Levente
On 2012-10-26 4:43 PM, Levente Uzonyi wrote:
On Thu, 25 Oct 2012, Chris Cunnington wrote:
On 2012-10-25 12:50 PM, Levente Uzonyi wrote:
These are good ideas and greater system administration ideas. But for today, I think the greater issue of Jenkins has been addressed. (As in, I need a bit of a break.)
I can do all this stuff if I get access to the box. ;)
If we could make Levente a sysadmin on box3 that would be great.
Chris
On 10/26/2012 03:52 PM, Chris Cunnington wrote:
On 2012-10-26 4:43 PM, Levente Uzonyi wrote:
On Thu, 25 Oct 2012, Chris Cunnington wrote:
On 2012-10-25 12:50 PM, Levente Uzonyi wrote:
These are good ideas and greater system administration ideas. But for today, I think the greater issue of Jenkins has been addressed. (As in, I need a bit of a break.)
I can do all this stuff if I get access to the box. ;)
If we could make Levente a sysadmin on box3 that would be great.
Chris
Added, sorry about the delay. I emailed Levente with the details.
Ken
On 2012-10-26 4:43 PM, Levente Uzonyi wrote:
On Thu, 25 Oct 2012, Chris Cunnington wrote:
On 2012-10-25 12:50 PM, Levente Uzonyi wrote: Yes. I would like to see that, please.
Chris
I uploaded the firewall script to http://leves.web.elte.hu/squeak/firewall . This basic version keeps port 22 and 80 open and drops all packets arriving to other ports. If you want something different, then you have to change the rules() function. When the file is copied to /etc/init.d , then
$ /etc/init.d/firewall start
will "start" the firewall. If you want to start it whenever the server is restarting, then just do
$ /etc/init.d/firewall install
I'd suggest commenting out the line
iptables -A INPUT -j DROP
the first time, so you won't lose the connection to the server if anything goes wrong. :)
Levente
This looks cool. I'll start to read it over. If it closes all the ports except 22 and 80, then I'm going to need to learn to add lines for RFB and the Altitude image:
iptables -A INPUT -p tcp --dport 5900 -j ACCEPT #vnc
Thanks for making this available.
Chris
On Fri, 26 Oct 2012, Chris Cunnington wrote:
On 2012-10-26 4:43 PM, Levente Uzonyi wrote:
On Thu, 25 Oct 2012, Chris Cunnington wrote:
On 2012-10-25 12:50 PM, Levente Uzonyi wrote: Yes. I would like to see that, please.
Chris
I uploaded the firewall script to http://leves.web.elte.hu/squeak/firewall . This basic version keeps port 22 and 80 open and drops all packets arriving to other ports. If you want something different, then you have to change the rules() function. When the file is copied to /etc/init.d , then
$ /etc/init.d/firewall start
will "start" the firewall. If you want to start it whenever the server is restarting, then just do
$ /etc/init.d/firewall install
I'd suggest commenting out the line
iptables -A INPUT -j DROP
the first time, so you won't lose the connection to the server if anything goes wrong. :)
Levente
This looks cool. I'll start to read it over. If it closes all the ports except 22 and 80, then I'm going to need to learn to add lines for RFB and the Altitude image:
iptables -A INPUT -p tcp --dport 5900 -j ACCEPT #vnc
Thanks for making this available.
You shouldn't open any ports for VNC (especially not the default 5900), but tunnel it through SSH instead, because the VNC connection is not encrypted[1]. If someone gets your password, then he gains full access to the site and some access to the box too. By default the Squeak RFB server disallows non-local connections in order to avoid this issue.
Levente
[1] http://en.wikipedia.org/wiki/Virtual_Network_Computing#Security
Chris
box-admins@lists.squeakfoundation.org
-
Chris Cunnington -
David T. Lewis -
Ken Causey -
Levente Uzonyi