We got a new downtime on champagne.m.o (2012 August 27 evening), making several of our websites unavailable. It occurred that it was caused by a small DOS (~100+ IP address, ~280k requests in ~2h). This could be prevented maybe (mod_evasive, mod_security, fail2ban, or have a cache in front - varnish?). But the server went down here particularly because httpd forked a lot of processes (200+) without a limit. Sander suggested that we should have a look at httpd memory limits config. Currently, no such limitation is set, thus Apache can start as many processes as it needs, with no memory cap. That may be useful for other services too.
Current existing config for champagne is: <IfModule mpm_prefork_module> AcceptMutex fcntl CoreDumpDirectory /tmp EnableExceptionHook Off ListenBacklog 511 LockFile logs/accept.lock MaxClients 256 MaxMemFree 0 MaxRequestsPerChild 4000 PidFile /var/run/httpd.pid ReceiveBufferSize 0 # ScoreBoardFile logs/apache_runtime_status SendBufferSize 0 ServerLimit 256 StartServers 8 MinSpareServers 5 MaxSpareServers 20 </IfModule>
CC: (none) => sander.lepikHardware: i586 => All
http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients If we assume that average apache process (with php loaded) will take ~15+ MB of memory and we can use ~1500MB memory for apache then the maximum we can set for MaxClients is 1500/15 = 100. Do we have some monitoring (munin or something like that) to see what's the average usage? I think 100 is enough. And what are the values for KeepAlive* stuff?
We use http://xymon.mageia.org/xymon-cgi/bb-hostsvc.sh?CLIENT=champagne.mageia.org php.ini memory_limit is 128M. max_execution_time is 30. Additional config values for Apache: Timeout 300 # largely excessive. 6 would be already a max here, we don't have KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 5
I've limited MaxClients to 80 for now to get some more stats on memory and cpu needed... I'll monitor it and adjust more when I have better stats
CC: (none) => tmb
Why not switch to nginx+php-fcgi?
CC: (none) => thierry.vignaud
i'm doing some defense with iptables it was in the standard script i've modified it a lil iptables -A LnD -p tcp\udp\icmp -m limit --limit 111/s -j LOG --log-prefix "[that's too much]" --log-level=info lnd is a custom drop table, and you can change amount of queries per second then i parse syslog (it is cron hourly) grep -iP "that's too much" /var/log/messages | awk '{a[$13]++;} END{for( i in a) {if{a[i]>7 && length(i) > 6) {print i;}}}' | awk -F '=' '{print $2}'|while read i;\ do X=$(grep $i /etc/firewall/firewall.banned); if [ $X ]; then echo "$i exist" >> /dev/null else iptables -A Banned -i $EXTIF -s $i -j Banned iptables -A Banned -o $EXTIF -d $i -j Banned echo $i >> /etc/firewall/firewall.banned; echo "Firewall ban rule: $i - this one is banned now!" >> /var/log/messages so it is banning for some reason that you can describe in ruleset, sometimes it is banning <> i don't know why, maybe need more work, of course for servers the rate can be raised a bit, but it needs more testing maybe
CC: (none) => oeai
oh and actually you don't need to ban, you just can redirect to drop table such connections
Closing
Status: NEW => RESOLVEDCC: (none) => boklmResolution: (none) => OLD
CC: boklm => (none)