At Wed, 11 Jul 2012 18:57:39 +0100, Adam Bower wrote:
On Wed, Jul 11, 2012 at 04:53:06PM +0100, Richard Lewis wrote:
Below is some console output from a DomU that suffered this problem earlier today. You can see that the OOM killer killed Apache. And I'm guessing it killed sshd too as I couldn't connect to the guest. I couldn't find any errors in Xen's logs.
Any thoughts on what might be going on here?
The oom killer doesn't necessarily kill what is using the RAM it guesses at the best thing to kill. See http://linux-mm.org/OOM_Killer for more information on that.
What I'd suggest is that you put a job into cron that runs every 5 minutes and grabs a complete process list and shows how much RAM each process is using. You should then be able to see where the memory is going and what is using it as I'm going to suggest something has a memory leak but it may not be Apache itself causing the problem.
Thanks for the suggestion. It's been executing a script along these lines:
#!/bin/bash
LOG=/var/log/rjl-mem-info.log NOW=`date +"%Y-%m-%d %H:%M:%S"` echo "============================" >> $LOG echo $NOW >> $LOG ps aux | awk '{print $4, $10, $11}' | sort -rn | head -30 >> $LOG free -m >> $LOG
every couple of minutes for the last few hours. So far swap usage has remained almost entirely static at 20 MB. And RAM usage has fluctuated between about 400 and 500 MB. Of course, this has reminded me that these VMs are using pre-fork Apache, so I see 10 Apache processes. Typically, 7 of them report that they are using ~9% of the RAM, and 3 that they are using ~8%. Then there's also the parent Apache process using ~2%.
I'm not (so far) seeing any other processes using any significant amount of RAM, apart from MySQL. But that seems fairly static at 1.7%.
And, of course, it's also failed to misbehave :-(
Cheers, Richard