On 11 July 2012 23:08, Adam Bower adam@thebowery.co.uk wrote:
On Wed, Jul 11, 2012 at 10:11:03PM +0100, Richard Lewis wrote:
I'm not (so far) seeing any other processes using any significant amount of RAM, apart from MySQL. But that seems fairly static at 1.7%.
That suggests it may be a single thing that happens that causes something to eat memory all of a sudden. I'm afraid you'll just have to keep waiting in this case :)
Part of the point of this exercise is simply to see if memory usage stays constant over time or if it suddenly gets used exponentially or linearly which might help you after it has gone wrong again.
I had an OOM killer problem on one of my vms hosted at bytemark for weeks before I managed to trace the problem: bots trawling the trac directory of an apache site. Banning them with robots.txt fixed it. I had one script running every five minutes that checked for memory usage, and if it was above a certain amount then to send all sorts of memory usage/processes data to an output file. From this I could see it was always apache (even though, as Adam says, oomkiller was randomly killing anything it could to reclaim memory), and from there start to monitor the apache connections until I found it always stuck on listening to googlebots. Memory usage would balloon from a few hundred MB to > 800 within minutes, when oomkiller kicked in, making it very hard to pinpoint the problem.
I've still got scriptage if that helps.
Jenny