This site has lately been plagued with mysql crashes. After lots of procrastination, I finally decided to look under the hood rather than just restart the mysql daemon. Here’s what I found in the mysqld.log:
Did you see the fatal error in there? Looks like this micro instance it having difficulty allocating memory. After doing a little Internet research, I discovered that AWS micro instances with Linux do not have a swap space configured by default. So, without swap space, the database was crashing whenever memory became unavailable (which was becoming more frequent). Boo!
There’s an easy solution, however — set up swap space on the instance. Here’s how I did that
dd if=/dev/zero of=/swapfile bs=1M count=1024
** this says, read from file /dev/zero, write to file /swapfile, reading 1m of data at a time, copying only 1024 input blocks.
** For more on dd utility, see the wikipedia dd entry.
** make the swap area
** turn on swapping
- In the /etc/fstab file, add this line:
/swapfile swap swap defaults 0 0
** this sets up the swapfile automatically after a restart
I then decided to take a closer look at total memory usage on the box by running
Whoa, that’s no good. Only 7M available? How much of that memory is actively being used by mysql? To check that out, I got the mysql pid by running
ps -ef | grep mysql. Then I looked at the process status file at /proc/$pid/status.
Humm, not much memory being used here, but I just restarted the process not that long ago. To be safe, I set a bound on the amount of memory that mysql can allocate. I updated the /ect/my.cnf file to set a buffer pool size:
datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock innodb_buffer_pool_size = 256M
I then went back to looking at how memory is being used across the system by running top, shift+m.
Looks like apache might be a bit of a pig too for this low memory machine. To limit the amount of work apache is doing on this little box, I made a special optimizing_memory.conf file in my /etc/httpd/conf.d directory; my http.conf file in /etc/http/conf automatically includes all config files from the httpd directory. Keeping my apache config changes in a separate file helps me keep track of my modifications versus the defaults. The content of the optimizing_memory.conf file are below, ripped directly out of this helpful apache tuning article.
Timeout 30 KeepAlive On MaxKeepAliveRequests 50 KeepAliveTimeout 10 <IfModule prefork.c> StartServers 3 MinSpareServers 2 MaxSpareServers 5 MaxClients 10 MaxRequestsPerChild 1000 </IfModule>
I don’t think I need to adjust the memory usage in my php.ini file — it looks pretty reasonable.
Ok, so how is memory usage now? One more time with top, shift+m.
And how about
Looks like things are a little more under control. Now that I have a gig of swap set up and some bounds on memory usage, it’s time to sit back and hope that the site stays up more reliably :)