Skip to content

Its Not: InnoDB, MySQL, Django, or Developer Its The Unix Admin

by Paul Kenjora on February 28th, 2008

Almost exactly a month ago in my “Easy Fix: Mysql InnoDB: Database was not shut down normally” post I blamed InnoDB for my periodic crashes of MySQL which in turn brought down all my Django applications. Well InnoDB I owe you an apology.

The Real Probelm

The probelm as it turns out was alluded to by the person who ended up fixing it for me. In the comments to my original post, Easy Fix: Mysql InnoDB: Database was not shut down normally, Brian Schneider, founder of Help Me Sue, suggested I check if the server was running a swap partition.

From a cause and effect perspective missing swap space makes perfect sense when looking at an InnoDB malloc error. As mentioned in the original post the server was crashing because InnoDB was crashing after MySQL threw the malloc error. I also mentioned that this was all triggered by Google Bot hitting the server (deduced from the logs). Even dumping the DB and restarting it threw me off, it freed memory temporarily. If the server was not using a swap partition then it was limited to 2GB of RAM and all of the above makes sense. Unfortunately I’m just a Django developer and under the assumption that my Unix admin set the server up correctly. I debugged as well as I knew down to the DB layer then stopped, Brian didn’t…

The Real Fix

Earlier this week I mentioned to Brian that the InnoDB problem has re-appeared. He immediately suggested looking at the swap partition. Just type “top” and see if any memry is allocated to top.


top - 16:15:07 up 30 days, 10:44, 2 users, load average: 0.19, 0.20, 0.09
Tasks: 256 total, 3 running, 253 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.7%us, 1.2%sy, 0.0%ni, 90.6%id, 2.3%wa, 0.1%hi, 0.2%si, 0.0%st
Mem: 1018696k total, 959300k used, 59396k free, 25000k buffers
Swap: 0 total, 0 used, 0 free, 0 cached

All the Swap fields read zero. This is bad. We checked if there is a swap partition and there was. We checked the /etc/fstab file and the swap entry looked funny. The /etc/fstab file si where partitions are mounted and allocated. There was some LABEL stuff in there that did not jive well with the other entries.


LABEL=/ / ext3 defaults 1 1
LABEL=/boot1 /boot ext3 defaults 1 2
devpts /dev/pts devpts gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
LABEL=SWAP-hda2 swap swap defaults 0 0

We did some searching on Google and found some information that convinced us to take a leap of faith. We changed the swap partition entry to look just like the other entries.


LABEL=/ / ext3 defaults 1 1
LABEL=/boot1 /boot ext3 defaults 1 2
devpts /dev/pts devpts gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
/dev/hda2 swap swap defaults 0 0

Running a few commands like “man swapon” and “swapon -a” we had success. The swap directory was mounted, activated and when we ran top things looked better.


top - 16:15:07 up 30 days, 10:44, 2 users, load average: 0.19, 0.20, 0.09
Tasks: 256 total, 3 running, 253 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.7%us, 1.2%sy, 0.0%ni, 90.6%id, 2.3%wa, 0.1%hi, 0.2%si, 0.0%st
Mem: 1018696k total, 959300k used, 59396k free, 25000k buffers
Swap: 2048276k total, 96k used, 2048180k free, 540012k cached

Moral Of The Story

Being just a Django developer is not always enough. Sometimes you need to wear other hats. If you don’t want to wear other hats youre stuck with shared hosting. Also it pays to have smart friends. And finally, there is no such thing as an easy fix.

For those of you interested in suing me for this blog and its sometimes erroneous content, I suggest you try Help Me Sue. You’ll feel good knowing you’re suing on a Django platform.

More from Aware Labs

Aware Labs Recommends

AwareLabs has content marketing with Arkayne Socialize.

From → Environment Setup, Tutorial

  • you should bitch out the guys at server pronto...the image that they gave you should certainly have had swap turned on..and no doubt you are not the only one that had this problem. maybe you can convince them to give you a free month of service for your troubles.
  • John M
    Wow, sounds like where I work. Basic 101 ideas not followed. I mean, you'd have to go out of your way to NOT setup a swap partition on most installs today!

    Crazy, and a nice post BTW

    John
blog comments powered by Disqus