Skip to content

When Django Apps Grow Up

by Paul Kenjora on December 11th, 2007

I started using webfaction.com 6 months ago for my Django website hosting. I’ve deployed several sites to their servers with no worries. Things went smoothly with my shared host for a while… then the warning signs that I needed to upgrade began to appear. Within a week it was time to move to a dedicated host. Arkayne.com hit the wall fast.

How Do You Know Its Time To Upgrade?

There were several warning signs that made it clear I needed to upgrade. The surprise was how fast things snowballed. Here is what you need to watch out for when using a shared host for Django applications:

  • Your apache process running mod_python sits in the top 5 CPU consuming spots (run top).
  • Your find yourself upgrading memory on your shared plan.
  • You receive several emails indicating long running processes more and more frequently.
  • Your site is sometimes down or very sluggish, proxy and gateway errors are common at this point.
  • The admin tells you to optimize code or queries.
  • You find yourself optimizing code for the 3rd time.
  • Other sites hosted on the same server are running slow.
  • Your hits are approaching 10K hits a day.

Can I Stall The Need to Commit?

If any of the above symptoms are present its time to think about a dedicated host. You may argue that optimizing some things can forestall the inevitable, here is what I tried:

Month Hits/Day Optimization Results
3 1,000 Optimize MySQL with indexes. Site became significantly more responsive, processing time cut 80% for batch jobs.
3.5 4,000 Switch static content to be served directly via Apache vs. Django. Webfaction used an Apache proxy, in essence I changed static content like images, JavaScript, and Style Sheets to be served by main Apache instance. Big improvement in memory usage, CPU dropped slightly. No longer got memory usage emails from the admin team.
4 10,000 Optimized my MySQL queries. No net effect, even though CPU load decreased the site was growing so quickly that WebFaction shut down my page for a few days. I was bringing down the shared server with alarming regularity.

What are my options for upgrading?

I spent alot of time on this and made a few mistakes. There are many hosts and the all offer slightly different deals. If youre looking for top of the line servers, inpenetrable security, or quadruple backups with guranteed fail safes then stop reading here. For the rest of you, like myself, you have a growing site and a constant budget. The important thing is to keep the site up and allow it to expand to the next level. I strongly recommned (after much research and discussion among collegues) ServerPronto.

http://serverpronto.com

They are the best and most cost effective step after shared hosting. I filled out the online form and within 1 hour received a call to verify I had placed the order and that it met my needs. Within 14 hours I recveived an email with full root login, control panel, and support numbers. Everything worked on the first attempt, my billing was clear and to the point.

From there on it took about 4 hours to migrate my site from WebFaction to ServerPronto. I’m waiting for my DNS to catch up and hope that I haven’t angered too many of my Beta users.

Thanks WebFaction – Shared Hosting just $9.95! for getting me here.

Thanks ServerPronto – Full Dedicated Servers just $29.95! for taking me further.

More from Aware Labs

Aware Labs Recommends

AwareLabs has content marketing with Arkayne Socialize.

From → Environment Setup, Tech News

  • Unfortunately the majority of content pulled by Arkayne is dynamic and unique for each call. I have Apache serving all the static content. The caching approach does sound applicable to a few of my other sites though. I'll give it a try.

    As far as running VMWare on Server Pronto, I'm not sure I have the need. VMWare is absolutely awesome by the way. My experience with deploying a django application on Fedora has been great. I used "yum" to install django, mysql, and a fe python libraries. Beyond that setting up Apache and copying my Django application was cake. All in all the setup took a few hours, and I could probably repeat it in 30 minutes this time around. Setting up extra IP addresses and configuring VMWare on the server sounds like a comprable task.

    If I ever plan to deploy to multiple servers then VMware may be the right solution.
  • Re Django caching:

    The invaluable James Bennett had a good article on performance tips that included a section on caching (http://www.b-list.org/weblog/2007/nov/27/performance/). Basically though Django gives you a bunch of options ranging from whole-site caching (eg: any page w/o POST or GET parameters is cached the first time it is generated and reused until the (configurable) time limit is reached.)

    Say for example the front page of your website is pulling dynamic data from the DB - the last ten somethings, a menu structure, a content block, etc. The process for generating all that involves a dozen queries, view code that builds data structures and templates that render the data. The result is just a text file though - and if you cache it for 10 seconds and get 100 hits in those 10 seconds only the first hit goes through the whole process of querying the db, rendering data, etc, and the other 99 simply feed the saved file. You can see that this would have a dramatic effect on performance.

    If you just have too much dynamic data for a whole site cache you can also cache individual views (as simple as adding a decorator to the view function) or even use caching template tags to cache fragments in a template. There's also access to the Cache API in python code so you can cache objects (like query results, etc). As usual the docs are pretty good...
  • Pete
    I agree about having to maintain your own server: it's a pain in the neck. At first everything runs smoothly but when you start having to apply patches yourself and re-compiling things that aren't in yum you start pulling your hair.

    Also, I'd be curious to know how far the 256MB of memory get you. Once you run the operating system, the web server and database server that doesn't leave much left for your actual application and memcache.

    With most shared hosting providers, the memory that they give you is for your actual application and you don't have to worry about the memory used by the OS, the web server and database server.
  • So, was it actually the number of hits per day on your server that was consuming all the system resources, or some background processes running? If it was the background process, how much of the database does it hit, is it a small enough piece that memcache would solve your problem? I can't imagine that 10k hits per day would be that hard on a server, unless each hit results in an overly expensive query....in which case...that should probably be fixed.

    It is also worth mentioning, in order to make serverpronto cost effective, you need to know how to sysadmin your site. They won't help you...unless you want to pay them, which blows the whole cheap thing out of the water.I recommend running in vmware server in order to be able to easily move from one serverpronto machine to another..upgrading as necessary. Purchase a second ip address - a 1 time fee of $10 and use bridged networking. Setup iptables on both host and vm....i tossed all that web based admin bullshit it comes with. weak sauce.
  • To clarify, I'm not slamming WebFaction. They have been great and I'm still hosting some of my sites on their machines.

    As far as efficiency. I was running a few expensive batch jobs every so often but without those it was a simple query. Fetch a set of indexed records and return them. Nothing too complex, query itself took very little time. My processes hovered arounf 18MB a piece but my CPU tended to spike to around 50%-90% on a hit.

    Every site is different I guess. As far as being lucky or unlucky, Web Faction offered to move me to another machine because that one was heavily loaded. I didn't take them up on it because I was worried I'd have the same problem all too soon. From looking around the processes I could tell lots of other popular sites were on there too. And we definately had more than 2 sites on the server.

    On the dedicated machine performance is significantly better without any code or query optimization. All things aside, given time and money its the best optimization I could have done at the time.

    Tell me more about Django caching, I may be doing it but know it by a different name. What is it and how do I activate it? What are the benefits?
  • Pete
    I'm thinking your code must have been grossly inefficient because I handle about 200K hits per day at WebFaction with no pb at all... My code is a basic Django site with a few queries per page. I didn't even have to enable caching, but I did have to fix a few very expensive queries.
  • I'd like to give a thumbs-up to WebFaction too!
    We've hosted http://ShowMeDo.com there for a year (and we eat cpu/memory with increasingly regularity now) - Remi, David and co have been incredibly responsive and helpful.
    I've only had memory-warnings when indeed I've gone over-quota, they're helpful and have offered advice. Switching to memcached was a big win for us, as was optimising SQLObject queries.
    I'm a little shocked by your comments about the team up there - that's the first bad news I've heard about them...I'd suggest other readers do some research (maybe I've just missed other bad write-ups?) but all I've heard are good things. My own experiences have been great - supportive, responsive and understanding.
    Ian.
  • Observer
    Umm. I handle a *lot* more than 10k hits on a shared webfaction account with Django, and my setup doesn't use much CPU at all. And the sites are fast as heck. Any special reason you don't mention caching at all? Using Django without any kind of caching is a bad idea, no matter what setup you're using. Using Django without any kind of caching on a shared setup should be punishable by law.
  • Mike
    Anonymous: Are you sure we're talking about the same company? I have had a complete opposite experience with them: fast server and helpful support. I can't speak for all the servers they have but I have 3 accounts on different servers and all of them have a load around 0.5, which is very good. Maybe you got unlucky and your server was more loaded ...
  • Yeah sorry about that, I woke up one day and Arkayne was getting hammered. I tried getting webfaction to upgrade the server but it was all too slow. The site was growing faster than I could optimize.

    To be fair, those guys have way too many sites crammed into one shared host. Jeff Croft's site is on the same server as well as a few other popular ones. I think the box we were on was destined to swell beyond capacity sooner or later.

    Good and bad news is Arkayne swelled sooner. Sorry for crowding the CPU, should all be fine now since I relocated. Unless one of the other sites does the same, which appearantly as everyone tells me is a nice problem to have. I always thought too much money was a nice problem, not increasing costs.
  • rezzrovvv
    I was one who emailed webfaction about your site ;) You were killing my site.
  • Noah
    We're moving from Web Faction to Rackspace.
  • Anonymous
    I actually manage django apps on many accounts on Web Faction's web 3 through web 15. I started using Web Faction about a year and a half ago and it's gone down hill so much since then, service response times are totaly unacceptable, we get out of memory warnings when we're using very little memory, well within our limits, support tickets take forever to get answered and most of the new staff is incompetent. I miss the old days. Alas we're moving to a dedicated machine because it's not worth it to deal with the never ending web faction problems.
blog comments powered by Disqus