How to handle 2500+ times your expected traffic, without killing your server

Today, I noticed one of the servers I run was showing some scary looking stats. In fact, it appeared to be averaging, over the previous 24 hours, 2 million hits an hour. To a specialist site, which I'd expect to get maybe 800 hits an hour.

The curious thing about this wasn't so much the number of hits, more that I hadn't noticed it until I looked at the stats on the server, despite having spent the previous 3 hours working on the server. It hadn't affected the response speed of the server at all.

Now, this might be an indication of a massively over-specced server, but in this case, it runs a whole load of sites, and at peak traffic on the larger ones, it does slow down slightly - not massively, but you notice it if you're working on the machine. In fact, I initially thought the stats generation system which reads the log files had gone wrong. I looked at the log files. The server processes were indeed serving the pages, and logging them. Without impacting anything else.

All of the traffic was coming from a single IP, although for a range of pages. It appeared, in fact, to be a crawler stuck in a loop. So, step 1, preserve the rest of my bandwidth - block the rogue crawler. Traffic stops, and now I can look at how it was doing this miracle.

For background, the site in question is Drupal based, using Views, CCK, a custom theme, and a few other easily available modules. It runs on a CentOS server, using Apache 2.2.x, PHP 5.2.x (from a third party repository) and MySQL 5.1. This server is fairly modest by modern standards - it's a dual processor box with 2Ghz dual core processors in, and just 2Gb of RAM.

The trick, in this case, appears to be in several things, some of which I could control, and one, very important one, which I couldn't. The one I couldn't control was the content being requested - it was a rogue crawler, which was grabbing text content. If the content had been graphics, or even full pages (each of which would trigger a lot more hits, thanks to CSS files, Javascript files, images, and so on), the server almost certainly wouldn't have handled it so gracefully.

The other bits were down to the server and module configuration. In particular:

1. APC - PHP op-code caching is essential for any large PHP deployment. Seriously, if you don't have an op-code cache running on your server, go and install one now. I like APC, which is developed by the PHP developers, but there are others too - take your pick. Then tune it - spend some time watching it fill up over a normal day/week, and make adjustments until it doesn't fill up under normal use (rebuilding cache contents after it overfills is slow - much better to rebuild it as required, when individual pages expire), and it is stable. In this case, I've got it using just under 200Mb of RAM, which neatly holds all the op-code for the various sites hosted on this box. A single Drupal site, with views, usually fits in 64Mb, but your milage may vary. Optimise for your setup, not off a script! In this case, the op-code cache meant that building pages was quick, when required.
2. Boost - static caching module. This builds pages once, then stores a static version of them on disk. Then, using some rewrite rules, it serves up the static version of a page, if it exist, else it builds the page, serves it to the user, then stores it for later use. In this case, it meant that the vast majority of the page requests were available statically. For the 48 million requests, only a hundred or so of them had to be built from scratch. The rest were already built.
3. CacheRouter - dynamic cache manager. This module lets you use various caching methods for specific parts of your site, such as menus, views content and so on - basically, if it's pulled from the database, it will cache it. In this case, I've got it configured to use a Memcache backend, which again reduces the build time for pages, as it just pulls unchanged content from memory, thanks to memcache, instead of loading it from the database.
4. Memcache - the software that handles the in-memory storage of database results. You can run it on any box with spare memory, and even quite a small cache can help. It's set to 32Mb on this server, which is plenty for the data I'm caching in it - menu entries mainly...
5. MySQL Query Cache - a nice feature of MySQL, which again, caches database results. It probably doesn't have much effect in this case, but it's cheap and easy to enable, and helps if memcache isn't running for whatever reason.
6. Deflate - Apache module that sends compressed content to clients that support it, which is pretty much any client. In this case, it meant that the textual content could compress massively to be sent out, without an extra effort (well, OK, it probably didn't do anything, really - Boost stores compressed versions itself, although I'm not sure if these are used unless the web server knows about sending compressed content anyway).

In short, everything is cached in memory if possible. Even the finished page was being held in memory, thanks to the system's in-built file-system cache, which holds files being accessed repeatedly (you don't have to do anything to benefit from this one either - pretty much all modern file systems will do this without you needing to do anything!).

There is also a trick of the HTTP protocol which can help with reducing load - requests can ask if the cache they have locally is up-to-date. In this case, because it was a text crawler gone wild, not a full browser, it didn't do this, which would have probably have meant it could handle even more requests!

So, how do you handle this much traffic? Easy: cache everything, and make sure your caching systems are tuned for your site content. Would it work for a site that got Dugg or /.ed? Probably not - the traffic patterns would be different, but a server with these abilities will handle it a lot better than one without them. In fact, any individual parts of the setup will help - anyone can use deflate, and MySQL query cache, and they don't take much to get working properly. Adding APC is pretty much a no-brainer too, although tuning it properly takes some care. Boost is massively useful, but won't work on all server setups, since it needs a few settings which not all Drupal users may have (ability to edit rewrite rules, mainly). Cacherouter is probably the last part to put in - you can use it without memcache if you're running a single site off a shared server, say. Using the APC engine it offers, you'll get a lot of the benefits of memcache, without the extra process. Even the file storage engine will help slightly, but memcache is fairly mature, and, once you've got the PHP extension set up correctly, just works. Again, though, it isn't really suited to shared server environments.