Hey,

I’m the dev / admin behind the Ludum Dare 48 hour game developer competition. This past weekend our competition grew from 350 contestants to 600 contestants thanks to @notch. Not only that but his live video blog of the event and twittering drove a TON of people over to the site just to spectate the event. This led to a ton of press coverage on. In short, 48 hour game development BECAME a spectator sport. Which is totally not something I had anticipated – here’s a completely inaccurate motivational poster I created a few years ago:

Anyway, that wasn’t the case, and this is what happened. (Short version: $8 shared hosting server burned to the ground and we got kicked off and had to find greener pastures, and then do a ton of optimizing.) I want to give a shout out to Mike Kasprzak for keeping the masses at bay and helping to endlessly test the site and hold my hand while I worked through all the mess. Here’s the lessons learned:

1. Profile site database use.

You can find out what queries are killing your site by using the WPDB Profiling plugin. This was a HUGE deal for us. We were able to see exactly what queries were destroying us. Without any debate this is the #1 tip I have, because if you don’t know what the problem is you can’t fix it!

2. Remove inessential plugins.

Not all WordPress plugins are developed with massing scaling in mind. We found by disabling all the plugins except for the essential plugins we were able to go much faster. Not only that but even efficient plugins add a CPU drain to the site just by having their PHP files loaded.

3. Clear out the sidebar.

Sidebar stuff can also be a huge drain. We just plain removed the sidebar for the competition. We’ll probably re-enable some of it eventually, but be careful, even some of the WordPress issued functions have some monster queries in them.

4. Optimize slow 3rd party plugins.

One plugin we wanted to keep using I like this is really cool, but it uses a table that is not indexed and it adds an extra query per-post on the front page of your site. I indexed the columns in the custom table it had as well as removed the unnecessary per-post query it was issuing. Now we can use that cool plugin without any notable slowdown. Here’s my bug report / patch.

5. Add in custom caching to 3rd party plugins.

Our custom competition plugin was really un-optimzed. I had written it to just work, and it worked great under low load. I used a number of techniques to reduce the load:

– I added in my own mini profiler so I could track how often certain functions were being called and how long they were taking. This helped me track down the issues to an even finer detail than I was able to with the WPDB Profiling plugin.

– I was using the get_userdata($uid) function from WordPress as many as 600 times on a page, so I cached that data with the records in my database so I would almost never have to call it, that was probably the biggest issue and was causing 600 database queries on those pages.

– I added paging to the various views so that instead of seeing all 600 entries at once, you only see 24 at a time, this really reduced load in terms of queries and images being loaded off the site in one request.

– I also added some page caching for the few pages that calculate results, this is largely a CPU save because generating those pages is pretty intensive

6. Be smart with 3rd party caching solutions.

Pre-made caching solutions for WordPress can’t save you if your site is largely driven by custom plugins and signed-in users using web apps (like our compo system), in fact those caching plugins actually can slow down the site quite a bit. They are designed more for static content blogs (which is more typical for blogs) and probably work great for those, but for us they didn’t work so hot.

… Conclusion.

I’m now quite happy with how the site is performing. We’ve reduced the load on our new VPS from 70 or so (meaning that we had about 30x as many processes wanting to use the CPU as the server could handle) down to about 0.5 which means we’ve got room to handle spikes again. I hope 🙂

-Phil

This entry was posted on Thursday, August 25th, 2011 at 10:46 am and is filed under crazy, wordpress. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.