Hey,

I’m the dev / admin behind the Ludum Dare 48 hour game developer competition. This past weekend our competition grew from 350 contestants to 600 contestants thanks to @notch. Not only that but his live video blog of the event and twittering drove a TON of people over to the site just to spectate the event. This led to a ton of press coverage on. In short, 48 hour game development BECAME a spectator sport. Which is totally not something I had anticipated – here’s a completely inaccurate motivational poster I created a few years ago:

Anyway, that wasn’t the case, and this is what happened. (Short version: $8 shared hosting server burned to the ground and we got kicked off and had to find greener pastures, and then do a ton of optimizing.) I want to give a shout out to Mike Kasprzak for keeping the masses at bay and helping to endlessly test the site and hold my hand while I worked through all the mess. Here’s the lessons learned:

1. Profile site database use.

You can find out what queries are killing your site by using the WPDB Profiling plugin. This was a HUGE deal for us. We were able to see exactly what queries were destroying us. Without any debate this is the #1 tip I have, because if you don’t know what the problem is you can’t fix it!

2. Remove inessential plugins.

Not all WordPress plugins are developed with massing scaling in mind. We found by disabling all the plugins except for the essential plugins we were able to go much faster. Not only that but even efficient plugins add a CPU drain to the site just by having their PHP files loaded.

3. Clear out the sidebar.

Sidebar stuff can also be a huge drain. We just plain removed the sidebar for the competition. We’ll probably re-enable some of it eventually, but be careful, even some of the WordPress issued functions have some monster queries in them.

4. Optimize slow 3rd party plugins.

One plugin we wanted to keep using I like this is really cool, but it uses a table that is not indexed and it adds an extra query per-post on the front page of your site. I indexed the columns in the custom table it had as well as removed the unnecessary per-post query it was issuing. Now we can use that cool plugin without any notable slowdown. Here’s my bug report / patch.

5. Add in custom caching to 3rd party plugins.

Our custom competition plugin was really un-optimzed. I had written it to just work, and it worked great under low load. I used a number of techniques to reduce the load:

– I added in my own mini profiler so I could track how often certain functions were being called and how long they were taking. This helped me track down the issues to an even finer detail than I was able to with the WPDB Profiling plugin.

– I was using the get_userdata($uid) function from WordPress as many as 600 times on a page, so I cached that data with the records in my database so I would almost never have to call it, that was probably the biggest issue and was causing 600 database queries on those pages.

– I added paging to the various views so that instead of seeing all 600 entries at once, you only see 24 at a time, this really reduced load in terms of queries and images being loaded off the site in one request.

– I also added some page caching for the few pages that calculate results, this is largely a CPU save because generating those pages is pretty intensive

6. Be smart with 3rd party caching solutions.

Pre-made caching solutions for WordPress can’t save you if your site is largely driven by custom plugins and signed-in users using web apps (like our compo system), in fact those caching plugins actually can slow down the site quite a bit. They are designed more for static content blogs (which is more typical for blogs) and probably work great for those, but for us they didn’t work so hot.

… Conclusion.

I’m now quite happy with how the site is performing. We’ve reduced the load on our new VPS from 70 or so (meaning that we had about 30x as many processes wanting to use the CPU as the server could handle) down to about 0.5 which means we’ve got room to handle spikes again. I hope 🙂

-Phil

I’ve been a bit quiet the last couple weeks .. after my spree of “python->c” converter posts I got pretty busy working on the Ludum Dare 10 website.  Ludum Dare is a 48 hour game development competition.

I used WordPress as the basis for this project and I think it was a pretty good choice.  Generally anything I needed to do, there was a hook in the API to let me do it.  I was able to keep all my compo code (theme voting, trophies, rating of entries,  tag clouds, screenshot grids, security tweaks) all within a module I wrote without having to modify any of the core WordPress files 🙂

There were around 150 signups on the site and 50 people completed entries.  The theme was “Chain Reaction” which won even in the first round of voting .. and still won after the 2nd and 3rd rounds.  I spent most of the competition sitting around on IRC doing nothing and occasionally working on my game.

The one notable glitch in the compo was the announcement of the theme ceremony.  I switched the voting to closed so we could see the results and it showed several themes with almost nobody voting for them.  Turns out those were two themes which I initially had in the final round but removed (because they didn’t make the cut).  The results of a few people voting was stored in the database so they still showed up.  Anyway, that was easily fixed.

At the end of the compo it took me an hour or so to get the entry rating system set up.  I also added the ability to leave a comment along with your rating to encourage more people to leave comments.  (They could use the WordPress blog commenting system, but that would take a few extra clicks and thinking.)  This way seems much nicer.

I think the funnest feature I added to the site was the Trophy feature.  This feature lets users award each-other 64×64 pixel trophies at any time.  It’s a nice community feature because it lets people recognize cool things that people did out-side of the pre-set rating categories in the contest.  (For example, one entrant recorded a tuba solo for their game.  Although the compo has a sound category, several people felt that the tuba playing merited special tuba trophies.)

I don’t know if I’ll be running the Ludum Dare competition again or not, but I’m sure I’ll be hosting some others now that I’ve got this swell compo system written 🙂

As far as code goes, I’ve usually been a do-it-yourself kind of guy. However, I’ve been so impressed with WordPress I’ve actually used it to implement four of my sites in the last couple months. WordPress is an easy to use, smart piece of blogging software. It really seems to have just the right set of features in its default installation to be useful for most cases out of the box.

However, there comes a time when what’s given just isn’t enough. Thankfully, its got an extensive collection of plugins! Everything from blog aggregation to voting to forms to photo galleries. Not all plugins are great, but usually if you check out a few you can find one that will do what you want.

That is .. until you want something different! I might be hosting the 10th Ludum Dare compo. For that I needed some special features for collecting ratings of contestants entries, showing screenshot grids, and giving trophies to entrants.

WordPress comes with a fairly nice themes and plugins system which made it possible to add all those features to my blog without modifying the core-code of WordPress. Frequently I would implement a feature, and after learning more about WP internals, I was able to refactor it to be simpler by using more of the existing WP framework.

It wasn’t all fun and games, though, the learning curve was a bit painful for some features. A couple WordPress features (like table deltas) seemed a bit too clever (not to mention broken) for their own good. Fortunately, I was able to get away with not using those features.

The other challenge I had was when I came across a bug in WordPress. I did my best to figure out the bug, but it appears to be some strange javascripty thing which was beyond me. So I’ve reported the bug, and according to their schedule, it probably won’t be fixed for about six months. Ah well, at least it’s pretty minor.

All that said, it has been a fairly enjoyable process. I’ve been able to develop more site in less time by working with the WordPress plugin system. I have *considerably* less code to maintain, since I’m only responsible for the plugins I’ve made. Had I created this from scratch, I wouldn’t have gotten even half as far given the amount of time I invested.

This just in, the WordPress spell checker chokes on the word “with” .. weirdness!