Last night, we attempted to upgrade the OpenAds installation of one of our clients. (OpenAds is a free ad-serving tool.)
We’d done this upgrade before for other websites, and for this client, too. This time, we were going from OpenAds 2.4.0 to 2.4.3—to close some security holes and maybe get a little bug or two fixed.
Doing it late at night means less disruption to the site’s visitors, and no disruption to the site’s editors. It also means you’re tired and you want to go to bed sooner rather than later, but hey, that’s part of the job.
Before you do an upgrade, it’s always a good idea to make a backup. In this case, a backup of the database as well as a back up all the program’s files were both necessary.
I uploaded all the new OpenAds files, which ended up taking about an hour because I was kind of dense and because OpenAds is based on a lot more files than I initially thought. (Tangent details: OpenAds ships in a ZIP archive, and our web host’s control panel can only unpack GZIPed archives. I could have unzipped and gzipped it, and saved myself a bunch of time. Ah well.)
I ran the upgrade script, which took about 5 minutes, and when I went to see how the upgrade went, it turns out that all the zones for the ads were gone. Doh! This is a bad thing—it means that while the upgrade completed, it failed in some important way along the way, which is truly hard to untangle. When this happens, the only thing you can really do is downgrade back to the way it was before you upgraded. Which means you need an exact backup of everything.
Luckily, I had in fact made a full database and file backup before I started. (Did you like that dramatic tension?) Unhappily, the database backup was a backup of ALL data in the database (articles, mailing lists, forums, and ads). Because of this huge amount of data, restoring the database took more than 2 hours, and in fact may not have run correctly. I could tell this, because after the restore, EVERY database-related service was down. Oh no.
Some more troubleshooting was necessary, and by the time the clients site was up and running, it was 7:30 a.m., and I’m still not sure that everything is perfect.
The moral of the story: Even when you do everything right, things can go wrong. Lessons can be learned from every failure. In this case, I learned that a) it’s worth it to think of efficient ways to upload, and b) it’s worth it to make backups of just specific parts of the database, in case a restore is needed. And the next time someone looks at me funny when I tell them to backup before doing a routine update, I’ll point them here.
Comments