| Navel-gazing

pwnd.

He didn’t use that term, though his message had enough unnecessary capitalization and punctuation marks that he might as well have.

I was just finishing off my shift at work on Sunday evening when I checked my email. You might think this funny, but the first indication that something was wrong was that I’d just gotten a bunch of new Twitter followers.

Figuring some witty comment of mine had been retweeted by admiring followers, I checked, and found this, followed by a few others like it. A frantic typing of my blog’s address later, and I got the message that my blog had been compromised.

Supposedly I deserved this because of thing I’d said about Islam. I find that highly unlikely. In any case, rather than try contacting this young chap through the cool hacker email address he so helpfully provided, I’d just restore the website from a backup.

Except I had to get home first. A much more anxiety-filled metro ride at 11pm on a Sunday than I had anticipated. Part of me is glad I hadn’t found out about this at the beginning of my shift, or I might have been completely useless and/or had a heart attack.

Warning: This story has a lot of technical jargon in it.

Once I got home, I did some investigating. I could still access my account on the hosting server. Files, including all images, were still there, as was another site on the same server. Eventually I narrowed it down to two things that I had lost: the custom WordPress theme (which controls how the blog looks and how it functions on a user-interface level) and all 2,663 posts as well as a few drafts. Other information like tags and settings were still in place. But, of course, the posts make the blog.

Restoring it should have been simple: restore the database from the latest backup and reinstall the theme.

You know those sentences that begin “what kind of moron…”? Well, I was the answer to a few of those, particularly “what kind of moron doesn’t back their database up on a daily basis”. I had a copy of a relatively recent stylesheet, but thanks to WordPress’s innovative in-browser theme editor, the customizations I’d made bit by bit over the years were only on the server and were now gone.

As for the posts, my most recent database backup was two months old, and that would have meant a lot of lost data, especially comments.

I spent about an hour scouring the website of my web host. But SiteGround (yeah, I know there are better providers now, but they were cheap and easy at the time) doesn’t have contact information unless you want to buy something, and their tech support system is designed to make it as hard as possible to waste their time with your silly emergencies. It was only when I found a section that offered backup restoration – for a price – that I could get any help.

The most important help came relatively quickly once I punched in my credit card number. The database was restored to a version from about 24 hours earlier, and the posts, comments and all the other database data came back.

As for finding out the vulnerability that caused this in the first place, they weren’t too helpful, offering a form-letter sales pitch about all the things they do to secure their servers, and changing a database password in case the intruder managed to get it somehow.

Rebuilding the theme took a while, and I had to repeat some steps I’d taken before, using an old page in the Internet Archive as a guide (yes, it’s been that longer than a year since I’ve had a significant redesign).

With a full backup sitting on my computer, I was still tweaking past 4am when he struck again. Same guy, different message. I don’t even remember if it was interesting.

What followed was a bizarre, surreal cat-and-mouse game where I’d reset the blog’s administrator password, only to have him reset it back again. Eventually I decided the easiest way to deal with this for the night was to lock out my WordPress installation from its own database. That put an abrupt end to it, but also made the blog inaccessible to everyone.

(To my horror, I thought that hadn’t been enough. I replaced an authentication key – a string of random characters in a text file that’s stored used for browser cookies – only to find it being rewritten back within seconds every time. It was only the next day that I realized that in my zeal for protectionism I had set permissions on this file to disallow writing from its owner, and I was ignoring the error messages that the file editor was giving me when I’d save.)

I eventually called it quits at about 6am, lying in bed with my laptop running out of battery power. I’d planned to sleep for a full eight hours, go to work and then deal with the issue on my day off. But I woke up four hours later and couldn’t get back to sleep again, despite valiant efforts. Throwing in the towel, I opened the laptop and got back to work. Rather than try working with a potentially compromised system, I started from scratch, reinstalling a fresh version of WordPress and then working on populating it with data (50MB of text, mostly in the forms of posts and comments).

Though the posts had been restored, I kept the website inaccessible and locked down as I went to work on Monday. Better to have my blog be blank for a day than have someone potentially have free reign through my database while I’m away from my computer for 8 hours.

Word seemed to spread quickly there, and I got a lot of concerned questions from coworkers and blog fans. (Thanks everyone, by the way, nice to know people care so much about this little thing.)

After I got home, I implemented a few simple security measures (nothing my readers will notice) and changed a bunch of passwords, so hopefully this won’t happen again. After reinstalling some plugins, moving the image and other data files back into their proper directories, and a few minor tweaks, it’s back to its old self again.

Since I hadn’t written any posts over that 24-hour data gap (it’s been a busy few weeks at work, sorry), all I lost was a bit of a draft post and about a dozen comments, and even those were salvaged from elsewhere (an open browser window and email notifications, respectively). If you added a comment during the day on Sunday and it hasn’t appeared, it might have been lost. So feel free to comment again.

Now, hopefully, I can get back to my life.

Well, in theory, were I to have a life to get back to, I would be doing so now. Instead, I’ll do laundry and groceries.

33 thoughts on “pwnd.”

  1. Carlos

    I feel your pain. I have spent many frantics nights like those over the years, sometimes because of my own stupidity (like deleting my entire site on the server and realizing I only had old backups, I basically scoured the server’s hard drive by hand for recoverable blocks of data) or because or server hardware and/or software issues. The idea of people finding an empty site and moving on to a competitor’s website without a second tought was frightening to me. But you have a loyal following and people are always going to come back (and you can always stay connected through Twitter). So no worries and sleep tight.

    Reply ↓
  4. Jim

    Enjoy your blog on a daily basis. Glad your back.

    You put alot of effort into your research and your articles are interesting, which is why I think you have so many followers.

    Reply ↓
  6. Soranar

    geez some people need to realize beating up your opponent is not an argument

    glad to see you back , guess stuff like that comes with the territory

    keep it up

    Reply ↓
  11. mare

    Doesn’t this WordPress hack leave a backdoor open somewhere, even if you update WP afterwards?

    He might have hacked WP before your provider restored the site from the backup.

    Ah well, time will tell.

    Oh, and people’s implementation of Islam can be evil.

    And your comments page is broken.

    Reply ↓
    1. Fagstein Post author

      This is why I completely reinstalled from scratch. Anything left in the original website has been removed and replaced, along with any potential backdoors. Even the theme was reinstalled and recustomized by hand. So unless this backdoor is in WordPress itself, or somehow hidden among the JPEGs in my uploads directories, I think it’s gone.

      Reply ↓
  15. David C.

    What a great read,from someone who was in the computer business for almost 30 years.Imagine if this blog was your livelyhood.I enjoy reading it everyday,what a shame its not a column in the newspaper you work for.
    Cheers

    Reply ↓
    1. Fagstein Post author

      Imagine if this blog was your livelyhood.

      Then I’d be paying a lot more than $10 a month for hosting it, and I’d have a team of trained monkeys performing hourly backups.

      Reply ↓
  17. Tux

    He was probably running some kind of exploit on your WordPress install, so changing the passwords didn’t do anything. You’re probably safe now that you’ve upgraded wordpress…

    Glad you didn’t suffer too much from the attack.

    Also, that’ll teach you not to read error messages ;)

    Reply ↓
  18. Shawn

    Thank the good lord we haven’t lost the “Howard Galganov is still an idiot” thread, else this would have been a dark day indeed for thoughtful dialogue in this country.

    Reply ↓
  19. Karine

    I have you in my blog feeds but sometimes I go straight to your blog and so I saw your blank page. For a second I thought it was the NoScript add-on to my Firefox browser that put you on it’s black list. Glad to see all ended well. Now you can get back to commenting about how in the Québécor world, there seems to be no mention about the Labonté scandal originating from RueFrontenac.com. Oh and do you know what happened to John Moore’s bit on Virgin? I feel with him gone there’s nothing to balance out the lack of depth of the on air personalities. I’m still from the time they said they didn’t know what gentrification meant…

    Reply ↓
    1. Fagstein Post author

      I saw it, and upgraded as soon as it was released. I don’t think it’s directly related to the vulnerability that was exploited here though.

      Reply ↓
  22. James Lawlor

    Just think how horrible it would be to lose all the comments to “Howard Galganov is still an idiot” post. Your (and your readers) attempt to break the World Record would be for naught!

    Reply ↓

  25. Pingback: Welcome to my new home – Fagstein

Leave a Reply

Your email address will not be published. Required fields are marked *