A letter from the editor
This is not an editorial, editorials contain an opinion of some sort and given that this is Anime News Network an editorial should contain something about Anime. What this is, is a brief explanation of what happened to ANN for the past 2 days, how it happened and just how bad it is.
It's bad, but it could be worse.
On Wednesday, May 29th the server that ANN was hosted on crashed. It turns out that our host, Liquid Web was using a workstation class machine, with a single IDE drive as a server (a big IT no-no). When the motherboard went it decided to take the hard drive with it (much less likely to happen in a proper server class machine) so LW lost all our data.
That shouldn't be a problem; they should have a backup from the night before. They didn't. Turns out that we had recently moved ANN onto a new “vserver”, something Liquid Web is trying out, vservers aren't quite dedicated servers, but they're “more dedicated” that the usual shared servers that we had previously been on. One problem, LW hadn't implemented a nightly backup on the vserver yet.
That shouldn't be that much of a problem, we only moved onto the vserver on the 27th, so LW should have a backup from the morning of the 27th, right?
Nope, apparently Liquid Web overwrites their nightly backups and only keeps a bi-weekly backup. A very big IT no-no… even a paper MCSE (the lowest class of IT professional there is, basically an amateur with a piece of paper) knows better than that. A fairly common backup rotation involves nightly back up (either full, differential or Incremental) with a pulled full backup every so often (say bi-weekly). The thing is, for the period since your last pulled backup, you always keep every backup, even if you've been doing full backups every night, because it often takes a day or two to realize that some piece of data has been damaged. Well, as it turns out, LW overwrites their nightly backup every night. Like I said, amateurs know better than that.
So when was their last full, kept backup? May 21st. In other words ANN has lost 10 days worth of data. 10 days of news, 10 days of webboard postings, 10 days of updates to the various databases (such as the release database), a number of unposted reviews and 10 days of work on a major, as of yet unpublished, project. How many hours of work are we talking about? Somewhere between 100 and 200 man hours of work, maybe more….
Yeah, I'm kinda annoyed about this. Now I have to take 2 days off from my job (as a network engineer) to fix ANN, 2 days off that I can't really afford right now because I'm very busy at work. I'm also kind of mad at myself, because as a network engineer, one thing I've learned over the years is that, professionally speaking, other people can't be trusted. We should have had our own nightly backup on ANN, but we didn't. As soon as the data is restored we will set one up.
And what about Liquid Web? While we're obviously annoyed with them, they have proven otherwise to be an excellent webhost, so we will be staying with LW, but we're going to strongly recommend they set up a proper backup rotation.
So, for the next couple of days we will be working our collective a**es off trying to recover and re-create all that data. Since we are Anime News Network our first priority will of course be the news, but the news will actually be one of the easiest items to re-do.
In the meantime, if you happen to have a cached or saved copy of anything from ANN during the past 10 days, please send it to editor (at anim8info.com) , we would be very grateful.
Christopher Macdonald
Co-Editor in Chief
Anime News Network
PS: We've also lost all e-mail sent to us since May 27th, so if you sent us a letter, a question, a comment, a press release, a tip or anything, please send it again, but at editor (at anim8info.com). Once the new DNS entry fully propagates (in a day or two), the @animenewsnetwork.com e-mail addresses will work again.
PPS: I apologize for the lack of editing and refinement in this letter, but right now I have other priorities.
It's bad, but it could be worse.
On Wednesday, May 29th the server that ANN was hosted on crashed. It turns out that our host, Liquid Web was using a workstation class machine, with a single IDE drive as a server (a big IT no-no). When the motherboard went it decided to take the hard drive with it (much less likely to happen in a proper server class machine) so LW lost all our data.
That shouldn't be a problem; they should have a backup from the night before. They didn't. Turns out that we had recently moved ANN onto a new “vserver”, something Liquid Web is trying out, vservers aren't quite dedicated servers, but they're “more dedicated” that the usual shared servers that we had previously been on. One problem, LW hadn't implemented a nightly backup on the vserver yet.
That shouldn't be that much of a problem, we only moved onto the vserver on the 27th, so LW should have a backup from the morning of the 27th, right?
Nope, apparently Liquid Web overwrites their nightly backups and only keeps a bi-weekly backup. A very big IT no-no… even a paper MCSE (the lowest class of IT professional there is, basically an amateur with a piece of paper) knows better than that. A fairly common backup rotation involves nightly back up (either full, differential or Incremental) with a pulled full backup every so often (say bi-weekly). The thing is, for the period since your last pulled backup, you always keep every backup, even if you've been doing full backups every night, because it often takes a day or two to realize that some piece of data has been damaged. Well, as it turns out, LW overwrites their nightly backup every night. Like I said, amateurs know better than that.
So when was their last full, kept backup? May 21st. In other words ANN has lost 10 days worth of data. 10 days of news, 10 days of webboard postings, 10 days of updates to the various databases (such as the release database), a number of unposted reviews and 10 days of work on a major, as of yet unpublished, project. How many hours of work are we talking about? Somewhere between 100 and 200 man hours of work, maybe more….
Yeah, I'm kinda annoyed about this. Now I have to take 2 days off from my job (as a network engineer) to fix ANN, 2 days off that I can't really afford right now because I'm very busy at work. I'm also kind of mad at myself, because as a network engineer, one thing I've learned over the years is that, professionally speaking, other people can't be trusted. We should have had our own nightly backup on ANN, but we didn't. As soon as the data is restored we will set one up.
And what about Liquid Web? While we're obviously annoyed with them, they have proven otherwise to be an excellent webhost, so we will be staying with LW, but we're going to strongly recommend they set up a proper backup rotation.
So, for the next couple of days we will be working our collective a**es off trying to recover and re-create all that data. Since we are Anime News Network our first priority will of course be the news, but the news will actually be one of the easiest items to re-do.
In the meantime, if you happen to have a cached or saved copy of anything from ANN during the past 10 days, please send it to editor (at anim8info.com) , we would be very grateful.
Christopher Macdonald
Co-Editor in Chief
Anime News Network
PS: We've also lost all e-mail sent to us since May 27th, so if you sent us a letter, a question, a comment, a press release, a tip or anything, please send it again, but at editor (at anim8info.com). Once the new DNS entry fully propagates (in a day or two), the @animenewsnetwork.com e-mail addresses will work again.
PPS: I apologize for the lack of editing and refinement in this letter, but right now I have other priorities.
bookmark/share with:
