×
  • remind me tomorrow
  • remind me next week
  • never remind me
Subscribe to the ANN Newsletter • Wake up every Sunday to a curated list of ANN's most interesting posts of the week. read more

Forum - View topic
Can we get a reason why ANN was down?


Goto page 1, 2  Next

Anime News Network Forum Index -> Site-related -> ANN Feedback
View previous topic :: View next topic  
Author Message
PetrifiedJello



Joined: 11 Mar 2009
Posts: 3782
PostPosted: Tue Aug 16, 2011 6:19 am Reply with quote
This is a request, not a demand. If ANN feels its business need not be public, I'm okay with this.

However, I didn't make a single phone call to the companies listed on the splash page while ANN was down. I had no idea why I would be calling.

Think about it from our position, please, when dealing with this request. The history of "down time" seems to be the result of a technical failure, which placed significant load on the remain server(s).

Technical issues aren't pleasant to experience, but it looked to me as though people were patient while ANN addressed the issue.

Then, without warning, was a splash page from ANN asking us to call two companies and "firmly, but politely" ask for ANN's return.

I made no phone calls. To me, this was a technical issue which bore no reason why companies had to be bothered while it was being addressed. Harassing companies wouldn't make the situation change any faster.

The only other reason for the request would be related to the business, and there's no way I would be making "blind calls" to fix... contract dispute? Failure to resolve the technical issue in a timely manner? Lack of any response?

We were left to figure this out for ourselves, which seemed very unprofessional to me. I don't mean to be offensive here, but why did ANN put this burden on us without any justification for it (assuming a non-technical issue)?

Please use this feedback in the event future issues arise. I'd be very happy to assist ANN but I will not do so at the expense at being anything but "firm, and polite" as viewed from the businesses I'm calling.

Thank you for reading and have a great day.
-Robert
(PetrifiedJello)
Back to top
View user's profile Send private message
dtm42



Joined: 05 Feb 2008
Posts: 14084
Location: currently stalking my waifu
PostPosted: Tue Aug 16, 2011 7:31 am Reply with quote
Someone correct me if I'm wrong, but I thought ANN had already explained this, at least partially. As I recall, they wrote on Facebook that the physical servers were being moved by whomever in Texas runs/owns them, and ANN was somehow not informed/kept in the loop.
Back to top
View user's profile Send private message My Anime
ikillchicken



Joined: 12 Feb 2007
Posts: 7272
Location: Vancouver
PostPosted: Tue Aug 16, 2011 7:59 am Reply with quote
I really don't think Chris (or whoever at ANN decided to post that number) was attempting to use everyone as leverage in some kind of contract dispute PJ. In fact, I also don't think that Chris would ask us to pester this company if he thought they were already doing everything they could to resolve this issue. As long as I've been coming here, ANN has proven to be a super classy, super well run website. I'd say that it's pretty safe to assume, even if it's not explicitly stated, that if ANN asks users to call this company then it's because they are most definitely not doing everything they can to fix the issue.

(That said. I am rather curious so if ANN wants to elaborate on what happened I'd love to hear it).
Back to top
View user's profile Send private message My Anime
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7580
Location: Wales
PostPosted: Tue Aug 16, 2011 8:06 am Reply with quote
It was stated on Facebook while the site was down.

Quote:
Anime News Network is currently offline. Our network host is doing some "emergency maintenance" and, as a result, the site is offline. We are currently outside of the maintenance window we were informed of, so we do not have an estimated time for the site to be back online.

They were informed of downtime due to "emergency maintenance" (i.e. unlike planned maintenance there was not much if any warning). They were already outside of window of downtime they had been informed of when this message was posted on the 11th.

Quote:
WE ARE CURRENTLY DOWN DUE TO AN ISSUE WITH OUR CO-LOCATION SERVICE.
(The service that houses our servers and connects them to the internet.)

ANN’s CEO Chris Macdonald is flying from Japan to Texas right now
to get this resolved, but we’d like to be back up before he arrives.

YOU CAN HELP!
Call North Texas Connect at 877-905-8989 (press 2)
and Aeprio at 866-304-6930. Be polite but firm, and tell them to get
Anime News Network fixed!!

We love you guys. Thanks for your patience and your help.

Around 12 hours after the problem started, Chris resolved to cut short his trip to Japan and spend $4000 on a last minute plane ticket to get the servers moved to another facility as there was still no end in sight.

I don't know what uptime guarantee their hosts specify, but 36 hours of downtime for a company who's business depends entirely on their site is clearly unacceptable. Actually I do - North Texas Connect's SLA guarantees 100% availability. Since they have redundant connections (hopefully on opposite sides of the facility to avoid roadwork accidents) this sort of thing shouldn't happen unless they seriously screwed something up internally. Oh, but they have a "Force Majeure" get out clause with regards to maintenance (planned or emergency) and upgrades - so if they screw something up during a badly planned upgrade or "maintenance" they have no obligation to even credit you so the SLA means nothing.
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
Teriyaki Terrier



Joined: 26 Mar 2008
Posts: 5689
PostPosted: Tue Aug 16, 2011 8:35 am Reply with quote
As a subscriber, when ANN posted the message about technical difficulties that was good enough for me because Chirs (Tempest) himself (among others I imagine) actively working on this, so truth be told specific details would be welcomed, but since the issue is fixed I myself am fine with the message.

Here the link for the message I am talking about. animenewsnetwork.com/ann-is-back-but-might-be-a-little-shaky

But I am going to be very blunt here if this happened to anyone besides Anime News Network, I probably would want a clear explanation. That is party because seldom is any other website (except maybe a very select few) are nearly as willing to resolve matters (such as technical difficulties) as fast as ANN and partly also because ANN seldom ever has issues such as these for example.

In fact, in the years I've been going here, this incident really was the first time since I made my account that I had any issues with ANN.


On a different note, I really would to thank Tempest and everyone else at ANN for working so hard to address and fix this issue. It's hard work maintaining a website this size and I truly appreciate all that has been done.

Once again, thanks again. This is my two cents on the matter.
Back to top
View user's profile Send private message
RoverTX



Joined: 17 Dec 2008
Posts: 424
PostPosted: Tue Aug 16, 2011 9:21 am Reply with quote
I work part time for a web company that creates websites and handles hosting for websites, and let me say a Colo being down for the period of time ANN's was is highly unusual. Lets just say that the colo we use was down for only about 36 hours after being hit by anno, so their colo better have a pretty damn good reason. So what happened to ANN, lets just say, is just under "Act of God" and just above "Terrible Act of Nature".

The only way you could avoid something like that is have a mirror version of the site ready to go in some type of cloud solution, either through amazon's servers or a similar service, which is not exactly cheap sense your basically having one server just sitting their most of the time. And even then cloud solution aren't fail proof either, we have had I think up to 12 hours down time with amazon once for some weird reason.

So basically stuff like this happens, and hey it isn't like ANN stopped posting articles, they just did it through facebook, so its not like there was even that big of an interruptions in coverage.
Back to top
View user's profile Send private message
Blood-
Bargain Hunter



Joined: 07 Mar 2009
Posts: 23769
PostPosted: Tue Aug 16, 2011 10:16 am Reply with quote
Like a lot of people, I had full faith that the ANN braintrust was doing all it could at its end to "encourage" a speedy resolution to the problem. However, I also agree with PJ that, in cases where a site is asking its customers to contact a third party about a specific problem, the site should give as much detail as possible to help ensure that anyone who chooses to make a call knows what he or she is talking about.
Back to top
View user's profile Send private message My Anime My Manga
PetrifiedJello



Joined: 11 Mar 2009
Posts: 3782
PostPosted: Tue Aug 16, 2011 10:18 am Reply with quote
@Shiroi Hane:
Thanks for the insight and confirming my feeling the message was more than a technical one.

I guess my title's a bit misleading, but given the space, I didn't know how else to add "Can we get a reason why ANN was down and requested the message we call the datacenter?"
Razz

***

@ikillchicken:
Your inability to comprehend what you read is a problem you need to correct and I strongly suggest you spend less time at this site and more time at this one.

***

@Teriyaki Terrier:
Your request of another site is "me being you now", if you will. Razz

My issue wasn't the downtime. It happens (though unusual for this amount of time). My issue was the "call to arms", so to speak, without a justifiable reason for us to take up muskets and dead racoons.

I've had experience with such downtime and it was later discovered the datacenter was down because it had flooded due to a burst water pipe. Their phone lines were also affected, leaving its customers clueless. Of course, after the fact is when one goes "Oh, perhaps I shouldn't have been so angry" once the information is known.

Of course, the context would be very different if the business(es) were indifferent to resolving the issue, but this doesn't make sense to me (due to my job) as this would be suicide for a datacenter.

***

I want to stress I'm not angry at ANN, its staff, or even the datacenter hosting the serves (should I be?). I just believe a bit more insight to why we're calling to be very helpful to me.

Hope this clears the confusion. Smile
Back to top
View user's profile Send private message
Tuor_of_Gondolin



Joined: 20 Apr 2009
Posts: 3524
Location: Bellevue, WA
PostPosted: Tue Aug 16, 2011 1:58 pm Reply with quote
I'd be interested in knowing the specific reason for the site being down.

From the various things that occurred, my impression was that what happened was not a simple unplanned maintenance issue. If that is all it was, then why ask us to call? And why cut short your trip to fly back from *Japan* unless you felt that your presence would accomplish something. This implies to me something other than a purely maintenace-related issue.

But that's all: it just implies, nothing more. The actual reasons may turn out to be very innocuous, or they may not. That's why I'm curious. However, my curiosity isn't very strong as the site is now back up and everything seems stable again. If I never learn the specifics, I'm okay with that, but given the option I'd rather know what happened.
Back to top
View user's profile Send private message Send e-mail My Anime My Manga
RoverTX



Joined: 17 Dec 2008
Posts: 424
PostPosted: Tue Aug 16, 2011 4:54 pm Reply with quote
PetrifiedJello wrote:
My issue was the "call to arms", so to speak, without a justifiable reason for us to take up muskets and dead racoons.

I've had experience with such downtime and it was later discovered the datacenter was down because it had flooded due to a burst water pipe. Their phone lines were also affected, leaving its customers clueless. Of course, after the fact is when one goes "Oh, perhaps I shouldn't have been so angry" once the information is known.

Of course, the context would be very different if the business(es) were indifferent to resolving the issue, but this doesn't make sense to me (due to my job) as this would be suicide for a datacenter.


A colo being down for 36 hours is pretty terrible and extraordinary. Again the one my company uses was down for 36 hours (not down down, just slowed really, and sometimes almost gone) after Anonymous attacked a group of servers someone had running there. Anything less then that or some kind of natural disaster and I can tell you ANN was well within its right to sick its reader on them, especially if they where just brushing them off, while Zack and Chris where loosing cash........

Shit for example, my boss is one of the most laid back people I have ever worked with, but one of the cloud solutions we used had a random 3 hour downtime one day, and lets just say he was on the phone hopping mad......

As for spending the money to rush back to the states, trust me stopping a day of down time by getting a $4,000 ticket and moving the servers yourself physically can be more then worth it. Its not just the revenue lost that day, but its also pissed off visitors who might not return, possible refunds with their video systems, and its hard to get people to take you seriously in a business meeting about your website when its down, which of course just feeds into your general reputation. I have had to move servers around from colo to colo before because of very similar circumstances(just with out the $4,000 ticket), and I can tell you it was worth it.


Last edited by RoverTX on Tue Aug 16, 2011 4:54 pm; edited 1 time in total
Back to top
View user's profile Send private message
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7580
Location: Wales
PostPosted: Tue Aug 16, 2011 4:54 pm Reply with quote
Tuor_of_Gondolin wrote:
And why cut short your trip to fly back from *Japan* unless you felt that your presence would accomplish something. This implies to me something other than a purely maintenace-related issue.

I don't know if he actually did in the end, but Chris' intention in flying over was to physically move the servers to another location (something which can't easily be delegated for both technical and administrative reasons) - this would obviously solve the connectivity problem, and I also assume the relationship with the current hosts had rather soured anyway.

The "call to arms" is something that's not my place to comment on.
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
samuelp
Industry Insider


Joined: 25 Nov 2007
Posts: 2231
Location: San Antonio, USA
PostPosted: Tue Aug 16, 2011 7:12 pm Reply with quote
Colo's don't go down for 36 hours unless there's a damn good reason.

Some of those reasons include natural disasters (clearly not in this case), internal employee sabotage, armed robbery, fire/explosions/flooding of one's own doing, or internal business structure conflict (aka "the boss ran out with all the money").

I highly doubt we'll even know exactly what happened to this colo publicly (and I bet ANN doesn't know the full story either), but ANN has their servers back at least now...
Back to top
View user's profile Send private message Visit poster's website
Zac
ANN Executive Editor


Joined: 05 Jan 2002
Posts: 7912
Location: Anime News Network Technodrome
PostPosted: Tue Aug 16, 2011 8:39 pm Reply with quote
We'll probably talk about this on the podcast this week. I'm not sure how much of the full story they really want or feel the need to tell - especially since it's all behind us now and there's no point in dwelling on it - but we'll probably at least discuss it.
Back to top
View user's profile Send private message Visit poster's website My Anime
ikillchicken



Joined: 12 Feb 2007
Posts: 7272
Location: Vancouver
PostPosted: Tue Aug 16, 2011 10:11 pm Reply with quote
PetrifiedJello wrote:
@ikillchicken:
Your inability to comprehend what you read is a problem you need to correct and I strongly suggest you spend less time at this site and more time at this one.


Uh...okay. That seems like a really needlessly vile and obnoxious response. I don't think I said anything whatsoever to warrant you getting all pissy. But I guess no matter what you're just gonna throw a little tantrum. Well, knock yourself out.
Back to top
View user's profile Send private message My Anime
Rukiia



Joined: 30 Aug 2010
Posts: 1897
Location: British Columbia, Canada
PostPosted: Wed Aug 17, 2011 12:23 am Reply with quote
I gotta say....why do some of you guys have to know the exact details on what happened? The site is back up and running and that is all I, and many others who use the site, care about. Like Zac said, why dwell on it? Be happy that the issue has been fixed and everything has gone back to normal.

But this is only my opinion on the topic.

*waits for the horde of angry post(s) to come flying her way*
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
Display posts from previous:   
Reply to topic    Anime News Network Forum Index -> Site-related -> ANN Feedback All times are GMT - 5 Hours
Goto page 1, 2  Next
Page 1 of 2

 


Powered by phpBB © 2001, 2005 phpBB Group