We Don’t Want to Leave You in the Dark

FacebookTwitterGoogle+LinkedInDiggEmail

…like our night shift system administrators were on March 30th.

I should probably start this post by introducing you to our VP of Systems Operations, Rick. This is a person who is not only passionate about his job, but also highly intelligent and creative.

OK, enough of the positive. He’s a little crazy. His insanity does happen to be almost wholly attached to the safety of the data center (he’s a really cool guy otherwise), but that insanity is there and it is something worth poking fun at.

Super, High-Tech Tea and Coffee Station

Super-High-Tech Tea and Coffee Station

Danger: Tea and Coffee May Cost You More Here Than it Does at Starbucks

For example, when our team decided to create a coffee and tea station in the kitchen (it’s not even as elaborate as I just made it sound…see photo), Rick immediately felt the need to establish regulations. We are not allowed to leave the “station” until whatever process we initiated is complete, and then when it is complete, we must unplug every surrounding appliance. If we use the microwave, we must watch it like a hawk—watch it like that microwave could steal our livelihood at any moment. You know why? Because it can. At least, that’s what Rick thinks. Any of these appliances can start a fire, which could spread if our fire-suppression system doesn’t kick in, which could enable that fire to somehow reach the servers, which could melt our customer data, which could make our customers very angry with us, which would make us very angry with ourselves, which could sink our business.

All for a plate of bland, re-heated Chinese food.

If Rick is this way about appliances that lie hundreds of yards away from the actual servers, imagine how he careful he is about the servers themselves. Our data center is about as redundant as they come—millions of dollars went into making it this way.

Well, in the middle of the night on March the 30th, a power line went down in our corporate park. All buildings in the complex (from one end of Dividend Dr to the other) lost power. No problem, right? We have backup generators, backup batteries…we even have the energy Rick’s undying concern for the data center would provide if we could only find a way to harness it. Unfortunately, one branch of our backup setup failed (namely, a generator) and a good percentage of the machinery we need for operations was left powerless (namely, our network). Fortunately, our network was back up fairly rapidly (within minutes) and it didn’t take long for the servers to catch up.

For those of you that experienced outages, however short, we are unbelievably sorry. We were able to quickly return to status quo, but many of our customers still experienced downtime and we know that this is unacceptable. We are really, really sorry.

The issue that we had with the power loss is being investigated with the utmost priority because we don’t want anything like this happening again… and neither does Rick. I came late onto the scene (at about 2 AM, when the power had been long-restored and the servers were coming back up from their temporary, power-starvation induced sleep), but I knew Rick was still inside because his car was parked parallel to the building, nearly on the sidewalk entryway and I could still hear the echo of his tires screeching against pavement. When I got inside, the root of the problem had been taken care of, but we had another problem. Our main System Administrator/Network Engineer had clearly arrived straight out of bed, wearing MC Hammer pants… and he was doing so without shame.

While this event that took place never should have happened, we are going through great pains to make sure that it doesn’t happen again. (The downtime, that is. We haven’t yet decided what to do about those pants.) We, again, are terribly sorry for any downtime that was caused by this downed line, though we were fortunate that it happened at such an odd hour of the night. If we find out that the line went down because it was hit and knocked over by a clumsy driver, we’ll make sure to find and then glare at them on behalf of all of us.

We’re sorry, we’re sorry, we’re sorry! (I can be redundant too!)

Thanks for reading!

Michelle Schroeder

P.S. If you want to look like one of our very own sysadmins, you can get your own set of updated, metallic MC Hammer pants for one payment of only $19.95 from this website. Oh wait, I’m sorry. I put the decimal where I thought it belonged, not where it actually was… Turns out that the net-a-porter site has nudged the price into the “can’t touch this” range: $1,995.00.

That site will even suggest what to pair with them, though I personally think this other site does a better job in helping you achieve the whole “MC Hammer” package: http://www.ehow.com/how_2058695_dress-like-mc-hammer.html.

FacebookTwitterGoogle+LinkedInDiggEmail

33 Comments to "We Don’t Want to Leave You in the Dark"

  1. Those were not MC Hammer pants… they were actually my jammies. I got called at night, and I was told it was an emergency, so I didn’t wait to get dressed. I jumped in the car, and I showed up at the data center, to fix the issues.

  2. Lauren -

    Suuure, Tibi. You know you are up nights working on your Hammer-time moves.

  3. Michelle, your redundant apologies is well and good however this is of serious concern as IX boasts about its datacenter as one of the best.

    For a customer as myself, we look at things like backup power, fire-suppression and so forth which IX has in place, however what good does this do when they fail themselves.

    I hope not much harm was done to anyone. It is also good to know that IX has employees who are genuine with their work. Sort of reassures us.

  4. Thanks for your comment, Ritesh! To give you more comforting information, this is the first time anything like this has happened since we moved our data center to Columbus, OH. We’ve also shut the power to the data center down four times since this incident and each time, our backup system responded without fail. We will keep investigating, however until we’re satisfied that this will never repeat itself. I agree that this was a rather serious event.

  5. Hi all of you moles that have to work at night. One bit of light here. When you are working nightshift like right now, we are up and running the early morning stuff. It is now 08h40 in South Africa and the sun is mup and the leaves are falling getting ready for them colder months.

    One question I would like to know. I have experienced that quite regularly all my websites seem to go down around this time of the day. Does that have anything to do with maintenance on your side during the night shift?

    You must bear in mind, the early morning session here in Africa (07h30 to 09h30 CAT which is GMT+2) is crucial. That is when all the users fire up and use the websites intensely. After that it is just normal daily traffic.

    If the downtimes are related to maintenance, could the “African” time zone stuff not be run around midnight your time.

    The reason why I have a suspicion that the downtimes are maintenance related is that all the sites seem to have exactly the same time being down as well as the period they are down.

    Your input would be greatly appreciated, even if it is a bit off this point.

  6. Hi Marcel,

    It is possible that some periodic tasks that we run at night on the servers could affect your business hours performance if you are in another time zone. The only real remedy to that will be a new hosting infrastructure that we’re currently working on deploying later this year. That will make all these issues disappear. We’ll show you more on that here shortly.

    In the meantime, I’ve asked Tibi to take a look as to what may be running during this time period, because your sites should definitely not be down, even while we run periodic tasks.

  7. Marcel,

    It is never ok for your website to go down. We indeed have maintenance windows, but they are usually very short (5-10 minutes), and do not affect customers on multiple servers. And affected should receive an email informing them about the downtime. Therefore, if your sites are going down, please open a ticket with details, and let me look into it (please ask for my attention in the ticket, so whoever picks it first will add me to the ticket)

    Thank you.

  8. Michael Parenteau -

    Those Pants!!!!! If they came with sunglasses, I might buy them.

  9. there’s this one episode in the “NUMB3RS” TV series when terrorists are hitting power stations in the big city in an attempt to cascade a huge grid breakdown.

    I’m just saying, maybe you’ve been hit by a similar plot?

    *innocent smile from an otherwise happy customer who happened to sleep when the power was down so he didn’t notice thus never gave hell to support center*

  10. Yeah I was on one of my sites whenever the outage happened. I knew something really bad had happened when I noticed the IX web corporate site down too.

    When this happened my databases were rolled back to some version of what they were many months ago last year, but thankfully it was all restored perfectly within a 2 hour time window. If I had been asleep at the time, I wouldn’t have even noticed it.

    This is the second database failure since I’ve been a customer of IX, but thankfully it was resolved quickly thanks to you hardworking folk over there. Lately I’ve been having MySQL access time issues periodically on my database driven websites (mostly WordPress) – I talked to Lauren about it and she said to report any slowness I experience in a special support ticket she opened up for me.

    Since my web server is great performance-wise, all we need to do is smooth out these MySQL hiccups and I’ll be singing IX’s praises from the rooftops :)

  11. Been a customer of IX for several years now, this is great addtion.
    Thank IX :>

    Chris

  12. Good to know that you guys are working for your customers, however I have been having pretty lame experience lately with hosting servers as one or few of my database driven sites go blank every now and then, I have to get in touch support to get it fixed and so on..

    Customer service is always good but what about quality of hosting service?

    • Hey Jay,

      You may have noticed that a ticket was put in for you concerning a query of yours that was causing a server overload. Kenneth sent you a copy of that query that caused the overload earlier today. All you have to do is optimize it (or remove it) in order for your database issues to be resolved. Just update that ticket (999187) that was opened for you to get things started and if you have any more issues, give someone in support a call.

      Thanks for the comment!

  13. You know, I think that this was the best idea you guys have had in a very long time. If you think about it, there is nothing better than reaching out to your customers in a way that allows them the freedom to express their views.

    I wonder if you could think about expanding this into more of an open forum. I notice (and mostly my fault) that this one blog has ran away as a rant board.

    Just thinking it would be cool for everyone to have a place just like this that is open and gives your customers the freedom to rant in public (with other Ix customers and staff). I just think it would give IX a better idea of how the customers feel and can make better judgments on how to go about fixing those issues.

    Just a thought, Thanks again for all that you have done.

    • Hi Blaine,

      You should know that we do have a forum in the works and it will be rant-ready/blast compatible. We do hope that it will serve a more “customers helping customers” kind of purpose rather than an “Argh @ IX” type, but that will of course be up to you guys.

      Thanks for the suggestion! I’ll be sure to let you know when it’s available.

  14. Hey guys/girls… I have to come in and state my facts as a customer.. I’ve been with IX exclusively since I started online back in 1998… I’ve been with IX since it’s arrival on the net in 1999.

    I wouldn’t host my sites anywhere else!

    I have had several issues that I’ve created myself that have been solved by the tech crew and a couple of issues that were not my fault that were corrected within literally minutes of logging onto web support.

    I have to say you guys rock and I love being a customer!

    Have fun and stay safe… don’t be calling my buddies out in the middle of the night!

    Firewoman!

    (yes I’m a firefighter!)

    • Thanks so much for your comment!

      By the way, with the extreme anti-fire regulations we have in place for our data center, hopefully we’ll never have to call you or your firefighting buddies. That may not be so true for my house, however, as I haven’t fully mastered the art of cooking (or the art of cooking without enacting some kind of hazard- be it fire or food-poisoning). :)

  15. A bit unhappy you’re doing away with FP Extensions especially Forms and the ease of FP file loading to and from Expression Web. Any suggestions to ease the pain??

  16. Graig Thompson -

    To Ian R:
    Come on man, the FPSE no longer supported by microsoft – why IX should support it?
    Well, I am using EW without any.. Go to LiveChat – they’ll instruct you.

    Nice service, thanks!

  17. Hi,
    I’m really happy to see that kind of communication from IxWeb! So March the 31th when the ftp, webshell, stat tools, live chat and all my websites went down for the evening it was caused by one faulty generator? Just one? That’s scary… I know that you will do your best to prevent situations like this.
    Keep the good work and remember, MC pants are better than no pants…

  18. It’s good to see such an open discussion. I have had some issues with my sites being unavailable for a half hour or so. How long did the complete outage last? After a year at this game I am still a pretty rank beginner so if the forum idea comes off, it will be a useful addition to your good value service.

    Oh and I’ll pass on the pants thanks – I’ll just go commando!

  19. Gary Stadter -

    Any update on the availability of an IX customer forum?

  20. I am in the process of transferring my hosting account from “Justhost.com” to IX. I am surprised to see that there is no standard installation of a CGI based counter. While justhost randomly bothers its customers with non-specific terms of service violations that cannot be mitigated, IX does not have a simple to implement clickable gui counter. Why not?

  21. i m new customer and this is gr8 new for me

  22. Robert Mroczka -

    Hi All,

    I just wanted to update my post on May 31, 2009.

    So far so good, havent had any further problems with sites being down and am actually in the process now of moving my business site and domain here. Every things has gone well and have received continued qualtiy support in this matter.

    Cant wait to see a Forum for this hosting provider as sharing tips and tricks is a great way to get the most out of the service.

    Peace,
    Rob M.

  23. I am looking forward to the IX Forum. I’ve found that a lot of problems can be solved by customers helping customers. Tech support has always been extremely helpful but filling out a support ticket seems like overkill to ask a simple question and Live Chat often suggests that you fill out a ticket.

    Please consider including parts of the forum dedicated to status reports, tips and a part for customers to have ther sites reviewed by other customers.

  24. Its always bad when something unpredicted like this happens. I have understanding for this, but it would be great if its fixed promptly and to be secure with backup systems as some of the webmasters in the meantime lose money and custormers…

    However the nice and clear explanation is good and i am also looking forward to IX Forum as well :)

  25. appointmentschedulingsoftware -

    Awesome! thanks for adding this
    Thanks

  26. Aurelio -

    Do we have an interactive blog/forum where IX customers can discuss ideas and “how to” topics? I'm tring to do some things on our web site that I'm sure fellow IX customers can help with or at least share some ideas.

    Just a thought…

    Aurelio Martinez
    InsideBoxing.com

    • IX: Lisa G. - -

      Aurelio,

      We are currently in the process of re-working our forum area, so we don't have a one on the main site currently, but you do have a personal support hero – James Holt, who has a team blog site and lots of good resources and ideas to share. I have sent him a note with your information to get in touch with you.

      Hope that helps!
      Lisa Grice
      Director of Customer Service

  27. IX: Lisa G. -

    Aurelio,

    We are currently in the process of re-working our forum area, so we don't have a one on the main site currently, but you do have a personal support hero – James Holt, who has a team blog site and lots of good resources and ideas to share. I have sent him a note with your information to get in touch with you.

    Hope that helps!
    Lisa Grice
    Director of Customer Service

  28. In my opinion, IX hasn't improved much at all. Yes, the support guys are responsive and they usually fix any issue and are more than willing to help BUT the that is not what defines a good host. We experience too many issues in the first place. It is from webserver, email server ftpserver or you name it. There is any issue appearing literally every week.

    Recently, our IP was blocked for ftp. The claim that there was malicious activity from this IP! Yet, they failed to describe that activity in detail. No code was changed and nothing was found. So why did they do this? I believe their monitoring tool is incorrectly identifying 'malicious activity'. Our cpanal and ftp is accessed from one new computer which is totally scanned and protected.

    I am a web designer with many customers. I know this stuff and in my opinion IX need to offer better stability in addition to excellent support. They need people like me to have enough confidence in their platform to bring more businesses. Currently, they are focussing on managing attrition by dropping prices rather than by improving stability.

    • IX: Rob H. - -

      @FJ: Thank you for your word of support with regard to our customer service! We definitely understand your frustration with issues occurring and it is a very high priority of ours to improve on system issues. In response to the issue you experienced with your ftp, I'd be happy to look into that for you to provide you with more information. I wasn't able to find your account using the email you provided so let me know what your customer ID is and I can get you more information.

We're Always There When You Need Us The Most!

Your Dedicated Support

At IX, we take care of our customers. And dedicated support is one of the ways we prove to you again and again that we are here to help you every step of the way, regardless of your skill level. With IX dedicated support, you get a support technician personally assigned to assist you. You get their name, number, email, social media connections, and work schedule! It's just one more facet of our service which proves our deeply rooted belief that being a great hosting provider requires more than just cutting-edge technologies, but the best in support and service.