Here is the story from the downtime we have experienced for the main SultanHost.com server located at the The Planet datacenter on 06/01/2008-06/02/2008 (EEST). Note that we re-post official The Planet updates so when you see "we" it means The Planet, our comments are bold and/or red. Note that we post using EEST timezone and The Planet mostly refer to CDT timezone which is 7 hours behind.

06/01/2008 01:15 EEST - Our monitoring servers shown that main server of SultanHost is unreachable - reboot not helped, no response from The Planet.

06/01/2008 02:29 EEST - The Planet is currently experiencing an outage which is effecting a number of customers' servers.

06/01/2008 03:05 EEST - We have rerouted our mail servers and DNS should be propagated during next 12-24 hours (it may take longer for some customers, so, please, don't worry if we won't respond via email to you - perhaps we not yet read it). Note that we have/had Backup MX server so most emails should reach us sooner or later, we just not expected The Planet issue is so serious and that's why not rerouted mail earlier. SultanSupport.com portal will be inaccessible.

03:36 EEST: Today at approximately 5:45 PM, a transformer in our H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers are down.
We are working with the fire department, with our facilities staff on site, to assess the situation.

03:51 EEST: We have determined that no servers in the data center have been damaged. Nonetheless, they are down because power is out. Teams across the board are working to take appropriate action.

04:17 EEST: In our latest assessment, we have determined that networking gear has not been damaged, but we are without power so assessments continue. All disaster recovery systems are in motion, and we have teams already working in the data center.

06:46 EEST: OFFICIAL STATUS REPORT - From Doug Erwin: This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
There is no impact in any of our other five data centers.
I am sorry that this accident has occurred and I apologize for the impact.

09:56 EEST: Our various departments continue to work very hard to restore service to our H1 data center. We expect to have ServerCommand access restored within the next several hours as well as access to single homed nameservers out of H1 redirected to another facility.
We do not have an Estimated Time to Repair at present.

11:24 EEST: Our facilities team continues to work with our vendors at the H1 site to restore service to affected clients. We do not have an Estimated Time to Repair at present. Our staff and management continue to work through the night and we will continue to provide hourly updates.

14:54 EEST: We are continuing to work through various issues this morning. We will have additional contractors on-site this morning starting at approx. 7 AM. Some will hand-off from contractors who worked overnight and others will start the recovery/installation of new electrical gear to power the data center.
We are still working through the EV1 DNS and ServerCommand items. We are making progress on both items and expect to have both functional within the next 120 minutes.
In addition to the above, the network engineering group worked overnight to prepare the network for the recovery of H1. We expect the reconvergence of the network to go smooth once H1 comes back online.
We do not have an Estimated Time to Repair at present; we should have a better estimate this morning. Our staff and management continue to work through the night and morning-- we will continue to provide hourly updates.

18:11 EEST: More teams from The Planet are coming along with more contractors from key vendors for electrical and facilities to help get H1 online. At this time, DNS infrastructure continues to propagate. ServerCommand servers are installed, but the teams are making sure all networking is intact and ready. Additional support techs are available.
At this time, we do not have an updated Estimated Time to repair.

20:24 EEST: We expect to be able to provide initial power to parts of the H1 data center beginning at 5:00 p.m. CDT. At that time, we will begin testing and validating network and power systems, turning on air-conditioning systems and monitoring environmental conditions. We expect this testing to last approximately four hours.
Following this testing, we will begin to power-on customer servers in phases. These are approximate times, and as we know more, we will keep you apprised of the situation.

23:31 EEST: As you may have already noticed, our forum servers continue to lag due to very heavy load. This is in part to due to the fact that our outage is now being carried on several sites (including Slashdot). Even though we added servers to our forums last night, we are looking at alternatives at this time to provide simple status updates quickly.
We are still working on getting all management systems up, but Legacy EV1 domain customers can access a backup management panel hosted by Tucows at https://manage.opensrs.net. There are some limitations to these backup systems, changes may be restricted if the domain is locked, and use may be intermittent until the servers hosting the main domain management systems in H1 are back online.
We have no other update at this time regarding our planned power test at 5:00pm.

06/02/2008 01:02 EEST: We continue to pursue our plans to provide initial power to our H1 data center this evening. It will take several hours to assure power can be safely restored to the facility. Based on how the initial work goes, we will have more information to provide you in the upcoming hours. We will post another update by 7:30 pm tonight.
In the meantime, we have further rerouted both old and new IP addresses for our name servers previously housed in H1. This means that the servers can start resolving IP addresses on both their former and new addresses, and this will alleviate the issues we have been seeing with propagation delay from this address change.

07:14 EEST: OFFICIAL STATUS REPORT - As previously committed, I would like to provide an update on where we stand following yesterday's explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.
As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.
There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.
We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.
Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.
We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do.
We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed.
I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service.

07:53 EEST: BAD LUCK - We found yourself to be on Floor 1 (Phase 1) which means outage will last much longer for our main server (which is the only affected The Planet server amoung number of them).

08:55 EEST: After the fire marshall inspected the H1 location, we were given the green light to bring power back to the facility. The generators have been turned on, and we are receiving power on the second floor. The generator power restoration is the first step in the full restoration of service to the data center.
From here, we will begin the process of cooling the DC floor, which could take a few hours. As soon as the power integrity is confirmed and the DC floor is ready for operation, we will be restoring power and checking server hardware on a rack-by-rack basis.

11:05 EEST: We are continuing the process of turning on and verifying hardware integrity of customer servers on the second floor of H1.
Our network operations team is currently working on the ev1servers.net nameservers to ensure that they are online, are routed to correctly, and propogate as quickly as possible.

15:39 EEST: With the start of another official workday (though a large number of people on the team have been working through the night), we are poised for a significant amount of work on both phases of our H1 data center. We have a full team of people on site working to ensure our targets are met to restore power to H1, Phase 1 (first floor).

17:19 EEST: We now have 90% of servers located on the second floor of H1 online. Support technicians are on location to manually bring the remaining 10% online.

19:06 EEST: Onsite technicians are currently working to restore service to the remaining 10% of Phase 2 upstairs servers. The Phase 1 downstairs servers are expected to start coming online late this evening.

22:59 EEST: Power in H1 Phase 1 has been restored. We are starting to turn customer servers on in batches.

23:05 EEST: CAME BACK - Our server finally has been back online. We are working on it in order to synchronize changes, re-launch SultanSupport.com, etc.

We all here, at SultanHost, do hope that the issue not brings you much troubles! Disasters like this may happen even on redundant systems. The Planet got another lesson and we may expect more stronger power redundancy. We will definitely do more robust setup on our primary servers to prevent issues like this from affecting our business.