A serious electrical fire cut power to a large Web hosting company in Seattle, knocking numerous sites off the Internet on July 3 and the early hours of July 4, including WindowsSecrets.com.
All of Windows Secrets’ data was fully backed up, and all subscriptions will continue just as before the power outage, but it took longer to get our site back online than I’d like.
In February 2006, as I described in an article at the time, we upgraded Windows Secrets’ server equipment and relocated it to a secure carrier hotel in Seattle named Fisher Plaza. Our Web host in this building, named Adhost (short for “advanced hosting”), maintains an elaborate system of uninterruptible power supplies and two diesel generators that can supply electricity indefinitely if city utility power is cut.
An electric arc blaze in Fisher Plaza’s basement garage, however, destroyed the connection between the city’s power and the building. The fire forced the evacuation of Adhost and every other company in the building, including KOMO-TV, Seattle’s ABC network affiliate. The station’s newscast that evening stated that the fire reached a temperature of 5,000F/2,760C and melted thick metal plates. (See Figure 1, photo courtesy of KOMO News.)
Figure 1. The Fisher Plaza fire, which knocked out hundreds of Web sites, was so hot that it melted metal plates in the basement.
Besides Windows Secrets’ little collection of articles, the fire wiped some very big sites off the Internet:
- Authorize.net, one of the world’s largest credit-card gateways, is headquartered in Massachusetts but chose to locate servers at Adhost on the strength of the hosting company’s redundant systems. The credit-processing firm maintained a completely separate data center as a backup, but the servers there failed to come online when the crisis struck, according to a Data Center Knowledge article. Authorize’s 238,000 clients were unable to process credit-card transactions for more than 12 hours.
- Bing Travel (a feature of Microsoft’s new Bing.com search engine) was the last affected Web site to restore connectivity, according to the Dennis Schall blog. The travel site was down for 36 hours.
- Geocaching.com, a real-time geolocating service, was down for 29 hours. Jeremy Irish, president of parent company Groundspeak, spoke for many webmasters when he explained in a blog entry why his firm didn’t pay to keep two data centers running at all times:
“We’re not a bank, so although 29 hours is a long time to be down, we do not plan to duplicate our infrastructure so we are completely redundant. It is just too expensive to make fiscal sense.”
To be sure, Fisher Plaza did experience an earlier electrical fire on June 21, 2008 (as described in John Cook’s Venture Blog). But connectivity in that case was completely restored in “only” eight hours. At the time, we thought that our server being down for a few hours was an acceptable risk as a rare, worst-case scenario.
WindowsSecrets.com was unavailable for about 34 hours on July 3 and 4. For most of that period, however, we were able to display to visitors an alternative home page. This backup page displayed a notice that a fire at our hosting company had affected our site, so at least some explanation of what was happening was provided.
Low-cost tricks for short-term disaster recovery
If your company maintains a Web site, you might be interested in some low-cost, minimalist disaster-recovery systems we maintain in case of a catastrophe:
- Alertra is a service that periodically tests Web services and notifies the affected parties if a site is down. We pay about U.S. $7 per month for Alertra to monitor our Web site and our separate mail server’s SMTP service. My developers and I were notified by e-mails to our cell phones just before midnight, about 20 minutes after the Fisher Plaza outage began. (Alertra’s e-mail notifications are free; text messages and automated voice calls cost a few cents per incident.)
- EasyDNS offers a real-time DNS (domain name system), which allows a Web site to switch from one IP address to another within minutes. Without a service like this, a new server may not be findable by visitors for 24 to 48 hours while its IP address “propagates” gradually around the Internet. We pay EasyDNS about $20 per year. Once my developers determined that Fisher Plaza wouldn’t come back up for hours, they were able to reroute visitors from our power-deprived Web equipment to a spare server that displayed our explanatory notice.
After Fisher Plaza restored power, and our server got back to normal on July 4, I e-mailed a short news update to Windows Secrets subscribers, apologizing for the inconvenience of the outage.
Interestingly, the cost of maintaining duplicate data centers has dramatically dropped in just the last three years. I explained in my update that Windows Secrets has been testing virtual servers using “cloud computing” since January 2009.
My staff and I plan to move our site to a Web service like EC2 (Elastic Compute Cloud) within one year. When that upgrade is completed, a new virtual server can be created automatically within minutes if any one data center goes down. Little cost is incurred unless the backup server is needed.
Since my news update went out, a few readers have contacted me, wondering whether their e-mail addresses will remain private if Windows Secrets uses cloud computing. I can assure you that the security of our subscriber list will be even better after we make the move.
If you’re really paranoid, you should consider this: every time you send or receive an e-mail, your address is revealed to the owners of every router your message passes through. Mail servers routinely exchange your e-mail address in plain text. Until a new, super-secure e-mail standard is adopted, it’s theoretically possible for dishonest ISP workers to “sniff” addresses at will. So much for privacy.
I’ll give you more technical details about cloud computing as soon as our plans firm up. In the meantime, please enjoy Windows Secrets and thanks for your understanding during our all-too-human stumbles.
Brian Livingston is editorial director of WindowsSecrets.com and co-author of Windows Vista Secrets and 10 other books.