Internet data centers battle back from Sandy

Tracey Capen

We all know that Superstorm Sandy was immensely destructive to the eastern coast of North America, with New York and New Jersey particularly hard-hit.

While individuals battled to save their property — and in many cases their lives — huge Internet data centers struggled to stay online.

Sandy didn’t hit people and businesses just on the East Coast. The huge storm’s effects were felt across North America. As anyone who read last week’s Windows Secrets knows, the servers that deliver the Windows Secrets newsletter were knocked out just as we were preparing to publish.

Fortunately, we were able to switch to servers on the West Coast, and we believe most subscribers received the Nov. 1 issue.

The flooding caused by Sandy had a huge effect on the Internet. Numerous major data centers lost power and went dark, despite having spent millions of dollars on backup systems.

Most Internet users rarely think about data centers, but they are the core of the Web. They provide access to the websites we visit, the videos we stream, the connections we make with our cellphones, and the cloud services on which we’re increasingly storing our personal and business data.

In the wake of Sandy, we thought you might be interested in a few of the behind-the-scenes stories about the damage the storm inflicted on data centers in the New York/New Jersey area, as reported (beginning Tuesday, Oct. 30, 2012) by one of our sister publications, Data Center Knowledge. We edited the stories a bit to fit Windows Secrets, and we’ve provided links to the original, as-it’s-happening reports.

You can read the original article, “In Sandy’s aftermath, epic challenges for data centers,” at Data Center Knowledge.

— Tracey Capen, Windows Secrets editor in chief.


In Sandy’s aftermath, epic challenges for data centers

By Rich Miller

As Superstorm Sandy brought destruction and death to the East Coast, data centers in New York and New Jersey faced extraordinary challenges. Here’s a look back at the storm and its aftermath, as chronicled in our coverage at Data Center Knowledge.

Oct. 30

Massive flooding damages several NYC data centers

Flooding from Hurricane Sandy has hobbled two data center buildings in Lower Manhattan, taking out diesel fuel pumps used to refuel generators. A third building at 121 Varick is also reported to be without power. There were also reports of outages for some tenants at a major data hub at 111 8th Avenue, and many other New York–area facilities were running on generator power amid widespread utility outages.

Both Internap and Peer 1 are struggling to continue operations at 75 Broad Street after basement-level flooding disabled critical diesel fuel pumps, leaving the providers no way to refuel generators on mezzanine floors.

Customers of Datagram were knocked offline Monday evening as water flooded the basement of its building at 33 Whitehall, knocking out high-traffic sites including Gawker, Gizmodo, BuzzFeed and Mediaite.

Both 75 Broad and 33 Whitehall were located in “Zone A,” the flood zone inundated by a storm surge of more than 13 feet as Sandy’s landfall in New Jersey coincided with high tide in lower Manhattan.

Meanwhile, Atlantic Metro reports that several of its data centers are offline, including its facility at 121 Varick, which has “significant flooding” in its generator room.

There were also multiple reports of downtime for some tenants at 111 8th Avenue, a major communications hub owned by Google. Some reports attributed the outages to network problems; others said that an electrical panel had failed while parts of the building were switching to generator power.

Meanwhile, several data centers in northern New Jersey were reported to be running on generator power, as PSE&G and other utilities suffered widespread outages. More.

Oct. 31

Diesel ‘bucket brigade’ keeps Peer 1 online

A determined team of employees of Peer 1 Hosting, blog host Squarespace, and Fog Creek Software have formed a “bucket brigade” to relay five-gallon buckets of diesel fuel up 17 flights of stairs at 75 Broad Street to refuel a generator providing emergency power to the Peer 1 data center.

It is a decidedly low-tech, brute-force solution to the challenges at 75 Broad, located in Lower Manhattan. Strong arms and strong backs are replacing fuel pumps that normally bring diesel from tanks in the basement to the upper floors of the building, where the backup generators are providing emergency power to keep the servers online in the Peer 1 data center. Those pumps shut down yesterday when the storm surge from Hurricane Sandy flooded the basement levels of 75 Broad, rendering the pumps unusable.

Peer 1 had initially planned for a “controlled shutdown” at 10:45 a.m., Tuesday. But when the generators chugged on past that point, the company and its customers came up with the alternate plan (hauling fuel up 17 floors).

“We have 25 people stationed throughout the building to relay more fuel to the roof once it arrives,” Squarespace reported on its status blog. “The building now has powerful pumps clearing out the basement, which we hope will expose the main pump lines — which would allow us fuel for days. Hopefully our manual efforts … can see us through. … Spirits are strong and everyone from Peer 1, Fog Creek, and Squarespace is working together.”

On Tuesday, Nov. 6, Squarespace reported that it had brought in a rolling generator, but the building still suffered from water damage. More.

Oct. 31

Diesel: Lifeblood of the recovery effort

Much of the Internet is currently running on diesel fuel and priority service contracts. Emergency backup generators powered by diesel are helping many East Coast data center providers weather Superstorm Sandy. The importance of diesel in the post-Sandy economy is hard to overstate. In the wake of Sandy, diesel is the lifeblood of lower Manhattan, where generators are currently providing virtually all the area’s electricity.

Data centers often tout their ability to operate for days, if not weeks, on diesel reserves stored on site in case of disaster. Two potential problems can arise when a data center relies on diesel generation to get it through a disaster. The first is running out of diesel fuel. Although data centers are preferred customers and have contracts in place, there are others ahead of them in the line, during disasters, such as hospitals and water plants. There’s also the challenge of fuel contamination, which is closely tied to maintenance practices. [Fuel contamination can occur from poor maintenance or, in the case of Sandy, flooded storage tanks and pumps.]

Data centers spend millions of dollars on their emergency power so they can continue to run during events such as Sandy. Should power go out, an uninterruptable power supply kicks in while the diesel emergency generators get warmed up.

Once the generators are running, a data center is still not totally in the clear. Issues and complications can arise during and after the switchover, as illustrated by a series of outages in 2010 (story) in which automatic transfer switches failed to handle the transition properly. Fortunately, most data centers seamlessly switched over during this storm. More.

Nov. 1

Temps soar at data center inside NY building

Temperatures inside a Zayo Group data center at 111 8th Avenue in New York City briefly soared past 100 degrees Fahrenheit this afternoon, as generator problems forced the company to power down its cooling systems. The company eventually stabilized the temperature and brought the cooling system back online. Customers’ equipment stayed on throughout the incident — just one of many challenges that confronted New York’s data center community during and following Superstorm Sandy.

Like Datagram, Zayo Group brought in a rolling, two-megawatt generator to keep operations running.

Telx, which also has a data center at 111 8th Avenue, asked customers to power down nonessential equipment to conserve electricity and diesel fuel and reduce heat generation.

111 8th Avenue is among the world’s most wired buildings. It was originally built in 1932 as the Port Authority Commerce Building, and was redeveloped for telecom use by Taconic in the late 1990s.The enormous building houses major data center operations for Digital Realty Trust, Equinix, Telx and many other providers and networks — as well as 500,000 square feet of office space for Google. More.

Nov. 1

New York data centers battle back from storm damage

As Thursday dawned on lower Manhattan, the city’s battered data centers continued their recovery efforts. Wednesday was a day of fast-moving events, as some facilities that had been down came back online, and some that had been up went down. Several of the hardest-hit facilities were hoping to be back in action Thursday, as Con Edison gradually begins to restore power to some neighborhoods below 34th Street.

Verizon Business: The extent of the damage to Verizon Business facilities became more widely understood Wednesday and was described in detail in Thursday’s Wall Street Journal.

“The company’s headquarters — a key communications hub just north of the World Trade Center site at 140 West Street — was in a state of crisis not seen since the 9/11 attacks, which partially destroyed the building,” The Journal reported. “Mud still covered parts of the ornate lobby.”

“Down below, three and a half floors of the building’s five-level basement were still submerged, the brackish water sloshing up the walls of the stairwell. Verizon employees said Monday night’s storm surge was so powerful that it breached the protective plugs that surround cables coming into the building. As a result, water flooded the critical basement cable vault that takes in communications cables and directs them to switching gear upstairs, which wasn’t damaged.”

Verizon published a photo of the flooding at 140 West on its blog, showing at least three feet of water filling the building’s lobby. More.

Nov. 2

Generator delivery boosts hopes at Datagram

The power loss at hosting provider Datagram provided the earliest warning that New York’s Internet infrastructure was facing serious trouble. Datagram hosts high-traffic websites such as the Gawker network. It was knocked off the Internet Monday evening when the record storm surge from superstorm Sandy flooded key equipment in the company’s basement. Down for a day, Gawker eventually set up temporary, ad-free sites on the free blogging service Tumblr.

For the staff and customers of hosting provider Datagram, the cavalry arrived shortly after 2 p.m. Friday, in the form of a two-megawatt diesel generator on the back of an 18-wheeler. The huge engine had to make its way through the storm-battered streets of lower Manhattan to a data center at 33 Whitehall. By 2:30 p.m., the generator was in place and 30 workers labored to get it operational. More.



Subscribe to our Windows Secrets Newsletter - It's Free!

Get our unique weekly Newsletter with tips and techniques, how to's and critical updates on Windows 7, Windows 8, Windows XP, Firefox, Internet Explorer, Google, etc. Join our 480,000 subscribers!

Windows 8 Hacks: Tips & Tools for Unlocking the Power of Tablets and Desktops

Subscribe and get our monthly bonuses - free!

Want to hack the new Start screen and tiles for your Win8 Device, the new Lock screen, the new tile-based apps, or the automatic notification information? Yes, you can do that. How about running other operating systems inside Windows 8, running Windows 8 on a Mac, or hacking SkyDrive and social media? We'll show you how to do that as well. Get this excerpt and other 5 bonuses if you subscribe now!

= Paid content

All Windows Secrets articles posted on 2012-11-08:

Tracey Capen

About Tracey Capen

Editor in chief Tracey Capen was the executive editor of reviews at PC World magazine for 10 years, from 1995 to 2005. He was InfoWorld's managing editor of reviews from 1993 to 1995 and worked in the magazine's test center and as networking editor from 1989 to 1992. Between his stints at InfoWorld, he was senior labs editor at Corporate Computing magazine.