The new generator system at the southeast corner
of the Mesa Lab
|
Seven years
from conception to implementation, a new standby generator system became
operational at the NCAR Mesa Laboratory on 6 March 2004. The system
will provide unlimited emergency power to the NCAR Computer Room, protecting
millions of dollars worth of advanced computing and data storage equipment
in case of electrical outages—a much-needed precaution in Boulder,
Colorado, a city prone to high winds, heavy snows, and peak power demands
during the heat of summer.
At the heart of the system, a pair of twin generators will be able
to run the NCAR supercomputers and maintain electrical and cooling
services for the computing facility. The generators will also support
critical building functions such as emergency lighting, the boilers
that heat the Mesa Lab, and cafeteria freezers.
Powered by California emissions-rated diesel engines that are extremely
quiet and burn far cleaner than natural gas, the generators will run
as long as they have fuel. Eight hours of fuel are stored onsite, and
more can be delivered as needed.
The generators project, implemented by the Scientific Computing Division
in cooperation with NCAR's Physical Plant Services, cost $2.1 million.
SCD conducted preliminary studies in late 2000 and began custom-designing
the system in spring 2001. Contractor bids were solicited in early
2003, with construction starting in in the fall. The generators arrived
on 17 January 2004 and were fired up for the first time on 17 February.
The system went online on 6 March.
A critical need
Aaron Andersen, manager of SCD's Operations and Infrastructure Support
Section, began researching the possibility of backup power for NCAR
in 1997. "That year I was promoted to head the infrastructure
group, and I was amazed a facility of this size didn't have standby
power capabilities. San Diego had its own generator facilities; Cornell
had them, NERSC [the National Energy Research Supercomputing Center]
had them.
"I wrote a proposal '97; it really picked up momentum when [current
SCD director] Al Kellie came on board. He was struck like I was that
we didn't have standby generator capabilities. He was coming from the
Canadian Meteorological Centre, where they ran for almost 30 days on
generators during the big ice storm, so he knew the need." (The
Ice Storm of 1998 was the most expensive weather disaster in Canadian
history.)
"There's a strong likelihood of power hits here because of our
location. Chinooks, blizzards, heat waves—outages due to any one
of these could cost considerable staff time and lost production for
a week or more of science runs."
Aaron notes that SCD sustained three outages greater than two hours
in the past year, resulting in thousands of CPU hours of lost simulation
time. Last fall, the Computer Room was down for 17 hours due to high
winds (see "High
winds knock out power," and Aaron remembers a blizzard in
1995 that took out the regional power grid for nearly two days.
Gary New, an SCD computing facility engineer and technical lead on
the generator project, agrees that lost production time on computer
systems is disrupting and expensive. The March 2003 blizzard, Colorado's
worst in 90 years, paralyzed the region and shut down all of UCAR.
The bluesky, blackforest, babyblue, chinook, chinookfe, and mouache
computing systems remained down for nearly 36 hours due to continuing
power hits. (See "Digging
out.")
"With an outage like that, the scientists are trying to get
work done; it's costly in that it stops the flow of their work. So
many jobs are in the queue, and if the computers are shut down, it
backs up the queue. Also, computers are much happier when they run
continuously, and shutdowns and powerups are difficult. An enormous
amount of staff resources are required get the Computer Room back up
and working."
Another reason emergency power is more important than it used to
be is that NCAR has moved from proprietary vector computer systems
to commodity-based clusters. The amount of staff time to recover a
1,200-processor cluster can be measured in weeks, as opposed to hours.
Clean, quiet, and environmental-friendly
From the project's outset, SCD was highly conscious of the need for
a clean and quiet emergency power source.
"We, being the National Center for Atmospheric Research, wanted
the best environmental-friendly system we could get," Gary explains.
"We conducted a study that compared diesel to natural gas. It
showed, hands down—and much to my surprise—that diesel burns
much cleaner."
In the design specification, SCD required California emission standards
for the generators. "These are some of the cleanest diesels installed
anywhere," Aaron says, "the best emissions-rated engines
that can be had."
They are also quiet. "A good one-quarter of the enclosure design
is to mitigate noise," Aaron notes. "There's a combination
of mufflers, sound dampening, baffles, and acoustic insulation. You
can stand next to them and have a conversation."
"You won't find this at Midas," adds Gary, pointing to
a photo of an eight-foot muffler. "These generators make less
noise than a delivery truck."
A look under the hood
The generators themselves are huge. Each of the two "gensets,"
consisting of a generator, a V16 diesel engine, and a radiator, weighs
33,800 pounds—nearly 17 tons.
"We've got two 16-cylinder diesel engines," Gary says.
"Each one puts out 1.5 megawatts of electricity. They're actually
rated for 1,750 kilowatts, but at our altitude, it's 1,500."
Each genset sits on a skid atop a flat, 950-gallon fuel tank and
is is hidden by a steel enclosure.
Other components of the generator system include a control switchboard,
containing breakers that route the power, and a load bank—both
also hidden by steel enclosures.
"The load bank, for all intents and purposes, is one huge toaster,"
Gary says. "It's got heating elements that induce an artificial
load, so the generators can be tested once a month and run full speed
without a live load."
"This keeps them in good operating order," Aaron adds.
"It's just like if you have a classic car—you want to crank
it up and take it for a spin every once and a while."
During acceptance testing, the generators ran for a combined total
of 20 hours supporting the artificial load of the load bank, which
is noisier than the generators. Some staff thought the sound was coming
from the generators, but that was not the case. During an actual emergency,
the generators would be driving the computing facility, not the load
bank, and the area would be much quieter. The load bank will be employed
only 30 minutes a month for preventive maintenance.
To minimize the aesthetic impact of the generator system and help
it blend into its architectural surroundings, the steel enclosures
are painted a muted, dusty pink to match the sandstone of the Mesa
Lab.
More protection
"Until we installed the new generators, our backup power was
strictly UPS—uninterruptible power supply," Aaron observes.
"That's a battery backup, which is a finite power source that
depends on how heavy the load is. At the loads we're running in the
NCAR Computer Room, we'd be lucky to get 30 minutes of emergency power."
"The idea with generators," says Gary, "is that as
long as you fuel them, the power source is unlimited. We have eight
hours of fuel onsite, so we have eight hours to make a decision. During
that time, we'll be supporting the Computer Room, but we'll also be
picking up some kitchen equipment—for instance, the freezers,
so we don't lose food—and other things for life-safety, such as
the egress lighting and the boilers in the physical plant for heating
and keeping the building going. The generators won't be any good if
the building freezes. So we'll be supporting a minimum of building's
mechanical systems."
Improved reliability = more science
"The main purpose of the new generators is to keep NCAR computing
systems online, which will in turn advance science," Aaron concludes.
"There's no question that it will vastly improve SCD's reliability
and up-time as we deliver computing services to the user community."
—Lynda Lester
|
