YORKTOWN HEIGHTS, N.Y. — At the headquarters of IBM Research, a curved glass structure surrounded by acres of woods, clusters of high-powered servers hum constantly, their digital eyes trained on the clouds.
Information from weather services, satellites and a fleet of cloud-tracking cameras near solar arrays streams into the computers, whose singular goal is to crack the billion-dollar problem facing the nation's electric grid: knowing ahead of time how weather changes will affect the network's ever-growing number of solar panels.
As many states, including Connecticut, have increased the use of wind and solar power, energy officials and researchers have warned that the greater slice of intermittent generation is a liability for the nation's electric grid, which was designed to have clear on and off switches for its portfolio of centralized power plants.
Forecasting methods for wind power advanced rapidly as the technology spread, but the ability to know in advance when and how the sun will light up the nation's countless solar panels is about a decade behind, researchers say.
"Solar is without a doubt the hardest to predict," said Hendrik F. Hamann, project lead for IBM's Watt Sun program, one of two funded by the U.S. Department of Energy to develop accurate forecasting systems for the energy industry.
The main issue is that solar panels turn on and off with little advanced notice, and require other power plants to increase or decrease output in response. In a few years, when it is estimated that solar will make up a larger slice of New England's electric grid, the momentary shifts in power output will require immediate responses from power plants to maintain balance in the electric grid.
Hamann described the need for forecasting through this metaphor: If you need to write an email, but your computer is off, you will have to wait a few minutes for it to start up. Likewise, grid operators need advanced notice of big shifts in the grid so they can call on power plants to start up or turn off.
A recent report from the California Institute of Technology found that the unpredictability and extreme power changes can create instability in the power grid. It concluded that without dramatic change the "grid will be unable to meet the demands of a digital society or the expansion of renewable energy."
In response, electric grid operators, like ISO New England, are working to anticipate the dips or spikes in solar output — days, hours and minutes before they occur.
Solar Nerve Center
Housed in a climate-controlled room on the first floor of IBM Corp.'s Thomas J. Watson Research Center, the cluster of servers digests the weather data, passing through sieve-like algorithms that result in fine-tuned solar forecasts.
Hamann, 46, standing in front of a grid of four flat-screen computer displays, selected a small array in Rutland, Vt. The monitor slowly zoomed into the location — a 50-kilowatt solar farm. The forecast showed a slow morning rise — because of cloud cover — which increased sharply at midday when the weather cleared up.
John Tedesco, an innovation team member at Green Mountain Power, the Vermont electric utility contributing information about the Rutland panels to the IBM solar forecasting effort, said the program is an important sign that the grid is getting smarter.
"We're learning how to allow the grid to function without always having someone switching something on or off," he said. "Just like a thermostat in your house, the grid is starting to regulate its current."
"If we can bank on a certain amount of generation with a degree of certainty, then the intermittency argument really goes away," he said. "Having a smarter grid, we can have a grid that can handle more intermittent technologies, more renewable technologies."
The volume of data and specificity of modeling need to accurately predict where, when and how long the sun will shine on solar panels throughout the country make the Watt Sun application a "perfect big data problem," Hamann said.
The computer cluster powering the Watt Sun program builds on the "machine-learning" technology that sent IBM's Watson computer through a successful run on "Jeopardy!" beating Ken Jennings, who holds the show's record for the longest winning streak. Other Watson applications include reading medical records to assist doctors, improving voice recognition, and developing a system that grows more grapes on a vineyard with less water.
The original Watson servers live in the same first-floor room as the Watt Sun computers.
The Watt Sun cluster interprets the meteorological data into predictions for solar power output. Those predictions are evaluated against actual outputs and are continually adjusted for accuracy, highlighting a bedrock fact of the statistical world: all models are wrong, but some are useful.
Bri-Mathias Hodge, a senior engineer at the National Renewable Energy Laboratory in Golden, Colo., likes the program for just that reason.
"Most people will try to focus on one model and improving that single model to get a better forecast," he said. "The IBM approach takes all the models out there and learns when they perform well and when they don't perform well. … This approach tries to emphasize each model when it is most useful and ignore it when it is the least useful."
Big Money Savings
In December 2012, federal energy officials invested about $4 million into Watt Sun because it promises to cut power costs. Currently, the electric grid keeps some power plants running in reserve so it can respond as output from solar and wind resources changes. As forecasts become more exact, fewer reserves will be necessary.
In the first year, Watt Sun improved forecasting by 50 percent; its goal was 33 percent. By the end of the three-year program, predictions are expected to improve by 100 percent. According to IBM Research, for every percentage point increase in solar forecast accuracy, ratepayers save $1 million for every gigawatt of solar power.
In New England, with its 0.4 gigawatts of solar resources, a 100 percent increase in forecast accuracy would save about $40 million a year. Possible savings from the country's roughly 10 gigawatts of solar would reach $1 billion. The numbers grow as installed solar capacity increases.
The Watt Sun program, and a sister study underway in Colorado by the University Corporation for Atmospheric Research, are just two steps many say are needed to upgrade the electric grid for a nation that has made large preferences for cleaner power.
"Our nation's grid was designed in the last century … around a model of centralized power plants," said Minh Le, head of the U.S. Department of Energy's solar technologies programs, which oversees the forecasting effort. "That model is starting to change."
IBM's Watt Sun project is a part of the energy department's 2011 Sun Shot program, whose goal is to reduce the overall cost of solar power from $4 a watt to $1 a watt. By the end of the project, the grid will "respond to that cloud going over your house."
Le also went to a metaphor in describing the program's value. "It's the same if an airport knew that there would be snow in the next hour. … You may not take off from California if there's going to be a blizzard in Chicago."