Interesting article. And some claim that we can predict the mean world
temperature in 100 years.

I am not surprised that private forecasters beat the government
forecasters (Harold take note).

RL

Grading Weather Forecasts
September 14, 2006

There are few numbers that have as much impact on Americans' daily
routines than the predicted high and low temperatures and the chance of
rain. Yet, competing weather companies sometimes disagree on forecasts,
leading to confusion about whether it's better to pack a picnic or an
umbrella for that weekend trip.

I occasionally get mail from Numbers Guy readers suggesting I write a
column on weather, and, in particular, the accuracy of forecasts. But
crunching the numbers to see which forecaster is the most accurate is
beyond my capabilities.

Last week I received an email from Eric Floehr, a 36-year-old computer
consultant in Marysville, Ohio, who has long been interested in
meteorology. (When he was five, his mother wrote in his baby book that
his fascinations were dinosaurs and the weather.) Three years ago, as a
part-time venture, Mr. Floehr developed software to compare the major
weather forecasters. Recently, he launched a Web site
(http://www.forecastadvisor.com/) that makes the comparisons available
to the public, and gave me access to some of his underlying data.

The site, Forecast Advisor, tracks the changing city-by-city forecasts
from major weather companies and the National Weather Service, and
evaluates how close the forecasters came to actual conditions. Plug in
a Zip Code, and the site offers up the local track records for each
provider.

The site doesn't offer overall accuracy scores for the different
forecasters, though Mr. Floehr has computed those numbers. He was
hesitant to give them to me. "I don't want anyone to misconstrue that
there is any one most accurate weather forecast," he told me. After
looking at his numbers, I could see why he felt that way: It turns out
that, despite the city-by-city variations, the national average
accuracy ratings for the five major providers are all pretty close.
Still, over the past three years, the rankings of the five forecasters
have remained consistent. "Please notice all the caution tape around
this one!" he wrote in the email in which he included the overall
scores. So I will save those scores for the end of this column.

But it would be a mistake to focus on crowning a winner. Mr. Floehr's
site shows how much more valuable weather forecasts can be when put
into context. And it also offers some fascinating insights into
forecasting.

First, some definitions: Forecasts are considered accurate if they come
within three degrees of predicting the high and low temperatures for a
given day. Standards were more rigorous for precipitation: A forecast
for even a small chance of rain is considered incorrect if the day was
dry.

One of the more interesting tidbits that shows up in Mr. Floehr's data
is that accuracy rates vary widely by city. For example, in the past
year, the overall accuracy of forecasts for Miami ranged between 78%
and 84% among the different providers -- a respectable showing. But for
Columbia Falls, Mont., forecasters were right just 60% to 67% of the
time.

Other findings reflect well on weather companies. You might once have
cursed a false forecast and vowed that you could have done better, but
every major weather company trounces a more simplistic method -- an
almanac-style approach that predicts that today's weather will be just
like the weather was, on average, on this date in the past. Of course,
professionals do much better on short-term forecasts. Forecast Advisor
shows that when it comes to nine-day forecasts, the national providers
are about as reliable as your almanac.

The service also keeps track of how forecasters have adjusted their
predictions. There was more than a third of an inch of rain Wednesday
in South Bend, Ind., but as recently as three days ago the National
Weather Service was predicting a partly cloudy, dry day, according to
Forecast Advisor.

Most of the weather companies I talked with said they welcome Mr.
Floehr's analysis. Several have worked with Mr. Floehr to help him
develop his technology, and have also purchased in-depth reports to
help them track the competition and improve their own forecasts.

"It can provide a lot of value for weather companies and others who are
looking for specific information about the accuracy of forecasts for
their specific needs," said Michael Steinberg, an AccuWeather Inc.
senior vice president. Geoff Flint, chief executive of CustomWeather
Inc., which runs MyForecast , said, "Eric has taken a good stab at
trying to pin down how accurate people are." "We think he's doing a
pretty good job there," said Bruce Rose, a vice president and principal
scientist for weather systems for the Weather Channel, which serves up
forecasts on Weather.com. All three of the meteorologists' companies
have purchased analysis from Mr. Floehr. (Two others, WSI Corp.'s
Intellicast and the National Weather Service, declined to comment on
Mr. Floehr's analysis.)

He sells individual reports, generally for between $250 and $4,500, and
is seeking to sell subscriptions to a premium service called Forecast
Watch for $100 a month. Fewer than 10,000 people visited the free
Forecast Advisor site last month, he said.

Of course, Forecast Advisor's accuracy numbers don't tell the whole
story. Predicting extreme weather accurately is especially valuable,
yet the forecasters get no extra credit for that, nor are they
penalized for missing really badly -- four degrees off is as bad as 20
degrees off in the eyes of the Web site. Likewise, temperature and
precipitation predictions are weighted equally, even though an
unexpected rain shower can be much more disruptive than a chilly day.

"Does it really matter to most people if the high temperature is 75
degrees or 77 degrees? Probably not," AccuWeather's Mr. Steinberg said.
"But does it matter if severe thunderstorms come through this afternoon
or if there are just brief showers? That does matter."

Meteorologists have long had their own internal quality measures.
AccuWeather's Mr. Steinberg proudly pointed out that, with the
exception of a couple of months about a decade back, his site's
forecasts for average temperature in each month going back to January
1988 were better than those of the National Weather Service for
Washington, D.C. -- which AccuWeather picked as a point of reference
"because it's the headquarters of the National Weather Service, and
also where their funding oversight is," Mr. Steinberg said. ("We don't
benchmark against other forecasters," National Weather Service
spokesman Greg Romano told me. "It's really not as relevant to us as it
may be to others. Our primary mission is to protect lives and
properties.")

Dr. Rose, of the Weather Channel, told me that July and August are the
easiest months to forecast for temperature, with February the toughest.
But the inverse is true for precipitation, because winter storms tend
to be governed by major, well-understood systems. He added that so far
this month, 16.9% of forecasts of precipitation were false alarms, and
there was precipitation on 4.5% of days when Weather.com didn't
forecast it. (Weather.com said it intentionally errs on the side of
false alarms, reasoning that consumers are inconvenienced more by an
unexpected storm than by surprising sun.)

Aimee Devaris, a verification expert at the National Weather Service,
passed along performance measures mandated by Congress for major
weather events. Tornadoes were accurately predicted 79% of the time
from October 2005 through May 2006, with an average of 13 minutes of
lead time. That's improved from 68% and 10 minutes in 2001.

Ms. Devaris also emailed me a trend analysis showing that forecast
accuracy has improved, though not dramatically, over the past two
decades, as computer power increased and the agency collected more
data. The three-day temperature forecasts were within three degrees 66%
of the time in 2004, compared with 52% of the time in 1985. The rate of
errors of more than 10 degrees slipped to 2% from 8% during that time.
(The weather service switched to a different forecasting method in
2005, so more recent forecasts aren't directly comparable.)

So, which weather service is the best? I agree with Forecast Advisor's
Mr. Floehr that the question isn't as straightforward as it seems.
Still, I know you're probably as curious about his findings as I was,
so here, as promised, are the rankings for overall accuracy for this
year thus far: The Weather Channel (right 73% of the time), followed by
MyForecast (72.3%), AccuWeather (71.4%) and Intellicast (71.4%), and
the National Weather Service (71%). But accuracy in your area may vary.

How much faith do you put in weather forecasts? Do you think they've
generally improved? Join a discussion with me on this column.