Interesting article. And some claim that we can predict the mean world
temperature in 100 years.

I am not surprised that private forecasters beat the government
forecasters (Harold take note).

RL

Grading Weather Forecasts
September 14, 2006

There are few numbers that have as much impact on Americans' daily
routines than the predicted high and low temperatures and the chance of
rain. Yet, competing weather companies sometimes disagree on forecasts,
leading to confusion about whether it's better to pack a picnic or an
umbrella for that weekend trip.

I occasionally get mail from Numbers Guy readers suggesting I write a
column on weather, and, in particular, the accuracy of forecasts. But
crunching the numbers to see which forecaster is the most accurate is
beyond my capabilities.

Last week I received an email from Eric Floehr, a 36-year-old computer
consultant in Marysville, Ohio, who has long been interested in
meteorology. (When he was five, his mother wrote in his baby book that
his fascinations were dinosaurs and the weather.) Three years ago, as a
part-time venture, Mr. Floehr developed software to compare the major
weather forecasters. Recently, he launched a Web site
(http://www.forecastadvisor.com/) that makes the comparisons available
to the public, and gave me access to some of his underlying data.

The site, Forecast Advisor, tracks the changing city-by-city forecasts
from major weather companies and the National Weather Service, and
evaluates how close the forecasters came to actual conditions. Plug in
a Zip Code, and the site offers up the local track records for each
provider.

The site doesn't offer overall accuracy scores for the different
forecasters, though Mr. Floehr has computed those numbers. He was
hesitant to give them to me. "I don't want anyone to misconstrue that
there is any one most accurate weather forecast," he told me. After
looking at his numbers, I could see why he felt that way: It turns out
that, despite the city-by-city variations, the national average
accuracy ratings for the five major providers are all pretty close.
Still, over the past three years, the rankings of the five forecasters
have remained consistent. "Please notice all the caution tape around
this one!" he wrote in the email in which he included the overall
scores. So I will save those scores for the end of this column.

But it would be a mistake to focus on crowning a winner. Mr. Floehr's
site shows how much more valuable weather forecasts can be when put
into context. And it also offers some fascinating insights into
forecasting.

First, some definitions: Forecasts are considered accurate if they come
within three degrees of predicting the high and low temperatures for a
given day. Standards were more rigorous for precipitation: A forecast
for even a small chance of rain is considered incorrect if the day was
dry.

One of the more interesting tidbits that shows up in Mr. Floehr's data
is that accuracy rates vary widely by city. For example, in the past
year, the overall accuracy of forecasts for Miami ranged between 78%
and 84% among the different providers -- a respectable showing. But for
Columbia Falls, Mont., forecasters were right just 60% to 67% of the
time.

Other findings reflect well on weather companies. You might once have
cursed a false forecast and vowed that you could have done better, but
every major weather company trounces a more simplistic method -- an
almanac-style approach that predicts that today's weather will be just
like the weather was, on average, on this date in the past. Of course,
professionals do much better on short-term forecasts. Forecast Advisor
shows that when it comes to nine-day forecasts, the national providers
are about as reliable as your almanac.

The service also keeps track of how forecasters have adjusted their
predictions. There was more than a third of an inch of rain Wednesday
in South Bend, Ind., but as recently as three days ago the National
Weather Service was predicting a partly cloudy, dry day, according to
Forecast Advisor.

Most of the weather companies I talked with said they welcome Mr.
Floehr's analysis. Several have worked with Mr. Floehr to help him
develop his technology, and have also purchased in-depth reports to
help them track the competition and improve their own forecasts.

"It can provide a lot of value for weather companies and others who are
looking for specific information about the accuracy of forecasts for
their specific needs," said Michael Steinberg, an AccuWeather Inc.
senior vice president. Geoff Flint, chief executive of CustomWeather
Inc., which runs MyForecast , said, "Eric has taken a good stab at
trying to pin down how accurate people are." "We think he's doing a
pretty good job there," said Bruce Rose, a vice president and principal
scientist for weather systems for the Weather Channel, which serves up
forecasts on Weather.com. All three of the meteorologists' companies
have purchased analysis from Mr. Floehr. (Two others, WSI Corp.'s
Intellicast and the National Weather Service, declined to comment on
Mr. Floehr's analysis.)

He sells individual reports, generally for between $250 and $4,500, and
is seeking to sell subscriptions to a premium service called Forecast
Watch for $100 a month. Fewer than 10,000 people visited the free
Forecast Advisor site last month, he said.

Of course, Forecast Advisor's accuracy numbers don't tell the whole
story. Predicting extreme weather accurately is especially valuable,
yet the forecasters get no extra credit for that, nor are they
penalized for missing really badly -- four degrees off is as bad as 20
degrees off in the eyes of the Web site. Likewise, temperature and
precipitation predictions are weighted equally, even though an
unexpected rain shower can be much more disruptive than a chilly day.

"Does it really matter to most people if the high temperature is 75
degrees or 77 degrees? Probably not," AccuWeather's Mr. Steinberg said.
"But does it matter if severe thunderstorms come through this afternoon
or if there are just brief showers? That does matter."

Meteorologists have long had their own internal quality measures.
AccuWeather's Mr. Steinberg proudly pointed out that, with the
exception of a couple of months about a decade back, his site's
forecasts for average temperature in each month going back to January
1988 were better than those of the National Weather Service for
Washington, D.C. -- which AccuWeather picked as a point of reference
"because it's the headquarters of the National Weather Service, and
also where their funding oversight is," Mr. Steinberg said. ("We don't
benchmark against other forecasters," National Weather Service
spokesman Greg Romano told me. "It's really not as relevant to us as it
may be to others. Our primary mission is to protect lives and
properties.")

Dr. Rose, of the Weather Channel, told me that July and August are the
easiest months to forecast for temperature, with February the toughest.
But the inverse is true for precipitation, because winter storms tend
to be governed by major, well-understood systems. He added that so far
this month, 16.9% of forecasts of precipitation were false alarms, and
there was precipitation on 4.5% of days when Weather.com didn't
forecast it. (Weather.com said it intentionally errs on the side of
false alarms, reasoning that consumers are inconvenienced more by an
unexpected storm than by surprising sun.)

Aimee Devaris, a verification expert at the National Weather Service,
passed along performance measures mandated by Congress for major
weather events. Tornadoes were accurately predicted 79% of the time
from October 2005 through May 2006, with an average of 13 minutes of
lead time. That's improved from 68% and 10 minutes in 2001.

Ms. Devaris also emailed me a trend analysis showing that forecast
accuracy has improved, though not dramatically, over the past two
decades, as computer power increased and the agency collected more
data. The three-day temperature forecasts were within three degrees 66%
of the time in 2004, compared with 52% of the time in 1985. The rate of
errors of more than 10 degrees slipped to 2% from 8% during that time.
(The weather service switched to a different forecasting method in
2005, so more recent forecasts aren't directly comparable.)

So, which weather service is the best? I agree with Forecast Advisor's
Mr. Floehr that the question isn't as straightforward as it seems.
Still, I know you're probably as curious about his findings as I was,
so here, as promised, are the rankings for overall accuracy for this
year thus far: The Weather Channel (right 73% of the time), followed by
MyForecast (72.3%), AccuWeather (71.4%) and Intellicast (71.4%), and
the National Weather Service (71%). But accuracy in your area may vary.

How much faith do you put in weather forecasts? Do you think they've
generally improved? Join a discussion with me on this column.

raylopez99 wrote:
Interesting article. And some claim that we can predict the mean world
temperature in 100 years.


Weather is the local transient state of the atmosphere.
Climate is the wide area steady state of the atmosphere.
They are different.

I can not predict the transient state of a single fair coin.
It will fall heads or tails up as it pleases.

I can predict that steady state of many coins very well.
In fact, the more coin tosses there are, the more accurate
my prediction that they will fall heads up about half the time
is likely to be.

raylopez99 wrote:

Interesting article. And some claim that we can predict the mean world
temperature in 100 years.

Climate is like a gradually warming oven, while weather is the
individual vortices that occur inside that oven during the entire
warming period. Guess which phenomenon is easier to track? I need to
add that said oven would both warm and cool over time, but the net
effect would be warming. A lot of people are thrown off by the idea of
winter in a warming system. That's why averages are so important.

If you're still confused about the difference between climate and
weather, you ought not to post here.

R. Lander

R. Lander wrote:

If you're still confused about the difference between climate and
weather, you ought not to post here.


I'll stop posting here as soon as you do.

As for the rest of my post, you misread it, and ROger too.

Actually climate is easier to predict the closer to the initial
conditions you get. Thus predicting tommorrows weather is doable about
70% of the time (see the original post, at the bottom) whereas
predicting next month's weather is harder still, and still harder is
next year's weather, and so on.

You people are stupid. I have science degrees, doctorates, and became
a millionaire before age 30, and now run my own business.

What have you done R (Ridiculous) Lander? At least Roger is
successful, and for that I admire him.

RL

raylopez99 wrote:

You people are stupid. I have science degrees, doctorates, and became
a millionaire before age 30, and now run my own business.

If your ambition has truly made you a millionaire, fine, but it doesn't
prove the correctness of your positions on the environment. There are a
lot of rich a-holes in the world who step on toes and abuse the land.
I'm particularly tired of the real estate industry and mansions built
with funds from selling other mansions (net effect is more paved land).

You measure success in terms of money, but the money-less, ego-less
world existed much longer than our modern cash 'n' grab experiment. If
you want to understand environmentalism you need to put things in their
full context. The comforts you see around you today are largely built
from oil and impermanent.

What have you done R (Ridiculous) Lander? At least Roger is
successful, and for that I admire him.

I work as a field repair tech and try to contribute as little as
possible to consumerism. That's my measure of success. When you look at
the net impact of the human race it's hard to support our "progress"
unless you've decided that nature doesn't matter (big mistake). Much of
our so-called wealth exists in credit notes and one-time resource
grabs. Man is the only animal that uses (non-solar) external energy and
requires so much living space and food acreage. I wouldn't gloat about
increasing the size of this mess for the sake of personal profits.

R. Lander

raylopez99 wrote:

R. Lander wrote:

If you're still confused about the difference between climate and
weather, you ought not to post here.


I'll stop posting here as soon as you do.

Well, is that an admission that you _don't_ understand the difference
between climate and weather? Why keep posting the same easily-debunked
rhetoric? Repetition won't make it any less false.

R. Lander

The article strikes me as another vainglorious attempt by AccuWx to wrest
routine forecasting responsibility away from the NWS.

Regardless of intent, a binary verification scheme for temp and
precipitation is meaningless as a measure of forecast skill.

More better would be total absolute temp and quantitative precipitation
errors as well as on-set and duration of precipatation.

TQ wrote:

More better would be total absolute temp and quantitative precipitation
errors as well as on-set and duration of precipatation.

Indeed. More better. So do it. And if the private sector wins again,
what excuse will you have?

RL