uk.sci.weather (UK Weather) (uk.sci.weather) For the discussion of daily weather events, chiefly affecting the UK and adjacent parts of Europe, both past and predicted. The discussion is open to all, but contributions on a practical scientific level are encouraged.

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Old February 9th 12, 09:28 PM posted to uk.sci.weather
external usenet poster
 
First recorded activity by Weather-Banter: Aug 2011
Posts: 55
Default Mathsy question (sorry!)

I know this forum is populated by various people with meteorology
backgrounds, so thought I'd ask here first...

For my PhD I am assimilating data into a general circulation model. Now
what I want to do is compare the assimilation result (a simple lat-lon
plot of whatever) with a lat-lon plot of the observations.

I can simply subtract one from the other to get a difference plot, but I
was wondering if there was a statistical way to compare the 'fit'
between two 2D plots? The model calculates the average rms error over
all lat-lon points, but this isn't very helpful, as one particularly bad
gridpoint can skew the whole result.

If you can give any helpful advice I shall endeavor to mention you in
the acknowledgements of my thesis. )

Many thanks,

--
Liam (Milton Keynes)
http://physics.open.ac.uk/~lsteele/

  #2   Report Post  
Old February 9th 12, 09:42 PM posted to uk.sci.weather
external usenet poster
 
First recorded activity by Weather-Banter: Feb 2009
Posts: 88
Default Mathsy question (sorry!)

On Feb 9, 9:28*pm, Liam Steele wrote:
I know this forum is populated by various people with meteorology
backgrounds, so thought I'd ask here first...

For my PhD I am assimilating data into a general circulation model. Now
what I want to do is compare the assimilation result (a simple lat-lon
plot of whatever) with a lat-lon plot of the observations.

I can simply subtract one from the other to get a difference plot, but I
was wondering if there was a statistical way to compare the 'fit'
between two 2D plots? The model calculates the average rms error over
all lat-lon points, but this isn't very helpful, as one particularly bad
gridpoint can skew the whole result.

If you can give any helpful advice I shall endeavor to mention you in
the acknowledgements of my thesis. )

Many thanks,

--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


Why not treat the assimilated version as a forecast? You can
construct various forecast scores averaged over spatial zones. We
often use ROCs but it depends what you want to test. Anything you
might do with a time series i.e. sequence of forecasts you can use
for a single event over space. You could use this to test the
improvement between different assimilation methods and different
assimilation starting points. I have no direct experience of working
in assimilation schemes but we look for skill in much that we do.

There is a good page on verification here.

http://www.cawcr.gov.au/projects/verification/


Andy
  #3   Report Post  
Old February 9th 12, 10:26 PM posted to uk.sci.weather
external usenet poster
 
First recorded activity by Weather-Banter: Jan 2005
Posts: 4,152
Default Mathsy question (sorry!)

On Feb 9, 9:28*pm, Liam Steele wrote:
I know this forum is populated by various people with meteorology
backgrounds, so thought I'd ask here first...

For my PhD I am assimilating data into a general circulation model. Now
what I want to do is compare the assimilation result (a simple lat-lon
plot of whatever) with a lat-lon plot of the observations.

I can simply subtract one from the other to get a difference plot, but I
was wondering if there was a statistical way to compare the 'fit'
between two 2D plots? The model calculates the average rms error over
all lat-lon points, but this isn't very helpful, as one particularly bad
gridpoint can skew the whole result.

If you can give any helpful advice I shall endeavor to mention you in
the acknowledgements of my thesis. )

Many thanks,

--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


You mention a lat-long plot of the observations as the base to
compare the assimilated figures with but surely observations are not
made on a lat-long grid but at irregularly spaced points. If one of
these observations is grossly in error it can be weeded out before
attempting the fit but to do so would require the meteorological
expertise acquired through experience.
Alternatively, all the observations could be used, whether
suspect or not and if the error at one point exceeds the rms error of
the entire grid by some specified amount it could be removed at that
stage though that would make the comparison invalid though it may well
be more use operationally.
I can't see any way round using the rms error even if a rogue
point or two spoils it. December 2010 "spoilt" the (fairly) smooth
winter warming of the last few decades but it happened and can't be
ignored.

Tudor Hughes, Warlingham, Surrey
  #4   Report Post  
Old February 9th 12, 11:22 PM posted to uk.sci.weather
external usenet poster
 
First recorded activity by Weather-Banter: Aug 2011
Posts: 55
Default Mathsy question (sorry!)

On 09/02/12 21:42, Andy M. wrote:
On Feb 9, 9:28 pm, Liam wrote:
I know this forum is populated by various people with meteorology
backgrounds, so thought I'd ask here first...

For my PhD I am assimilating data into a general circulation model. Now
what I want to do is compare the assimilation result (a simple lat-lon
plot of whatever) with a lat-lon plot of the observations.

I can simply subtract one from the other to get a difference plot, but I
was wondering if there was a statistical way to compare the 'fit'
between two 2D plots? The model calculates the average rms error over
all lat-lon points, but this isn't very helpful, as one particularly bad
gridpoint can skew the whole result.

If you can give any helpful advice I shall endeavor to mention you in
the acknowledgements of my thesis. )

Many thanks,

--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


Why not treat the assimilated version as a forecast? You can
construct various forecast scores averaged over spatial zones. We
often use ROCs but it depends what you want to test. Anything you
might do with a time series i.e. sequence of forecasts you can use
for a single event over space. You could use this to test the
improvement between different assimilation methods and different
assimilation starting points. I have no direct experience of working
in assimilation schemes but we look for skill in much that we do.

There is a good page on verification here.

http://www.cawcr.gov.au/projects/verification/


Andy


Thanks Andy, that site is very informative. Something like the
Displacement and Amplitude Score is what I would like, which compares
the magnitude of the model predictions with the spatial miss-match as
well. However, actually getting and implementing the code will be
something else entirely!

--
Liam (Milton Keynes)
http://physics.open.ac.uk/~lsteele/
  #5   Report Post  
Old February 9th 12, 11:28 PM posted to uk.sci.weather
external usenet poster
 
First recorded activity by Weather-Banter: Aug 2011
Posts: 55
Default Mathsy question (sorry!)

On 09/02/12 22:26, Tudor Hughes wrote:
On Feb 9, 9:28 pm, Liam wrote:
I know this forum is populated by various people with meteorology
backgrounds, so thought I'd ask here first...

For my PhD I am assimilating data into a general circulation model. Now
what I want to do is compare the assimilation result (a simple lat-lon
plot of whatever) with a lat-lon plot of the observations.

I can simply subtract one from the other to get a difference plot, but I
was wondering if there was a statistical way to compare the 'fit'
between two 2D plots? The model calculates the average rms error over
all lat-lon points, but this isn't very helpful, as one particularly bad
gridpoint can skew the whole result.

If you can give any helpful advice I shall endeavor to mention you in
the acknowledgements of my thesis. )

Many thanks,

--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


You mention a lat-long plot of the observations as the base to
compare the assimilated figures with but surely observations are not
made on a lat-long grid but at irregularly spaced points. If one of
these observations is grossly in error it can be weeded out before
attempting the fit but to do so would require the meteorological
expertise acquired through experience.
Alternatively, all the observations could be used, whether
suspect or not and if the error at one point exceeds the rms error of
the entire grid by some specified amount it could be removed at that
stage though that would make the comparison invalid though it may well
be more use operationally.
I can't see any way round using the rms error even if a rogue
point or two spoils it. December 2010 "spoilt" the (fairly) smooth
winter warming of the last few decades but it happened and can't be
ignored.

Tudor Hughes, Warlingham, Surrey


Hi Tudor.

Yes, I thought about restricting the domain that is used for quantifying
the analysis, but that seems a bit sneaky, as though I don't trust the
model results! What seems to be happening is that the model predicts
broadly the right features (e.g. clouds) but these might be shifted
slightly so they are in an adjacent grid box to the observations. Thus,
when calculating the rms, the model is penalised twice: once in the grid
box with too much cloud and again in the adjacent grid box with too
little cloud.

What I would ideally like is something that can compare magnitudes as
well as the spatial offset of model predictions. A look on the link Andy
posted suggests I need something like the Displacement and Amplitude
Score. How I would implement something like this is not clear though. I
imagine it's not simple!

--
Liam (Milton Keynes)
http://physics.open.ac.uk/~lsteele/


  #6   Report Post  
Old February 9th 12, 11:34 PM posted to uk.sci.weather
external usenet poster
 
First recorded activity by Weather-Banter: Aug 2011
Posts: 55
Default Mathsy question (sorry!)

On 09/02/12 21:42, Andy M. wrote:
On Feb 9, 9:28 pm, Liam wrote:
I know this forum is populated by various people with meteorology
backgrounds, so thought I'd ask here first...

For my PhD I am assimilating data into a general circulation model. Now
what I want to do is compare the assimilation result (a simple lat-lon
plot of whatever) with a lat-lon plot of the observations.

I can simply subtract one from the other to get a difference plot, but I
was wondering if there was a statistical way to compare the 'fit'
between two 2D plots? The model calculates the average rms error over
all lat-lon points, but this isn't very helpful, as one particularly bad
gridpoint can skew the whole result.

If you can give any helpful advice I shall endeavor to mention you in
the acknowledgements of my thesis. )

Many thanks,

--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


Why not treat the assimilated version as a forecast? You can
construct various forecast scores averaged over spatial zones. We
often use ROCs but it depends what you want to test. Anything you
might do with a time series i.e. sequence of forecasts you can use
for a single event over space. You could use this to test the
improvement between different assimilation methods and different
assimilation starting points. I have no direct experience of working
in assimilation schemes but we look for skill in much that we do.

There is a good page on verification here.

http://www.cawcr.gov.au/projects/verification/


Andy


Further reading of the web page has brought up the CRA verification
method, which compares spatial details as well as magnitudes. And, it
even has some IDL code supplied! Looking good...

http://www.cawcr.gov.au/projects/ver...ification.html

--
Liam (Milton Keynes)
http://physics.open.ac.uk/~lsteele/
  #7   Report Post  
Old February 10th 12, 07:47 AM posted to uk.sci.weather
external usenet poster
 
First recorded activity by Weather-Banter: Feb 2009
Posts: 88
Default Mathsy question (sorry!)

On Feb 9, 11:34*pm, Liam Steele wrote:
On 09/02/12 21:42, Andy M. wrote:









On Feb 9, 9:28 pm, Liam *wrote:
I know this forum is populated by various people with meteorology
backgrounds, so thought I'd ask here first...


For my PhD I am assimilating data into a general circulation model. Now
what I want to do is compare the assimilation result (a simple lat-lon
plot of whatever) with a lat-lon plot of the observations.


I can simply subtract one from the other to get a difference plot, but I
was wondering if there was a statistical way to compare the 'fit'
between two 2D plots? The model calculates the average rms error over
all lat-lon points, but this isn't very helpful, as one particularly bad
gridpoint can skew the whole result.


If you can give any helpful advice I shall endeavor to mention you in
the acknowledgements of my thesis. )


Many thanks,


--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


Why not treat the assimilated version as a forecast? *You can
construct various forecast scores averaged over spatial zones. *We
often use ROCs but it depends what you want to test. *Anything you
might do with a time series i.e. *sequence of forecasts you can use
for a single event over space. You could use this to test the
improvement between different assimilation methods and different
assimilation starting points. *I have no direct experience of working
in assimilation schemes but we look for skill in much that we do.


There is a good page on verification here.


http://www.cawcr.gov.au/projects/verification/


Andy


Further reading of the web page has brought up the CRA verification
method, which compares spatial details as well as magnitudes. And, it
even has some IDL code supplied! Looking good...

http://www.cawcr.gov.au/projects/ver...ification.html

--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


Yes it is a good site .., I have not used those scores but having
some idea of the spatial coherence and the departure from the target
is good ... might look myself.

You should also think about getting hold of a copy of Jolliffe and
Stephenson http://books.google.co.uk/books/abou...d=Qm2MjWVvUywC
I am told by Dave that a new edition is on its way.

What model are you building the assimilation system for?

Good Luck,

Andy

  #8   Report Post  
Old February 10th 12, 09:07 AM posted to uk.sci.weather
external usenet poster
 
First recorded activity by Weather-Banter: Nov 2010
Posts: 44
Default Mathsy question (sorry!)

On Feb 10, 7:47*am, "Andy M." wrote:
On Feb 9, 11:34*pm, Liam Steele wrote:



On 09/02/12 21:42, Andy M. wrote:


On Feb 9, 9:28 pm, Liam *wrote:
I know this forum is populated by various people with meteorology
backgrounds, so thought I'd ask here first...


For my PhD I am assimilating data into a general circulation model. Now
what I want to do is compare the assimilation result (a simple lat-lon
plot of whatever) with a lat-lon plot of the observations.


I can simply subtract one from the other to get a difference plot, but I
was wondering if there was a statistical way to compare the 'fit'
between two 2D plots? The model calculates the average rms error over
all lat-lon points, but this isn't very helpful, as one particularly bad
gridpoint can skew the whole result.


If you can give any helpful advice I shall endeavor to mention you in
the acknowledgements of my thesis. )


Many thanks,


--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


Why not treat the assimilated version as a forecast? *You can
construct various forecast scores averaged over spatial zones. *We
often use ROCs but it depends what you want to test. *Anything you
might do with a time series i.e. *sequence of forecasts you can use
for a single event over space. You could use this to test the
improvement between different assimilation methods and different
assimilation starting points. *I have no direct experience of working
in assimilation schemes but we look for skill in much that we do.


There is a good page on verification here.


http://www.cawcr.gov.au/projects/verification/


Andy


Further reading of the web page has brought up the CRA verification
method, which compares spatial details as well as magnitudes. And, it
even has some IDL code supplied! Looking good...


http://www.cawcr.gov.au/projects/ver...ification.html


--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


Yes it is a good site .., I have not used those scores but having
some *idea of the spatial coherence and the departure from the target
is good ... might look myself.

You should also think about getting hold of a copy of Jolliffe and
Stephensonhttp://books.google.co.uk/books/about/Forecast_verification.html?id=Q...
I am told by Dave that a new edition is on its way.

What model are you building the assimilation system for?

Good Luck,

Andy


I am working on a model of the martian atmosphere, and we are
currently using the 'Analysis Correction' assimilation scheme that
they used at the Met Office in the early 90s. There are more modern
assimilation schemes, but apparently they have difficulty when only
assimilating a few observations (and for Mars we only have one sounder
that gives us observations). I'm looking in particular at water vapour
and ice data, and trying to get the model's cloud prediction as
accurate as possible so we can work out what role they play in Mars'
climate.

As an example, I've put three plots up on my website:
http://www.physics.open.ac.uk/~lsteele/assimilation.png. The top panel
is observation data, the middle panel is an assimilation and the
bottom panel is a standard model run. To my eye the assimilation
resembles the observations more than the standard model, but it
actually has a larger rms error. Hence I was looking for something
that weights things spatially as well as simply by magnitude.

Liam
  #9   Report Post  
Old February 10th 12, 10:10 AM posted to uk.sci.weather
external usenet poster
 
First recorded activity by Weather-Banter: Nov 2010
Posts: 4,488
Default Mathsy question (sorry!)

Liam Steele wrote:
How I would implement something like this is not clear though. I
imagine it's not simple!
-----------------------------------
Something about me is glad it's still not "simple" to get a PhD ;-)
I can't help but it sounds interesting, good luck with it Liam,
Dave

  #10   Report Post  
Old February 11th 12, 09:16 AM posted to uk.sci.weather
external usenet poster
 
First recorded activity by Weather-Banter: Feb 2009
Posts: 88
Default Mathsy question (sorry!)

On Feb 10, 9:07*am, Sir Loin Steak wrote:
On Feb 10, 7:47*am, "Andy M." wrote:









On Feb 9, 11:34*pm, Liam Steele wrote:


On 09/02/12 21:42, Andy M. wrote:


On Feb 9, 9:28 pm, Liam *wrote:
I know this forum is populated by various people with meteorology
backgrounds, so thought I'd ask here first...


For my PhD I am assimilating data into a general circulation model.. Now
what I want to do is compare the assimilation result (a simple lat-lon
plot of whatever) with a lat-lon plot of the observations.


I can simply subtract one from the other to get a difference plot, but I
was wondering if there was a statistical way to compare the 'fit'
between two 2D plots? The model calculates the average rms error over
all lat-lon points, but this isn't very helpful, as one particularly bad
gridpoint can skew the whole result.


If you can give any helpful advice I shall endeavor to mention you in
the acknowledgements of my thesis. )


Many thanks,


--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


Why not treat the assimilated version as a forecast? *You can
construct various forecast scores averaged over spatial zones. *We
often use ROCs but it depends what you want to test. *Anything you
might do with a time series i.e. *sequence of forecasts you can use
for a single event over space. You could use this to test the
improvement between different assimilation methods and different
assimilation starting points. *I have no direct experience of working
in assimilation schemes but we look for skill in much that we do.


There is a good page on verification here.


http://www.cawcr.gov.au/projects/verification/


Andy


Further reading of the web page has brought up the CRA verification
method, which compares spatial details as well as magnitudes. And, it
even has some IDL code supplied! Looking good...


http://www.cawcr.gov.au/projects/ver...ification.html


--
Liam (Milton Keynes)http://physics.open.ac.uk/~lsteele/


Yes it is a good site .., I have not used those scores but having
some *idea of the spatial coherence and the departure from the target
is good ... might look myself.


You should also think about getting hold of a copy of Jolliffe and
Stephensonhttp://books.google.co.uk/books/about/Forecast_verification.html?id=Q...
I am told by Dave that a new edition is on its way.


What model are you building the assimilation system for?


Good Luck,


Andy


I am working on a model of the martian atmosphere, and we are
currently using the 'Analysis Correction' assimilation scheme that
they used at the Met Office in the early 90s. There are more modern
assimilation schemes, but apparently they have difficulty when only
assimilating a few observations (and for Mars we only have one sounder
that gives us observations). I'm looking in particular at water vapour
and ice data, and trying to get the model's cloud prediction as
accurate as possible so we can work out what role they play in Mars'
climate.

As an example, I've put three plots up on my website:http://www.physics.open.ac.uk/~lsteele/assimilation.png. The top panel
is observation data, the middle panel is an assimilation and the
bottom panel is a standard model run. To my eye the assimilation
resembles the observations more than the standard model, but it
actually has a larger rms error. Hence I was looking for something
that weights things spatially as well as simply by magnitude.

Liam


Thanks for this

If you ever get chance I would love to see a surface energy budget
(for somewhere) from the simulation for summer time. I am currently
teaching this with my second years for the earth of course. I once
set an exam question based around a radiation budget for the Martian
surface with quite a few hints and values given. No one tried it so I
did not put it on ever again. I must have had data from one of the
landers in the late 1990s

If you have the 4 way radiation streams (simulated) at the surface (or
near surface) that would be great if you have the heat fluxes as well
even better.

Thanks,

Andy

google Andy Morse Liverpool if you want to contact me at work.


Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
The Most Important Single Question Before Us! — No Question At All GW Bollocks sci.geo.meteorology (Meteorology) 0 July 16th 10 02:28 AM
SERIOUS question about CO2 ( Sincere Question. Please Help if you can) Crackles McFarly sci.geo.meteorology (Meteorology) 5 July 18th 07 04:13 AM
metcheck says sorry.. well done nguk.. uk.sci.weather (UK Weather) 2 October 3rd 03 05:03 PM
Sorry started new post! - See Contrails tony_powell uk.sci.weather (UK Weather) 0 August 3rd 03 06:55 PM
I'm Sorry I Haven't a Clue TudorHgh uk.sci.weather (UK Weather) 1 July 16th 03 07:10 AM


All times are GMT. The time now is 04:22 AM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 Weather Banter.
The comments are property of their posters.
 

About Us

"It's about Weather"

 

Copyright © 2017