Jump to content

Welcome to our forums!

Sign In or Register to gain full access to our forums. By registering with us, you'll be able to discuss, share and private message with other members of our community.

Welcome!

Thanks for stopping by the Weather Forums! Please take the time to register and join our community. Feel free to post or start new topics on anything related to the weather or the climate.


Photo

Purging of Climate Records

- - - - -

  • Please log in to reply

#1
SilverFallsAndrew

Posted 03 December 2017 - 10:16 PM

SilverFallsAndrew

    Forum Fantastic

  • Members
  • PipPipPipPipPipPipPipPip
  • 11293 posts
  • LocationSilverton, OR

It has been talked about before, but it drives me nuts how they purge old records in their quality control process. Is there any way we can stop this or at least view the original data? I have noticed a lot of the cold records from December 1919 and 1930 show up as M now on the Utah State site. The -6 at SLE in 1919 is no longer listed as a record on the WRCC site. 

 

Also from more recent times the -10 at Eugene on 12/8/13 is now shown as M on the Utah State site. 

 

Even more shocking. The -12 at Eugene and Salem from 12/8/1972 has been scrubbed from the records! In fact the WRCC lists the record for the date at Salem as 8 set in 2013. Though oddly they list the -10 from 2013 as the record at Eugene. So disappointing. 

 

 


Snowfall

2017-18: 0"

2016-17: 49.2"

2015-16: 11.75"

2014-15: 3.5"
2013-14: 11.75"
2012-13: 16.75"
2011-12: 98.5"

 

 

 


#2
wx_statman

Posted 03 December 2017 - 10:46 PM

wx_statman

    Advanced Member

  • Members
  • PipPipPipPipPipPip
  • 3797 posts
  • LocationPortland, OR

The issue is over at NCEI, which has the database that everyone else pulls from (NCDC, WRCC, Utah Climate Center, etc). 

 

Their QC algorithm leaves a lot to be desired.



#3
snow_wizard

Posted 03 December 2017 - 10:53 PM

snow_wizard

    The Snow Wizard

  • Mods
  • 12058 posts
  • LocationCovington, WA

Sounds like they consider the readings to be suspect.  Somebody needs to let them know that is not the case.  In Utah they should understand what cold pockets are all about.


Death To Warm Anomalies!

 

Winter 2017-18 stats

 

Total Snowfall = 0.2"

Coldest Low = 25

Lows 32 or below = 14

Highs 32 or below = 0

Lows Below 20 = 0

Highs 40 or below = 4

 

 


#4
SilverFallsAndrew

Posted 03 December 2017 - 10:55 PM

SilverFallsAndrew

    Forum Fantastic

  • Members
  • PipPipPipPipPipPipPipPip
  • 11293 posts
  • LocationSilverton, OR

The issue is over at NCEI, which has the database that everyone else pulls from (NCDC, WRCC, Utah Climate Center, etc). 

 

Their QC algorithm leaves a lot to be desired.

 

I know you have done some follow up on this in the past. Is their any hope they will fix the algorithm? Also once these various records are purged are they lost to posterity or is their still some kind of original data set kept somewhere that is publicly available?


Snowfall

2017-18: 0"

2016-17: 49.2"

2015-16: 11.75"

2014-15: 3.5"
2013-14: 11.75"
2012-13: 16.75"
2011-12: 98.5"

 

 

 


#5
wx_statman

Posted 03 December 2017 - 11:00 PM

wx_statman

    Advanced Member

  • Members
  • PipPipPipPipPipPip
  • 3797 posts
  • LocationPortland, OR

Sounds like they consider the readings to be suspect.  Somebody needs to let them know that is not the case.  In Utah they should understand what cold pockets are all about.

 

There's no human input in this. It's a statistical algorithm. 


  • Scott likes this

#6
wx_statman

Posted 03 December 2017 - 11:03 PM

wx_statman

    Advanced Member

  • Members
  • PipPipPipPipPipPip
  • 3797 posts
  • LocationPortland, OR

I know you have done some follow up on this in the past. Is their any hope they will fix the algorithm? Also once these various records are purged are they lost to posterity or is their still some kind of original data set kept somewhere that is publicly available?

 

Yeah, I had an email conversation with a WRCC climatologist about this issue last year. They're frustrated too since it's not up to them which records get deleted. I have no idea how/when this will be resolved. 

 

The best source for original, unadulterated data are the monthly and annual climate reports. They're available through the NCDC. All the original data can be found here:

 

https://www.ncdc.noa.../IPS/cd/cd.html



#7
SilverFallsAndrew

Posted 03 December 2017 - 11:07 PM

SilverFallsAndrew

    Forum Fantastic

  • Members
  • PipPipPipPipPipPipPipPip
  • 11293 posts
  • LocationSilverton, OR

Yeah, I had an email conversation with a WRCC climatologist about this issue last year. They're frustrated too since it's not up to them which records get deleted. I have no idea how/when this will be resolved. 

 

The best source for original, unadulterated data are the monthly and annual climate reports. They're available through the NCDC. All the original data can be found here:

 

https://www.ncdc.noa.../IPS/cd/cd.html

 

Thanks for the link!


Snowfall

2017-18: 0"

2016-17: 49.2"

2015-16: 11.75"

2014-15: 3.5"
2013-14: 11.75"
2012-13: 16.75"
2011-12: 98.5"

 

 

 


#8
SilverFallsAndrew

Posted 03 December 2017 - 11:10 PM

SilverFallsAndrew

    Forum Fantastic

  • Members
  • PipPipPipPipPipPipPipPip
  • 11293 posts
  • LocationSilverton, OR

Thanks for the link!

 

Wallace Orchard hitting -14 in December 1919 is pretty impressive as I believe that is essentially West Salem in modern times. That is one of my favorite now defunct stations, along with Mt. Angel, and Sundown Ranch. 


Snowfall

2017-18: 0"

2016-17: 49.2"

2015-16: 11.75"

2014-15: 3.5"
2013-14: 11.75"
2012-13: 16.75"
2011-12: 98.5"

 

 

 


#9
wx_statman

Posted 03 December 2017 - 11:17 PM

wx_statman

    Advanced Member

  • Members
  • PipPipPipPipPipPip
  • 3797 posts
  • LocationPortland, OR

Thanks for the link!

 

No problem! It took me until last year to discover that these reports are available online. Makes looking up data for past events so much easier. 



#10
wx_statman

Posted 03 December 2017 - 11:28 PM

wx_statman

    Advanced Member

  • Members
  • PipPipPipPipPipPip
  • 3797 posts
  • LocationPortland, OR

The QC'ing issue is especially noticeable at stations like The Dalles. Their cold records have been gutted. Missing are the -30 from December 1919, the -25 from January 1930, and the -21 from February 1950. If you look at their daily records now, the coldest January reading shown is just -11, the coldest February reading shown is -12, and the coldest December reading is just -6. 

 

https://wrcc.dri.edu...iMAIN.pl?or8407

 

The end result is essentially a fake record book. Completely unrepresentative of the actual capabilities of that climate over the period of record (1893-present). 



#11
Scott

Posted 04 December 2017 - 12:11 AM

Scott

    Forum Member

  • Members
  • PipPipPip
  • 328 posts
  • LocationNear Craig Colorado
Their QC algorithm leaves a lot to be desired.

 

 

No doubt.  Here's an example of it not working.

 

I noticed that the -50 at Woodruff Utah on 2/6/1899 was purged.  This is the old state record if you discount the un-populated areas such as Peter and Middle Sinks and is probably accurate.

 

Meanwhile the bogus high records of 115 on 8/2/1907 and 111 on 9/4/1907 were left intact.   There's no way that it was that hot in Woodruff (elevation 6350 feet).



#12
snow_wizard

Posted 04 December 2017 - 12:19 AM

snow_wizard

    The Snow Wizard

  • Mods
  • 12058 posts
  • LocationCovington, WA

The QC'ing issue is especially noticeable at stations like The Dalles. Their cold records have been gutted. Missing are the -30 from December 1919, the -25 from January 1930, and the -21 from February 1950. If you look at their daily records now, the coldest January reading shown is just -11, the coldest February reading shown is -12, and the coldest December reading is just -6. 

 

https://wrcc.dri.edu...iMAIN.pl?or8407

 

The end result is essentially a fake record book. Completely unrepresentative of the actual capabilities of that climate over the period of record (1893-present). 

 

Totally ridiculous.  The West is capable of great anomalies and the idiot algorithm should know that.


Death To Warm Anomalies!

 

Winter 2017-18 stats

 

Total Snowfall = 0.2"

Coldest Low = 25

Lows 32 or below = 14

Highs 32 or below = 0

Lows Below 20 = 0

Highs 40 or below = 4

 

 


#13
SilverFallsAndrew

Posted 04 December 2017 - 08:48 AM

SilverFallsAndrew

    Forum Fantastic

  • Members
  • PipPipPipPipPipPipPipPip
  • 11293 posts
  • LocationSilverton, OR

I was thinking about this a little more on the drive into work this morning. It seems like cold records are most effected by this. Perhaps it is perception, but I don't see July 2009 being purged...Is that likely because extreme cold episodes in our climate are more outliers when compared to the historical norm? PDX has been in the 104-107 range may times, but a -30 at The Dalles, though legit is really a once in a century type event without a lot of other events that even approached it? 


Snowfall

2017-18: 0"

2016-17: 49.2"

2015-16: 11.75"

2014-15: 3.5"
2013-14: 11.75"
2012-13: 16.75"
2011-12: 98.5"

 

 

 


#14
SilverFallsAndrew

Posted 04 December 2017 - 08:50 AM

SilverFallsAndrew

    Forum Fantastic

  • Members
  • PipPipPipPipPipPipPipPip
  • 11293 posts
  • LocationSilverton, OR

It is also weird some data gets purged and some doesn't. For instance the 0 at Silver Falls in December 1983 has been purged, but the -2 from 1990 or the multiple sub-zero days from 1972 remain. It doesn't seem to have much of a reason, except perhaps because 83' had the low of 0 and then the other lows were around 10, so it seemed like a big outlier in comparison to even the other days around it?


Snowfall

2017-18: 0"

2016-17: 49.2"

2015-16: 11.75"

2014-15: 3.5"
2013-14: 11.75"
2012-13: 16.75"
2011-12: 98.5"

 

 

 


#15
SilverFallsAndrew

Posted 04 December 2017 - 08:57 AM

SilverFallsAndrew

    Forum Fantastic

  • Members
  • PipPipPipPipPipPipPipPip
  • 11293 posts
  • LocationSilverton, OR

Okay...One more...

 

Going back to December 1990. There is missing data in the Utah State data set for maximums on the 20th, 25th, and 29th. So it shows the average high at Silver Falls being 37.3 for the month.

 

The missing data for those dates is in the NCDC report.

 

20: 18

25:  33

29: 23

 

The average high for the month was 35.6.

 

The actual mean temp for the month was 30.5, not 31.8. Either way it is the coldest December on record at Silver Falls, but that is a big difference. 


Snowfall

2017-18: 0"

2016-17: 49.2"

2015-16: 11.75"

2014-15: 3.5"
2013-14: 11.75"
2012-13: 16.75"
2011-12: 98.5"

 

 

 


#16
wx_statman

Posted 04 December 2017 - 09:24 AM

wx_statman

    Advanced Member

  • Members
  • PipPipPipPipPipPip
  • 3797 posts
  • LocationPortland, OR

Totally ridiculous.  The West is capable of great anomalies and the idiot algorithm should know that.

 

I agree. It's ridiculous. 



#17
wx_statman

Posted 04 December 2017 - 09:34 AM

wx_statman

    Advanced Member

  • Members
  • PipPipPipPipPipPip
  • 3797 posts
  • LocationPortland, OR

Okay...One more...

 

Going back to December 1990. There is missing data in the Utah State data set for maximums on the 20th, 25th, and 29th. So it shows the average high at Silver Falls being 37.3 for the month.

 

The missing data for those dates is in the NCDC report.

 

20: 18

25:  33

29: 23

 

The average high for the month was 35.6.

 

The actual mean temp for the month was 30.5, not 31.8. Either way it is the coldest December on record at Silver Falls, but that is a big difference. 

 

It kills people like you and me, but the problem is that not a whole lot of other people care about this stuff. It was probably a lot cheaper for the gov't to develop the algorithm the way it is than to fine tune it. I'm guessing their goal was to knock out a bunch of clearly erroneous data in the COOP record (and there was PLENTY) in order to make the data more useful for scientific study. I'm not even sure if anyone in charge cares about the fact that so many legitimate records were knocked out. Most daily observations and records are still there...so they might run a report and say "well, 96% of daily data remains and we've statistically ascertained that it's all legitimate." Of course the other 4% are all the climatologically significant records that we as enthusiasts care about, in addition to whatever erroneous data that got knocked out.



#18
Phil

Posted 04 December 2017 - 10:27 AM

Phil

    Forum Fantastic

  • Members
  • PipPipPipPipPipPipPipPip
  • 15552 posts
  • LocationCabin John, MD.
This is why I think the management of climate records (and weather/climate modeling in general) should be turned over to the private sector. Preferably to multiple institutions.
Personal Weather Station, Live Stream on Wunderground: https://www.wundergr...BETHE62#history

Cold season 2017/18
Snowfall: 3.7”
Largest Snowfall: 3.4”
Number of winter events: 3
Coldest High: 28.9*F
Coldest Low: 20.8*F
Lowest Dewpoint: 1.8*F
Highest Sustained Wind: 37mph
Highest wind gust: 54mph

#19
WeatherArchive

Posted 04 December 2017 - 12:20 PM

WeatherArchive

    Forum Member

  • Members
  • PipPipPip
  • 218 posts
  • LocationStayton

Today's generation is all about automation. They do NOT believe in public feedback and that itself is damaging to our system.  Sure a lot of them will say you can email them but they just send it straight to the rubbish bin.   Today's generation is running blindly the wrong way meanwhile the countries that have experienced true dictatorship like China have slowly realized it and are turning slowly but surely towards freedom more and it's worked out the moment they first tried in the 1990s though still clearly have a ways to go yet.