How to Track, Score and Verify A Weather Forecast

Nov 4, 2014
Posted by Austin

Part of what makes a meteorologist better at forecasting is practice. But beyond practice is critical analysis. If one does not keep a record of the past, how would one be expected to predict the future? This is the cornerstone of why I developed a forecast tracking, verification and scoring program in Excel. I believe that the best way to improve myself as a meteorologist is to track all my forecasts, verify them against actual conditions and then attribute a numerical score.

Austin Oct 2014 - Forecast Tracking and Scoring

The Forecast Categories

There are four variables that I track in all of my forecasts:

  1. High/low temperature
  2. Probability of precipitation
  3. Quantitative precipitation forecast (or liquid equivalent)
  4. Maximum sustained wind speed

Error Points

In my opinion, weather forecasting should be scored like golf; the lowest score wins. The closer a forecaster gets to predicting actual conditions the lower the “error” and thus the fewer points he or she will earn. If a forecaster’s prediction hits spot-on with actual conditions then he or she earns 0 (zero) error points. But how do you attribute error points to each category?

1. High/low temperature: Temperature scoring is very straight forward. The number of error points earned is equal to the absolute difference between the forecast value and the actual high/low value for the relevant 12-hour period:

Error Points = |Actual – Forecast|

5 = |56°F – 61°F|

2. Probability of Precipitation (PoP): PoP is slightly more complicated to score because a forecaster can predict anywhere between 0% and 100% for the probability, however, the actual result is either yes, it did precipitate or no, it did not precipitate. For our purposes I am defining a precipitation event as when at least .01″ of liquid is measured at a given AWOS station (or similar official site). The only exception is if the station recorded an “event” (rain shower, light snow, etc.) but did not measure at least .01″ of liquid precipitation.

When forecasting, the meteorologist enters a number between 0 and 10 (10 = 100%). So the way the score is calculated is the total error points earned are equal to the square of the absolute difference between the forecast PoP and the actual conditions (divide score by half if an “event” was recorded at the station but no measurable precipitation is recorded) for the relevant 12-hour period:

Error Points = |Actual – Forecast|^2

16 = |10 – 6|^2

*Error Points = (|Actual – Forecast|^2) / 2

8 = (|0 – 4|^2) / 2

*only if an “event” is recorded with 0.00″ precipitation

3. Quantitative Precipitation Forecast (QPF): QPF is straight forward as well, and the scoring can be fairly flexible depending on the forecaster. The way I set up my program to score QPF uses fixed ranges and scores. So if the forecast value is within a predetermined range of the actual recorded precipitation total then the forecaster gets a fixed number of points (ex. 4 points).

If |Actual – Forecast| = 0 Then Error Points = 0

If |Actual – Forecast| > 0 and <= .10″ Then Error Points = 1

If |Actual – Forecast > .10″ and <= .25″ Then Error Points = 4

If |Actual – Forecast > .25″ and <= .50″ Then Error Points = 8

If |Actual – Forecast > .50″ and <= .75″ Then Error Points = 16

If |Actual – Forecast > .75″ and <= 1.00″ Then Error Points = 20

If |Actual – Forecast > 1.00″ Then Error Points = 24

4. Maximum Sustained Wind Speed: This is another variable that is easily scored. Simply put, the error points earned for the maximum sustained wind speed category are equal to the half of the absolute difference between forecast and actual values:

Error Points = (|Actual – Forecast|) / 2

2.5 = (|15 – 10|) / 2

How It Works

So now that we’ve covered how the error points are calculated all we need to do is develop a way to enter our forecasts, verify them and attribute a score. Thankfully, I’ve already handled this with a series of VBA macros and an Excel spreadsheet. Thus far I’ve only developed the sheet to track 48-hour forecasts. Once the forecaster enters their forecast into the sheet a button imports the forecast to the “verification” sheet where there are buttons next to each 12-hour forecast period. Pre-programmed macros pull historical data from Weather Underground’s site for any station and the score is automatically calculated.

After all the forecast periods have elapsed the forecaster can then “export” the forecast to a special sheet for saved forecasts and a total average score is calculated.

Letter Grade

Sometimes the reality of your success can be clouded by the ambiguity of using raw error points. Is 48 points for a 48-hour forecast good? Is it reasonable? Did you completely blow a forecast? To help clarify things I’ve developed a simple grading system based on the error points:

A = 0 -> 30 points

B = 31 -> 60 points

C = 61 -> 90 points

D = 91 -> 120 points

F = 121+ points

The Download

If you are interested in checking out this Excel-based program to track, verify and score your own weather forecasts you may feel free to download the document here:

Blank – Forecast Tracking and Scoring Ver 5

Requirements:

  • Windows Operating System (sorry Mac/Linux users, VBA scripts in Excel only works in Windows)
  • Updated .NET frameworks installed
  • An internet connection (to download past weather)
  • Microsoft Excel (the macros will not work in Open Office or other Excel-substitute programs)

Pricing

Website design and development rate is $50/hour. The scope and complexity of your website requirements will determine the overall cost. Please use the form below to request a quote.

To give you an idea of what an average website might cost:

  • A simple, single-page WordPress website with up to 5 sections will take about 5-8 hours ($250-$400).
  • A more complex website with multiple pages could take closer to 10-12 hours ($500-$600). 
  • A website that requires multiple pages and high levels of visitor interaction (such as memberships, forums, messaging, etc) could take more than 20 hours ($1000+) depending on the scope of the project. 

Virtual Customer Service & Social Media

Virtual customer service and social media management is billed hourly at $25/hour and invoiced monthly. We will work with you to establish budget limits so you’re not surprised at the end of the month by a large bill.

Managed Hosting and On-Going Support​

Minimum
$5/moYearly ($45)
Hosting
Updates
Uptime Monitoring
 
 
 
 
 
 
$50/hr Restoration/Recovery
$50/hr Migration Assist
 
Basic
$15/moYearly ($135)
Hosting
Updates
Uptime Monitoring
Daily Security Scans
Blacklist Monitoring
Audit Logging
 
 
Monthly Traffic Report
$50/hr Restoration/Recovery
$50/hr Migration Assist
Monthly Cloud Backups
Premium
$30/moYearly ($270)
Hosting
Updates
Uptime Monitoring
Daily Security Scans
Blacklist Monitoring
Audit Logging
Suspicious IP Lockouts
Image Optimization
Daily Traffic Report
Free Restoration/Recovery
Free Migration Assist
Daily Cloud Backups

Visit Us On TwitterVisit Us On Facebook