Part of what makes a meteorologist better at forecasting is practice. But beyond practice is critical analysis. If one does not keep a record of the past, how would one be expected to predict the future? This is the cornerstone of why I developed a forecast tracking, verification and scoring program in Excel. I believe that the best way to improve myself as a meteorologist is to track all my forecasts, verify them against actual conditions and then attribute a numerical score.
The Forecast Categories
There are four variables that I track in all of my forecasts:
In my opinion, weather forecasting should be scored like golf; the lowest score wins. The closer a forecaster gets to predicting actual conditions the lower the “error” and thus the fewer points he or she will earn. If a forecaster’s prediction hits spot-on with actual conditions then he or she earns 0 (zero) error points. But how do you attribute error points to each category?
1. High/low temperature: Temperature scoring is very straight forward. The number of error points earned is equal to the absolute difference between the forecast value and the actual high/low value for the relevant 12-hour period:
Error Points = |Actual – Forecast|
5 = |56°F – 61°F|
2. Probability of Precipitation (PoP): PoP is slightly more complicated to score because a forecaster can predict anywhere between 0% and 100% for the probability, however, the actual result is either yes, it did precipitate or no, it did not precipitate. For our purposes I am defining a precipitation event as when at least .01″ of liquid is measured at a given AWOS station (or similar official site). The only exception is if the station recorded an “event” (rain shower, light snow, etc.) but did not measure at least .01″ of liquid precipitation.
When forecasting, the meteorologist enters a number between 0 and 10 (10 = 100%). So the way the score is calculated is the total error points earned are equal to the square of the absolute difference between the forecast PoP and the actual conditions (divide score by half if an “event” was recorded at the station but no measurable precipitation is recorded) for the relevant 12-hour period:
Error Points = |Actual – Forecast|^2
16 = |10 – 6|^2
*Error Points = (|Actual – Forecast|^2) / 2
8 = (|0 – 4|^2) / 2
*only if an “event” is recorded with 0.00″ precipitation
3. Quantitative Precipitation Forecast (QPF): QPF is straight forward as well, and the scoring can be fairly flexible depending on the forecaster. The way I set up my program to score QPF uses fixed ranges and scores. So if the forecast value is within a predetermined range of the actual recorded precipitation total then the forecaster gets a fixed number of points (ex. 4 points).
If |Actual – Forecast| = 0 Then Error Points = 0
If |Actual – Forecast| > 0 and <= .10″ Then Error Points = 1
If |Actual – Forecast > .10″ and <= .25″ Then Error Points = 4
If |Actual – Forecast > .25″ and <= .50″ Then Error Points = 8
If |Actual – Forecast > .50″ and <= .75″ Then Error Points = 16
If |Actual – Forecast > .75″ and <= 1.00″ Then Error Points = 20
If |Actual – Forecast > 1.00″ Then Error Points = 24
4. Maximum Sustained Wind Speed: This is another variable that is easily scored. Simply put, the error points earned for the maximum sustained wind speed category are equal to the half of the absolute difference between forecast and actual values:
Error Points = (|Actual – Forecast|) / 2
2.5 = (|15 – 10|) / 2
How It Works
So now that we’ve covered how the error points are calculated all we need to do is develop a way to enter our forecasts, verify them and attribute a score. Thankfully, I’ve already handled this with a series of VBA macros and an Excel spreadsheet. Thus far I’ve only developed the sheet to track 48-hour forecasts. Once the forecaster enters their forecast into the sheet a button imports the forecast to the “verification” sheet where there are buttons next to each 12-hour forecast period. Pre-programmed macros pull historical data from Weather Underground’s site for any station and the score is automatically calculated.
After all the forecast periods have elapsed the forecaster can then “export” the forecast to a special sheet for saved forecasts and a total average score is calculated.
Sometimes the reality of your success can be clouded by the ambiguity of using raw error points. Is 48 points for a 48-hour forecast good? Is it reasonable? Did you completely blow a forecast? To help clarify things I’ve developed a simple grading system based on the error points:
A = 0 -> 30 points
B = 31 -> 60 points
C = 61 -> 90 points
D = 91 -> 120 points
F = 121+ points
If you are interested in checking out this Excel-based program to track, verify and score your own weather forecasts you may feel free to download the document here: