diamond geezer

 Thursday, February 15, 2018

Earlier this month the BBC ended its weather forecasting contract with the Met Office and changed over to a new provider - commercial operators Meteo Group. People hate change, and not every aspect of the switchover has been welcomed. But what interested me most were these three 'improvements'...
More locations: The BBC has added thousands of new locations, including many international, to our database
New 14-day forecast: Users can view 14 days of hourly forecast data for UK locations and major international cities
New forecast features: Added data fields like ‘chance of rain’ and ‘feels like’ temperature
So I thought I'd give the new 14-day forecast a proper test drive.

I started 14 days ago, on 1st February.
Every day since then I've checked the online weather forecast for 14th February, and made a note of what was predicted.

I've been using the weather forecast for Bow (which has different figures to Stepney, Poplar, Stratford and the generic London page). I have a lot more data than I'll be showing you here. And I can confirm that the concept of a 14-day forecast is definitely suspect.

It turns out that 14th February was an excellent day to have picked. Yesterday started off sunny and dry. In the middle of the day a band of rain arrived, lingered all afternoon and became heavier in the evening. The day ended cloudy and wet. With all sorts of weather to predict, and a very specific arrival time for the band of rain, there was plenty for the new forecast to get right.

First of all, the overall summary. The summary for the actual weather yesterday was "light rain and breezy", and the temperature for most of the day was 6 or 7 degrees.

Here's what the BBC Weather website predicted for yesterday's weather on each of the preceding days.
  1st Feb: Light rain and breezy (3-8°C)
  2nd Feb: Light rain showers and breezy (3-8°C)
  3rd Feb: Sunny intervals and breezy (4-9°C)
  4th Feb: Light rain showers and breezy (4-9°C)
  5th Feb: Light rain and breezy (4-9°C)
  6th Feb: Light rain and breezy (5-10°C)
  7th Feb: Light rain and breezy (5-9°C)
  8th Feb: Light rain and breezy (5-10°C)
  9th Feb: Light rain and breezy (5-9°C)
10th Feb: Light rain and breezy (7-10°C)
11th Feb: Light rain and breezy (7-9°C)
12th Feb: Light rain and breezy (7-8°C)
13th Feb: Light rain and breezy (6-7°C)
14th Feb: Light rain and breezy (6-7°C)
So that's pretty good. The forecast was right on 1st February, then wrongly optimistic on 3rd February (the only day when a 'sunny' symbol was displayed). But since 5th February the text summary for 14th February has been correct, which may be a coincidence, or is damned good going. The temperatures weren't quite so well targeted, and only in the last few days did the range close in on "cold, and not varying much". But overall, if you were planning an outdoor event, thumbs up.

But the most revolutionary feature is 14-day hourly forecasts, so let's dig into how they performed.

What these tabular graphics should have shown for 14th February was a dry morning and a wet afternoon, with the first rain arriving around noon. Here's what they actually predicted on the previous 13 days.
  1st Feb: sunny intervals, then light rain from 2pm
  2nd Feb: sunny intervals, with showers around 2pm
  3rd Feb: sunny intervals all day
  4th Feb: sunny intervals, with a shower at 1pm
  5th Feb: light rain all day
  6th Feb: light rain all day
  7th Feb: light rain all day
  8th Feb: sunny intervals, then showers from 1pm
  9th Feb: light rain all day
10th Feb: light rain all day
11th Feb: cloudy, then light rain from 2pm
12th Feb: clouding over, then light rain from 2pm
13th Feb: clouding over, then light rain from 2pm
14th Feb: clouding over, then light rain from 12 noon
And that's less good. The general shape of yesterday's weather only became clear on 11th February, having been too pessimistic for the previous six days, and was way off on 3rd February. That 1st February forecast still looks good, but with so many other different predictions floating around, who was to know at the time?

What the forecast never got right was that the rain would arrive at noon. Indeed even at 10am yesterday morning the BBC website was still confidently predicting the first drops would fall at 2pm. If the timing of a band of rain is that hard to predict even two hours ahead, why is anyone bothering a fortnight earlier?

And this is where another innovation, the ‘chance of rain’, is supposed to help. Meteo Group's computers churn away to produce a probability estimate of rain at a particular point in the future, and these are also displayed on the BBC website up to 14 days in advance.
0% = definitely dry
100% = definitely wet
20% = "out of 100 situations with similar weather, it should rain on 20 of those, and not rain on 80" (so, probably dry)
Here's the ‘chance of rain’ for noon on February 14th, as predicted over the previous fortnight.
  1st Feb: 13%
  2nd Feb: 12%
  3rd Feb: 13%
  4th Feb: 14%
  5th Feb: 30%
  6th Feb: 32%
  7th Feb: 27%
  8th Feb: 18%
  9th Feb: 14%
10th Feb: 28%
11th Feb: 11%
12th Feb: 9%
13th Feb: 4%
14th Feb: rain arriving
Being probabilities, it's never possible to say these were definitely wrong. But they're certainly not very good probabilities, with every single prediction suggesting it'd probably be dry at noon, whereas it fact it was just starting to be wet. The really bad forecast is that for 13th Feb, the day before, with a wildly over-optimistic 4%.

What's particularly intriguing is how the percentage for "rainfall at noon" changed over Valentine's Day morning.
14th Feb (9am): 9%
14th Feb (10am): 14%
14th Feb (11am): 81%
14th Feb (noon): 96%
If you'd checked the weather forecast at 10am, you'd have assumed it probably wasn't going to rain at noon. But by 11am a massive recalculation had taken place, and now it very probably was.

BBC Weather's FAQ page says "As MeteoGroup forecasts take advantage of hourly updates, which include real-time information from radar, satellite, and nearby weather station observations, you may notice the probabilities changing in the short-term (next 2-3 hours)." And change they do, indeed I can see it all over yesterday afternoon's figures, with percentages suddenly shooting up above 50% a few hours before the rainfall they're supposed to be predicting.

Another thing I've noticed is that the BBC's former forecasters, the Met Office, appear to calculate these percentages very differently. Here's their rainfall forecast published at 11am yesterday morning, with Meteo Group's forecast lined up underneath.

For a start, the Met Office only give values to the nearest 10%, which seems much more sensible than Meteo Group's spurious accuracy. More importantly, Meteo Group's percentage hits 81% at noon before settling into 30-60% for the afternoon, whereas the Met Office sticks to 10% until 4pm, then shoots up to nigh-certain at 6pm. The weather symbols are wildly different too. Why do these two predictions vary so much?

What seems to be happening is that the two forecasters have different opinions on what counts as rain. Yesterday's rain was only spits and spots from noon until around 4pm, indeed you might not even have counted it as proper precipitation. Then more showery rain continued until 6pm, at which point a much heavier band of rain arrived and continued for most of the evening. Meteo Group counted all of this as "single raindrop", with no hint of how wet it was going to be. But the Met Office differentiates a lot more, with 'spits and spots' not registering at all, and more relentless rain meriting a "double raindrop". I'd far rather have seen the top forecast than the bottom forecast if I'd been heading outside.

It seems the rainfall symbols in the BBC's online weather forecast don't mean the same as they did before, and you're a lot more likely to see a single raindrop when previously you'd have seen two, or none at all. Meteo Group appear to want us to forget intensity and learn to use their percentages instead, but when those percentages are often wildly inaccurate until a few hours beforehand, and most of the population doesn't understand numerical probability anyway, that doesn't seem very likely.

As for hourly weather forecasts 14 days in advance, these appear to be little more than a gimmick. I'm basing this on a single day's analysis, of course, which might be considered scant evidence. But when the BBC website is predicting that the whole of the last week of February will be "sunny intervals (3-9°C)", perhaps take that with an enormous pinch of salt.

<< click for Newer posts

click for Older Posts >>

click to return to the main page

...or read more in my monthly archives
Jan19  Feb19  Mar19  Apr19  May19  Jun19  Jul19  Aug19
Jan18  Feb18  Mar18  Apr18  May18  Jun18  Jul18  Aug18  Sep18  Oct18  Nov18  Dec18
Jan17  Feb17  Mar17  Apr17  May17  Jun17  Jul17  Aug17  Sep17  Oct17  Nov17  Dec17
Jan16  Feb16  Mar16  Apr16  May16  Jun16  Jul16  Aug16  Sep16  Oct16  Nov16  Dec16
Jan15  Feb15  Mar15  Apr15  May15  Jun15  Jul15  Aug15  Sep15  Oct15  Nov15  Dec15
Jan14  Feb14  Mar14  Apr14  May14  Jun14  Jul14  Aug14  Sep14  Oct14  Nov14  Dec14
Jan13  Feb13  Mar13  Apr13  May13  Jun13  Jul13  Aug13  Sep13  Oct13  Nov13  Dec13
Jan12  Feb12  Mar12  Apr12  May12  Jun12  Jul12  Aug12  Sep12  Oct12  Nov12  Dec12
Jan11  Feb11  Mar11  Apr11  May11  Jun11  Jul11  Aug11  Sep11  Oct11  Nov11  Dec11
Jan10  Feb10  Mar10  Apr10  May10  Jun10  Jul10  Aug10  Sep10  Oct10  Nov10  Dec10 
Jan09  Feb09  Mar09  Apr09  May09  Jun09  Jul09  Aug09  Sep09  Oct09  Nov09  Dec09
Jan08  Feb08  Mar08  Apr08  May08  Jun08  Jul08  Aug08  Sep08  Oct08  Nov08  Dec08
Jan07  Feb07  Mar07  Apr07  May07  Jun07  Jul07  Aug07  Sep07  Oct07  Nov07  Dec07
Jan06  Feb06  Mar06  Apr06  May06  Jun06  Jul06  Aug06  Sep06  Oct06  Nov06  Dec06
Jan05  Feb05  Mar05  Apr05  May05  Jun05  Jul05  Aug05  Sep05  Oct05  Nov05  Dec05
Jan04  Feb04  Mar04  Apr04  May04  Jun04  Jul04  Aug04  Sep04  Oct04  Nov04  Dec04
Jan03  Feb03  Mar03  Apr03  May03  Jun03  Jul03  Aug03  Sep03  Oct03  Nov03  Dec03
 Jan02  Feb02  Mar02  Apr02  May02  Jun02  Jul02 Aug02  Sep02  Oct02  Nov02  Dec02 

eXTReMe Tracker
jack of diamonds
Life viewed from London E3

» email me
» follow me on twitter
» follow the blog on Twitter
» follow the blog on RSS

my flickr photostream