▲ 189 points
back
112 comments
Years ago, I set up a simple website that screen-scraped the BBC's weather predictions, and compared them against the day's weather report to calculate a very crude and basic accuracy.
For the UK towns it monitors, a dumb prediction of "tomorrow's weather will be the same as today's" gives a 34% accuracy - which only falls to about 25% when predicting the weather for next week! Luckily, the proper weather forecasters do a bit better than this :)
https://weather.slimyhorror.com/
Excuse the basic site, I set this up over 17 years ago, and with minimal tweaks it has been left to its own devices since then.
The stats also compare the BBC accuracy over the last year vs all time, and it seems that they are getting better - I wonder if new AI techniques will really make a big leap in predictions or whether they are just more incremental improvements.
3rd Hand anecdote that I liked regarding this:
During World War II, [Nobel laureate, Ken] Arrow was assigned to a team of statisticians to produce long-range weather forecasts. After a time, Arrow and his team determined that their forecasts were not much better than pulling predictions out of a hat. They wrote their superiors, asking to be relieved of the duty. They received the following reply, and I quote "The Commanding General is well aware that the forecasts are no good. However, he needs them for planning purposes."
Via http://www.investorsinsight.com/blogs/john_mauldins_outside_...
Ok ... I am directly going to piggyback on this:
1. Subjectively this years weather predictions have been way off, compared to the years before. I heard several theories on that: (a) the year was extraordinary (less cars, less flights) and (b) predictions were worse because data from plane based weather radar was missing. -> Does anybody know if my subjective feeling is based in reality? And if true, what are the reasons?
2. Again subjectively, but I feel like most of my weather based decisions are "do I leave now or do I wait for the rain to pass". That question is answered pretty well by looking at the weather radar maps myself. I feel like an statistical/ML/AI approach that combines what was the weather yesterday and what is the weather in the surrounding cities should fair pretty well.
Very cool! Nothing wrong with crude; something crude that exists is better than something polished that does not exist!
I am curious about your implementation of 'accuracy':
> How do I measure 'accuracy'?
> Very simply! I take the BBC's weather icons and compare them, using a bit of leeway. So if the prediction is 'Partly Cloudly', then 'Sunny Intervals' is also considered equivalent. Likewise, 'Light Showers', 'Light Rain' and 'Drizzle' are all considered close enough to be an accurate forecast.
> E.g. as I write this, the table below shows that the weather forecast for Cambridge one day ahead was 53% accurate. In other words, the BBC's guess about tomorrow's weather in Cambridge was right roughly half of the time.
So no partial credit, then? Check my understanding: I think that you're simply matching the title text of the icon. If it's a match (or in a small group of synonyms) that's a point, if it's not, you score zero for that prediction. Yesterday, the forecast for today was "Partly cloudy", today, the actual weather was "Sunny" - it gets no credit.
The parent article neural network is, apparently, scoring itself on matching the radar results pixel by pixel and color by color, which is pretty neat. I think it's particularly interesting if it's essentially general-purpose, taking in one collection of input pictures and outputting another, or whether they also gave it information on high and low pressure zones, prevailing winds, bodies of water and elevated land masses, and so on.
Regardless, what I personally want to know (and what I think most people want to know) from the weather forecast is whether it's going to be suitable for a particular activity. Obviously, the hard part is that the activities may vary for each consultation. If it's predicted to be partly cloudy and mild, and was actually sunny and hot, I'd be pleasantly surprised if I had scheduled a day at the beach, but disappointed if I was sweating while working on some landscaping. Farmers want it wet in the summer for growth and dry in the fall for harvesting, sailors want to know the minimum wind, painters want to know the maximum wind; everyone has different goals day by day.
A basic site, sure, but eminently readable. Kudos!
Have you ever graphed the accuracy over time for the years you've been doing it? It would be interesting to see if there's a trend in forecasting improvement.
Love the design! Those table borders really take me back
This is really cool!
I wonder when they're going to tackle macroeconomic forecasting? It seems like a good candidate - a complex system with too many variables for analytical models to be very good, and indications that there are patterns and connections that we don't even understand theoretically yet, but which might be there in the data.
I guess for all we know, a sophisticated ML model like that might already exist in some hedge fund, but they'd be keeping quiet about it so they don't lose their edge.
Careful, dragons be there.
Day trading securities is a smaller problem than macroeconomic forecasting, probably... but down this path lies Asimov's psychohistory. It's basically forecasting history.
The problem is as soon as new predictions are made it causes the people participating in an economy to change their behavior.
Thats why people working at central banks like the Federal Reserve will try to downplay risks like inflation to prevent signalling that theyll raise interest rates causing a market correction or businesses from raising prices to deal with it.
I think the cats out of the bag now but they sure would love to stuff it back in the bag.
Hmm, but aren't most macroeconomic big swings triggered by Blackswan events? And aren't those, per definition, virtually impossible to predict?
Renaissance Technologies, Jim Simons and Bob Mercer’s fund solved the market with ML techniques ages ago. Can’t remember off the tip off my head but they have consistent 60% annual returns which is unheard of. There’s a great book about it - The Man who Solved the Market
A bigger problem though is the feedback loop between the model and policy makers in this context.
For instance, the model predicts 2023 GDP to decrease so we take all out measures to cause this to not happen.
Was the model wrong? How would you know?
In this context, the model would need to predict the actions by the Fed but the Fed would be the ones using the model.
So you would also have to predict the Feds reaction to the prediction the model made of the Fed's reaction to the model..on and on.
Trying to compete in markets is like Goodhart's law on steroids.
People use machine learning, but nothing this sophisticated.
Outside of the market makers, most of the successful firms are just selection bias.
>I wonder when they're going to tackle macroeconomic forecasting?
That was Dark Sky's play - and it was amazingly fruitful. It would even leverage the barometer in most phones. It was a bit disappointing when Apple acquired them since it cuts non-iOS users off, but it was a great app when it was out. For iOS users as of iOS 15 Apple's weather widget is now based on Dark Sky's tech making their app redundant.
A bit disappointed to be honest. They produced a bunch of models and then checked with a team of meteorologists what they thought of them vs the other models? This sounds more like GPT-3 writing sonets and getting a bunch of poets to evaluate them. Why not just check the predictions?
TBF, going by the problem statement in the article, the objective function could be pretty wacky. For example, do you weight accuracy by location? Is it more important to get your predictions right over sports stadiums than over residential areas? Would it be useful to weight accuracy by time, like, it's more important to get predictions right at 7 AM when people are driving to work than it is to get it right at 1 AM when everyone is asleep? I don't think that that's what's actually happening, but I do think that weather is a sufficiently ugly problem that comparing to human performance is useful.
Link to Nature paper that the article is based on: https://www.nature.com/articles/s41586-021-03854-z
This is a bit over-hyped and not exactly a breakthrough. It’s not doing true weather prediction but rather extrapolating the movement of radar images. This is nothing new. I remember as far back as the 1990s TV weathermen would draw a line across a line of moving rain and the computer would predict which town it would get to and when.
Slapping “AI” on this 25 years later is a good example of the whole present PR move of labeling things as “AI” that are just rather basic data analytics.
>This is a bit over-hyped and not exactly a breakthrough. [...] This is nothing new.
>Slapping “AI” on this 25 years later is a good example of the whole present PR move of labeling things as “AI” that are just rather basic data analytics.
The New Atlas article has a link to the Nature journal paper it's based on. Your dismissal and summary of DeepMind's work described by the article as a "public relations move" is a disservice to readers.
The more detailed explanation in the Nature journal describes a new technique of "deep generative models" applied to weather radar. This was not available 25 years ago. In tests, their DGM forecasts became preferred by meteorologists 93% of the time for accuracy compared to previous "data analytics". Excerpt from Nature:
>We use a single case study to compare the nowcasting performance of the generative method DGMR to three strong baselines: PySTEPS, a widely used precipitation nowcasting system based on ensembles, considered to be state-of-the-art3,4,13; UNet, a popular deep learning method for nowcasting15; and an axial attention model, a radar-only implementation of MetNet19
>[...] When expert meteorologists judged these predictions against ground truth observations, they significantly preferred the generative nowcasts, with 93% of meteorologists choosing it as their first choice (Fig. 4b).
I mean... You can look at the paper, they actually do use a novel generative AI model, so it's rather strange to criticize this being labeled AI.
And, the paper also shows the new model to outperform existing ones in 84% of cases according to a bunch of human forecasters, so calling this just PR is overly cynical IMO.
As far as "It’s not doing true weather prediction but rather extrapolating the movement of radar images.", both the paper and article say the paper is tackling short term rain prediction ('precipitation nowcasting'), so it's not oversold as far as I can see.
This is a very simplistic view. The complexity of weather prediction is a factor of both how far into the future and how fine-grained the prediction is. Predicting 1 minute ahead is trivial. Predicting temperature with a 20-degree accuracy a week ahead is trivial. What Deepmind does here is predicting precipitation ~1 hour ahead with a ~1km resolution, and they do it significantly better than existing models. Perhaps not a breakthrough, but a substantial technical development it is.
> I remember as far back as the 1990s TV weathermen would draw a line across a line of moving rain and the computer would predict which town it would get to and when.
Did it do it as accurately as this new method?
I think it was in an interview with someone from the ECMWF where they claimed that for each decade of progress, the accuracy of their models improved by one day. So we're currently at around three days of good forecasting.
These models do not use AI, they work by extrapolating, like you say.
If you have access to DWD's RADOLAN image data, for example as rendered images through the DWD WarnWetter-App (you need to pay a small one-time fee to access the radar data), you can clearly see how much this extrapolation leaves to be desired (even though it is extremely useful as it is). Actually, almost every German weather data provider which offers radar precipitation predictions is based on the raw data provided by DWD, this raw data can also be downloaded for free at https://opendata.dwd.de/weather/radar/radolan/rw/
Anyway, if you look at the predictions, they are pretty simple. As if the wind direction at two different altitudes is determined for each point, and then applied to the current precipitation data.
These wind vectors don't change during the (short term, max 2h) prediction, so you see the parts of the image moving at a constant velocity as soon as you're talking about the future.
This neglects two things: wind direction will change during these two hours, which is why you as the app user need to check often to verify if it is still accurate, but most importantly this simple model does not take into account the humidity in the air. So sometimes the rain will arrive sooner not because the wind got faster, but because new clouds are starting to build faster in your direction than the old cloud systems get to travel towards you with the wind.
And in both these cases AI provides a significant potential of improvement. By looking at more of the surrounding weather dynamics it will be able to predict better what is actually happening in the weather system. Currently we can only improve this by adding more sensors and more frequent radar scans, but AI can really start to interpret the past one-hour-weather and "understand" what is happening there in order to predict what will happen later. And there is a ton of data available for training.
Yep, unless they came up with a way to make the primitive equations solutions significantly computionally cheaper, you’re not getting any more accurate without better turbulence models.
Possibly, but this is a DeepMind result. They've had enough non trivial successes to earn some credibility. Even if it is hype, more understandable than most overhyped PR headlines.
I don't know much about meteorology -- what is "true weather prediction" then?
The top HN comments on anything related to ML are comically pessimistic and also extremely repetitive.
I do this with my own eyeballs looking at the rainfall radar.
It would be interesting if this could be made to forecast beyond what I do to avoid getting wet on my commute, but 60 minutes is good enough already, for me.
During winter prediction is easy. You just look at where the rain is, see where it is going and multiply the speed by the delta-time.
During summer, rain can suddenly form out of nowhere, making predictions much harder.
Living in California, I find the weather a lot more complicated than when I was living in a much flatter and consistent area of the country. Just within my city the variance is so large that any forecast that just says "Los Angeles" is just an average that doesn't exist in reality at all. In Marina del Rey it could be 60, cold, grey, windy, even raining, then you go seven miles northeast to hollywood and its sunny, 85, hot, without a cloud in the sky, no cool ocean breeze, then you go through the cahuenga pass and in a 10 minutes drive the temperature goes up another 10 degrees by the time you are in north hollywood. Then if the winds decide to shift and you get some Santa Anas blowing in, everything can turn on its head fast and its hot in marina del rey even at night.
Even with a storm moving directly above it could do remarkably different things whether you live in a flatter side of town or one on a hillside, which usually sees precipitation and even hail or sleet along with colder temps while it might remain bone dry in the flatter parts.
Accurate weather estimations for some places needs a very robust understanding of local topology, seasonal winds, and data, lots of data, from sensors that aren't there in enough quantities and in enough places to capture what is actually happening over varied terrain and changing conditions. I found localized apps like darksky very accurate in the midwest where weather is uncomplicated to model beyond occasional things like lake effect snow (which seemed to be well understood), but not very useful in Los Angeles where you practically need your own hardware to actually quantify what the weather is where you are at in your particular canyon today.
Predicting only two hours ahead doesn't seem that impressive or helpful to me in itself; major weather-related decisions usually need more time. Still, interesting stuff!
A few family members and I usually get little headaches hours before rain if it's preceded by a dip in air pressure, although I've never measured the accuracy or utility of this.
That's really interesting about the headaches. It makes sense.
The model is actually useful for google. Road traffic is closely linked to weather, with some routes worse impacted than others. If you predict the weather, you can predict changes in congestion patterns caused by the rain, so you can predict journey times better. Most journeys people are using google maps for are probably in the 30min-2hr range.
It's also simply interesting because predicting the progress of frontal rainfall is not something we're good at. We can apply conventional extrapolation, but this only considers the direction of the weather, not at all the changing saturation of the clouds.
Predicting 2 hours ahead is one of the most useful weather forecasting tasks. 2 hours is enough time to prepare for rain in loads of situations.
And sure the title is a bit ridiculous. Anyone can predict incoming rainfall. The question is can it predict it accurately. The abstract says:
> we show that our generative model ranked first for its accuracy and usefulness in 89% of cases against two competitive methods
Probably a bit better than your headaches.
One obvious thing that came to mind immediately is something like racing, for example F1. Knowing what tyres to put on is a make or break there. If DeepMind truly is better than anything else there, I'd imagine F1 to be one of the first to start using it.
I think farmers would like to know what is happening during the next hour or two during harvest.
Not talking specifically about this one, but to me it seems DeepMind is producing more higher quality research and breakthroughs than other parts of Google. I wonder why that is, it's not like other parts of Google are lacking in talented ML researchers.
Well... DeepMind is focused on very abstract research and publication. There aren't many projects of this scale operating with these goals.
Waymo's goals, for example, probably focus less on abstract research and publication. They exist to build a thing and make it a business eventually. Whatever AI research they do exists to support that. Waymo are really big, so that can still be a lot.
Most AI projects google have are probably smaller, and/or less publication focused. Also newer. Deepmind is >10 years old.
Google bought Deepmind when they already had impressively successful results. Once the bought the company, they started to apply stuff from Alphago to anything they could think of with Google level access to computing resources. Within a few years they started having successes in some of these areas. This is basically a project coming to fruition.
I think the project has a very "make it more general" orientation and their approach yields a "hey look, it works for this now" success story at regular intervals.
Start to worry when intervals hit the ohm frequency and deep thought pops into existence asking for a sandwich .
There seem to be less PR about those, but the rest of Google is doing a lot: https://research.google/pubs/
Agreed, few things are as flashy as protein folding :)
Wow, I really feel like this is an example of how quickly people adapt to and become accustomed to novel technology. DeepMind is legendary of course but the output of Google Brain is also just bananas. I can point my mobile phone camera at a Russian newspaper and get it translated into my language. That’s practical! The speech recognition on android, the Pixel camera, smart reply/compose in gmail, many other practical applications. And, brain team publishes constantly. Dozens of papers every year.
I've had this for years with DarkSky - who Apple bought and now the default weather app in iOS is based on.
When it first came out I can remember being at a friends baseball game - it was sunny but I got an alert that rain was going to start in 10 minutes. So I went and got an umbrella from the car. Got some snide comments - until 8 minutes later when it started pouring down rain!
It's not perfect, but it is spooky at how many times it is spot on with rain within the next hour. The more systems offering this kind of service the better since it is very useful!
While this would obviously be advantageous for the vast majority of situations, I also can't help but be annoyed how it may have a detrimental effect on racing. Just a few days ago we had some of the best action in a long time due to the teams predicting the rain differently [1].
A bit disappointed to be honest. The produced a bunch of models and then checked with a team of meteorologists what they thought of them vs the other models? This sounds more like GPT-3 writing sonets and getting a bunch of poets to evaluate them. Why not just check the predictions?
It would be great if they try to predict incoming typhoons formations, this will greatly help us anticipate and measure how dangerous the upcoming danger is.
I recently received a response to one of my comments saying that GPT-3 can't predict the future. From a naive understanding of the basic way it works, if it can generate subsequent words and phrases that follow the rules of a language based on tokens, couldn't it be trained on larger concepts if those are represented as tokens?
I understand that simulations are ultimately constrained by their assumptions, eg how the ultimate stable state of Sim City games seems to be some sort of authoritarian police-state. [1] Couldn't we create a simulation of human history, etc, and iterate on the rules by backtesting, ie doing something similar to the way stock trading indicators are tested?
I do fear that AI could be used for nefarious purposes in that regard. "I want to see what needs to occur and what I need to do for this company to gain a monopoly in this industry." I'd be surprised if this isn't already occurring. If anyone has any links they can share about how AI is being applied to social spheres, that would be awesome.
1. https://www.polygon.com/videos/2021/4/1/22352583/simcity-hid...