[HN Gopher] Google says AI weather model masters 15-day forecast
___________________________________________________________________
Google says AI weather model masters 15-day forecast
Author : lemonberry
Score : 86 points
Date : 2024-12-06 19:41 UTC (4 days ago)
(HTM) web link (phys.org)
(TXT) w3m dump (phys.org)
| BrawnyBadger53 wrote:
| I thought I remembered this model being released many months ago?
| Or is this a new update to the model?
| xnx wrote:
| New model. Much better than the last one.
| dylan604 wrote:
| It's so new that it has no long term track record. I want to
| see how its record looks after say 1/3/5 years. Any 15 day
| forecast is just too far into the future for me to take
| seriously. Even the 3 day forecast is loose at best. By the
| time it reaches 7-10 days, some of the forecast is completely
| off when it reaches the 3 day window.
|
| "But DeepMind said GenCast surpassed the precision of the
| center's forecasts in more than 97 percent of the 1,320 real-
| world scenarios from 2019 which they were both tested on."
|
| Great, so it's using historical data to make a prediction
| about what has already happened. I want to see it on current
| data where it is truly a prediction with no data to make that
| prediction.
|
| I'm not convinced the results haven't been juiced/salted
| giarc wrote:
| >Great, so it's using historical data to make a prediction
| about what has already happened.
|
| That's ok though because it doesn't know it's acting on old
| data. Researchers would give it data and ask for the 15 day
| forecast, the researchers then compare against that real
| world data. And as noted, "it surpassed the centres
| forecast in more than 97 percent" of those tests. This is
| all referred to as backtesting.
| IshKebab wrote:
| It's still not as good as _actually_ new data. The
| individual model may not overfit the existing data, but
| the whole system of researchers trying lots of models,
| choosing the best ones, trying different hyperparameters
| etc. easily can.
| zactato wrote:
| yeah, but weather patterns data isn't different now than
| 10 years ago, right? right? ...
| staunton wrote:
| Overfitting is always bad by definition. The model learns
| meaningless noise that happens to help getting good
| results when applied to old data (be it due to trained
| weights, hyperparameters, whatever) but doesn't help at
| all on new data.
| gpm wrote:
| In principle they trained on data up to 2017, validated
| (tried different hyper parameters) on data from 2018, and
| published results on data from 2019...
| dagw wrote:
| I wish these sort of papers would focus more on the 3
| percent that it got wrong. Is it wrong by saying that a
| day would have a slight drizzle but it was actually sunny
| all day, or was it wrong by missing a catastrophic rain
| storm that devastated a region?
|
| I've worked on several projects trying to use AI to model
| various CFD calculations. We could trivially get 90+
| percent accuracy on a bunch of metrics, the problem is
| that its almost always the really important and critical
| cases that end up being wrong in the AI model.
| gyrovagueGeist wrote:
| Yep, this is also the problem that self-driving cars have
| with the "accidents per mile" metric.
| zamadatix wrote:
| 2019 would already be a post-training data prediction:
|
| > GenCast is trained on 40 years of best-estimate analysis
| from 1979 to 2018, taken from the publicly available ERA5
| (fifth generation ECMWF reanalysis) reanalysis dataset
| mmmrtl wrote:
| Yeah the preprint was last year
| https://arxiv.org/abs/2312.15796
| xnx wrote:
| 6 days ago: https://news.ycombinator.com/item?id=42319062
| bad_haircut72 wrote:
| Obviously this team knows way more about this donain than me but
| I have to ask, wouldnt this only be able to predict weather which
| is in line with past weather patterns/indicators? I can imagine a
| weather analyst might be able to see "between the data" and
| recognise when some anomaly might be brewing, but an AI model
| would not
| mjburgess wrote:
| Yup. There's fundamentally no way to automatically verify
| predictive accuracy, the assumptions in ML papers about such
| accuracy is (the almost always false presumption) that there
| are no distributional shifts in the data generating process.
|
| Here, since weather certainly changes its fundamental patterns
| over time, there is no way of reliably predicting out-sample
| performance.
|
| It's highly likely that in 1-2 years time we'd take all
| physics-based predictions and all predictions of current AI
| models, and find the AI model accuracy drops off a cliff when
| faced with a real out-sample.
| wcoenen wrote:
| Weather models can't compute some of the important physics
| that happens at sub-grid scale, and use various tricks to
| deal with that. How sure are you that these traditional
| models aren't also heavily tuned on past data to maximize
| performance? Perhaps they will also perform badly "out of
| distribution".
| mjburgess wrote:
| Causal physical models have no distributional requirements,
| so they are not sensitive to distribution shifts. ie., A
| causal model accounts for all possible distributions.
|
| The kind of risk with a causal model is that the model
| itself, of reality, is incorrect. This is a different kind
| of risk than there being 'no model at all' as in the case
| of curve-fitting over historical data, which is radically
| more fragile to expected shifts.
|
| In general, we're quite good at knowing the limitations of
| causal models, ie., specifying model risk here is much
| easier. You even, exactly, point out a known problem in the
| modelling which is impossible to state for an AI model.
|
| Since the AI model is just a weak induction, there are no
| terms/structures/etc. within that model which can be
| analysed to understand what parts of it are sensitive to
| what aspects of the system from which the data was taken.
|
| All we can say is, we know that in general, train/test
| distributions have to be "nearly exactly the same" for any
| of these methods to show anything like cross-val levels of
| accuracy. So we can very confidently predict when we know
| train/test wont be the same, that this is a mumbojumbo
| metric.
|
| Indeed, in the vast majority of common ML examples you can,
| right now, just go and look at real out-sample data
| collected later than the "common dataset" and you'll find
| the val accuracy is random or worse-than-random despite
| arbitarily high cross-val scores.
|
| The dataset which drives me most up-the-wall on this is
| house price datasets, or pricing datasets in general.
| Prices, generally, follow geometric brownian motion and
| nearly all ML models are extremelhy bad at modelling
| prices. So it's basically pseudoscience whenver anyone uses
| these datasets to demonstrate anything, esp. predictive
| accuracy.
| staunton wrote:
| Ultimately, a hybrid approach might win out in the end.
|
| Use equations for what we know, use machine learning to fit
| parameters in the equations, as well as terms we don't know.
|
| Prople nowadays can solve differential equations where some
| terms are "neural networks" and train those networks on data
| while numerically solving the equation. (some people call it
| "neural differential equations", if you want a search query
| to start)
| rtsil wrote:
| I know nothing about weather, but aren't changes happening
| gradually instead of overnight? There's no major anomaly that
| appears all of a sudden. In which case we can assume the every
| new change will be incorporated in the AI model's training.
| throwway120385 wrote:
| They happen pretty quickly sometimes in the PNW. The bomb
| cyclone we had in November developed within a day and wasn't
| very well predicted in advance.
|
| You can get a lot of traction with your assumptions but it's
| that 2-3% of predictions where being accurate matters a lot.
| gpm wrote:
| > But DeepMind said GenCast surpassed the precision of the
| center's forecasts in more than 97 percent of the 1,320 real-
| world scenarios from 2019 which they were both tested on.
|
| > The model was trained on four decades of temperature, wind
| speed and air pressure data from 1979 to 2018
|
| The test set is taken from after the training set. I agree it
| seems likely that it will need to be continuously trained on
| new data to maintain it's accuracy going forwards, but this
| seems like a promising sign that it should work as long as you
| keep updating the model.
|
| Of course the "can compute weather in 8 minutes" thing doesn't
| include the ongoing training cost... so the "this is much
| cheaper to run" claim seems slightly suspect.
| lispisok wrote:
| It's much cheaper to run for Google as long as the ECMWF
| keeps cranking reanalysis for training data and keep
| generating new initial conditions for forecasts which are
| generated by combining new observations and the previous
| forecast generated by the ECMWF physical models which is not
| cheap to do.
| atonse wrote:
| This is great from a practical standpoint (being able to predict
| weather), but does it actually improve our understanding of the
| weather, or WHY those predictions are better?
|
| That is my issue with some of these AI advances. With these, we
| won't have actually gotten better at understanding the weather
| patterns, since it's all just a bunch of weights which nobody
| really understands.
| CabSauce wrote:
| But that's the point of a weather forecast. We don't do them
| for the science. (I'm not arguing against weather research.)
| atonse wrote:
| I would think we also use these models to run simulations. So
| maybe the AI models can be used to run simulations with
| different kinds of inputs to see if doing X in one place
| (like planting a lot of trees in one area) will have an
| outsized impact rather than doing Y in another place.
| SpaceManNabs wrote:
| do you want a causal weather model or one that can predict the
| weather well?
| jacobgkau wrote:
| When climate change occurs (which it is), we're going to want
| a causal one so we can actually make a forecast instead of
| predicting only based on past (unchanged) data.
| hotstickyballs wrote:
| What does a causal weather model even mean?
| IshKebab wrote:
| He meant one that is physically based and interpretable,
| rather than a black box based on pattern matching.
|
| But I'll take the accurate black box any day, at least
| for weather forecasting. Climate modelling is a totally
| different thing.
| SpaceManNabs wrote:
| Doesn't exactly answer your question, but maybe this link
| will give enough intuition so that i can be lazy and
| hand-wave away an answer
|
| https://en.wikipedia.org/wiki/The_Book_of_Why#Chapter_1:_
| The...
| SpaceManNabs wrote:
| ahhhh i was hoping someone wouldn't mention this. i wanted
| to add a caveat but it made my comment look ugly lol.
|
| but like someone else says weather and climate models
| forecast on different scales and for different purposes
| usually.
| criddell wrote:
| Is that similar to being uncomfortable with quantum mechanics
| and the shut-up-and-calculate advice?
| sgt101 wrote:
| whelp...
|
| Ok, how would you articulate an understanding of how the
| weather works?
|
| Let's face it, it's not going to be easy to frame in natural
| language...
| yunwal wrote:
| You can take a meteorology course that explains how the
| weather works. That course is the articulation.
| jncfhnb wrote:
| That's great but it's not going to scale to the level of
| detail that makes this better than traditional meterologist
| models
| bobthepanda wrote:
| I think it certainly doesn't foreclose on knowing more in the
| future, based on what we observe with this new model.
| chaos_emergent wrote:
| How many people can scrutinize the esoteric ensemble of climate
| models that existing climatologists use to generate weather
| predictions today?
| jacobgkau wrote:
| The climatologists can, at least. Can they scrutinize the
| esoteric ensemble of weights making up this AI model? And
| which type of model's going to be easier to update based on
| changing climate parameters, a climate model or an AI model?
| dartos wrote:
| More than 0
| ianburrell wrote:
| Climatologists use climate models to predict the climate.
| Meteorologists use weather models to predict the weather.
| They are different time scales and disciplines.
| zaptheimpaler wrote:
| I think this will be the next generation of science to some
| extent. The things we can understand and call
| explanations/reasons might be something involving 5 or 50
| variables with not too many interactions between them. They
| have some unifying principles or equations we can fit in our
| head. I think many things inherently just involve far too many
| variables and complexity for us to understand in a neat theory
| and we are hitting those limits in biology & physics. Even so
| I'm sure we will develop better and better ways to interpret
| these models and get some level of understanding. Maybe we can
| understand them but not create them past a certain scale.
| dartos wrote:
| We can work backwards from correct predictions and develop an
| understanding that way.
|
| But most people just need to know if it's going to be storming,
| hot or, going to rain on a given day and that is where this
| shines.
| adriancooney wrote:
| We can always work backwards, regardless of AI.
| dartos wrote:
| Sure, but with this new predictive model we will have
| better predictions to work backwards from.
|
| OC was saying (I'm going to paraphrase) that this is the
| death of understanding in meteorology, but it's not because
| we can always work backwards from accurate predictions.
| wongarsu wrote:
| Or we could wait 15 days and work backwards from what the
| weather actually turned out to be.
|
| I guess there could be some value in analyzing what
| inputs have the most and least influence on the AI
| predictions.
| kolinko wrote:
| Define "our understanding". With complex / chaotic systems
| there sometimes are no higher level laws that govern them - all
| we have is just modeling and prediction.
| raincole wrote:
| A fisherman was relaxing on a sunny beach, enjoying the day
| with his fishing line in the water. A businessman, stressed
| from work, walked by and criticized him for not working harder.
|
| "If you worked more, you'd catch more fish," the businessman
| said.
|
| "And what would my reward be?" asked the fisherman with a
| smile.
|
| "You could earn money, buy bigger nets, and catch even more
| fish!" the businessman replied.
|
| "And then what?" the fisherman asked again.
|
| "Then, you could buy a boat and catch even larger hauls!" said
| the businessman.
|
| "And after that?"
|
| "You could buy more boats, hire a crew, and eventually own a
| fleet, freeing you to relax forever!"
|
| The fisherman, still smiling, replied, "But isn't that what I'm
| already doing?"
| joe_the_user wrote:
| But AI models in no way free you from work and they are
| hardly relaxing.
|
| If anything, they "free" people from understanding but that's
| an activity that many if not most people value highly.
| raincole wrote:
| Prediction is the only way to test whether the
| "understanding" has any value.
|
| You can create an understandable physical model. It can be
| mathematically aesthetic. But if it can't predict the
| results of real world experiments, it's just "not even
| wrong".
| jacobgkau wrote:
| I _think_ that might have been their point. People moving
| work to AI so they can "relax" by working on more
| complicated technical matters (or more AI) are the
| businessmen, and the meteorologists just chilling out
| predicting the weather as best as they can with science are
| the fishermen.
|
| Edit: Just saw their reply to you, so maybe I was wrong
| about the parable coming across wrong.
| efxhoy wrote:
| This has been the case for years now, way before the AI craze.
| We just used to call it machine leaning. The best performing
| predictive models are black boxes which can't practically be
| interpreted by humans the same way you can say a linear
| regression model that gives easily digestible parameters as
| output. Boosted trees are a great example of very well
| performing models that quickly become impossible for humans to
| understand once they get big enough to be useful.
| Always42 wrote:
| Based on the Gemini release it's hard to take what Google claims
| at face value.
| amazingamazing wrote:
| Totally different constraints. It's like saying Nobel prize
| winning physicists can't be trusted because they get basic
| arithmetic wrong
|
| Gemini is fast food - it's not supposed to be the best.
| itake wrote:
| what about Google's flu prediction model release?
|
| https://blogs.ucl.ac.uk/pcph-blog/2018/01/23/google-flu-
| tren...
| moralestapia wrote:
| Your analogy doesn't make sense.
|
| Nobel winning physicists can definitely perform arithmetic.
| Maxatar wrote:
| It's trendy for accomplished people to talk down about
| themselves as a way to sound cute. It's similar to software
| developers who like to say "I don't know what I'm doing, I
| just Google and Stackoverflow all day." It has a certain
| charm to it, and certainly there are some people for whom
| it's true, but overall it's just a misguided attempt at
| being modest but ultimately a horribly misleading
| statement.
|
| "Do not worry about your difficulties in mathematics, I
| assure you that mine are greater" - Albert Einstein
| rurp wrote:
| A Nobel physicist that couldn't do basic arithmetic would
| definitely raise my eyebrows, but even taking your analogy at
| face value Gemini was _not_ marketed as fast food slop.
| Google can 't be trusted to hype products in a reliable way,
| regardless of their technical details.
| petesergeant wrote:
| Almost as bad as stuff branded IBM Watson for over promising
| fldskfjdslkfj wrote:
| what did Google claim about Gemini?
| computerex wrote:
| Google's work on Alphafold produced a Nobel prize.
| crowcroft wrote:
| Aside from the fact you're conflating two very different things
| comparing this to Gemini, what exactly is the problem with the
| Gemini?
|
| Specifically just that the release was a bit awkward and had
| some problems? I've found the latest model releases to be very
| good compared to other frontier models.
| kolinko wrote:
| Their Alpha* work from DeepMind is actually quite good and has
| a good track record. LLM/Gemini - yeah, what you said, I
| wouldn't trust a word their team says.
| wslh wrote:
| I really wonder if, like the weather, we should see new financial
| prediction models, understanding that there is randomness but
| also patterns.
| brcmthrowaway wrote:
| Meteorologists are quaking in their boots!
| dumbfounder wrote:
| "More accurate forecasts of risks of extreme weather can help
| officials safeguard more lives, avert damage, and save money,"
| DeepMind said.
|
| Did the DeepMind AI say this?
| magicalist wrote:
| ...why else do you think it's important to predict hurricane
| paths and tornado spawning storms and flooding rains and heat
| waves further ahead of time?
| ericra wrote:
| It does seem like this is one of those domains where new AI
| models could thrive. From my understanding, the amount and
| variety of data necessary to make these models work is huge. In
| addition to historical data, you've got constant satellite data,
| weather stations on the ground for data collection, weather
| balloons going high into the atmosphere multiple times daily per
| location, Doppler radars tracking precipitation, data from ships
| and other devices in the ocean measuring temps and other info,
| and who knows what else.
|
| It's incredible that we are able to predict anything this far
| into the future, but the complexity seems like it lends itself to
| this kind of black box approach.
|
| *This is all speculation, so I'd be grateful if someone more
| knowledgeable could tell if if I'm mistaken about these
| assumptions. It's an interesting topic.
| BadHumans wrote:
| I'm friends with a meteorologist and the 15+ day forecast is the
| bane of their existence because you can't accurately forecast
| beyond a week so I would love to know how they are measuring
| accuracy. The article doesn't say and I know the paper is going
| to go over my head.
| n4r9 wrote:
| I would guess that everyday they're comparing the current
| weather against the forecast from 15 days ago. Not a lot of
| data points to be sure, but perhaps enough to have confidence
| of very high accuracy.
| dgrin91 wrote:
| Alternatively the can do back testing - using historical data
| they feed a subset into the predictor, then compare it's
| predictions to actual history
| diego_sandoval wrote:
| I don't think that's necessary.
|
| You can backtest the weather for any point in the past, as
| long as you only use the data that was available until 15
| days before the day of the prediction.
| choeger wrote:
| Is that a prediction in the form of "There's a 30% chance of
| light rain that day." or "Temperature will reach 22,5degC at
| 14:00."?
| fancyfredbot wrote:
| It takes 8 minutes to produce a 15 day forecast. That's actually
| quite a long time for an AI model. I should probably read the
| paper to find out why but does anyone know? Is the model
| predicting the weather in 10 minutes time and just run
| iteratively 2000 times for a 14 day forecast?
___________________________________________________________________
(page generated 2024-12-10 23:00 UTC)