Stephan RasppersName, Peter D. DuebenpersName, Sebastian ScherpersName, Jonathan A. WeynpersName (2020date). “WeatherBench: A Benchmark Dataset for Data-Driven Weather Forecasting.” Journal of Advances in Modeling Earth Systems / arXivtitle·j, MunichplaceName.
Object of study — Standardized evaluation of machine-learning methods for medium-rangeterm weather forecastingterm·dom.
Research question — How can data-driventerm·key weather-forecasting models be compared fairly against each other and against physical numerical weather predictionterm·dom?
Contribution — Provides WeatherBench, a curated benchmark datasetterm·key and scoring protocol for machine-learning weather forecastingterm·dom, together with baselines, enabling like-for-like comparison of data-driventerm·key and physics-based forecasts.
phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | data-driven medium-range weather forecasting — “Here we focus on purely-data driven prediction of the global atmospheric flow in the medium-range.” | |
| claim-type | method — “Here we present a benchmark dataset for data-driven medium-range weather forecasting, a topic of high scientific interest for atmospheric and computer scientists alike.” | |
| method — evidence | — “We provide a ready-to-use dataset for download along with specific metrics to compare different approaches.” | |
| transfer | from artificial intelligence (deep learning): deep learning / data-driven methods (developed for image recognition and NLP) applied to weather forecasting — “Data-driven approaches, most prominently deep learning, have become powerful tools for prediction in many” | transfers[0] |
| mechanism | temporal correlation shrinks the usable training set Hourly sampling over 40 years gives about 350,000 samples, but consecutive hours are not independent; assuming one new weather situation per day leaves about 15,000, which is the count that bears on overfitting. — “Another potential issue is the limited amount of training data available. 40 years of hourly data amounts to around 350,000 samples. However, the samples are correlated in time. If one assumes that a new weather situation occurs every day, then the number of samples is reduced to around 15,000” | mechanisms[0] |
| builds-on | deep learning (data-driven prediction) — “Data-driven approaches, most prominently deep learning, have become powerful tools for prediction in many” | claim_relations[0] |
| replicates | Scher and Messori (2019a) — “This confirms the findings of Scher and Messori (2019a) whose experiments showed that training with longer lead time yields better results than chaining together short-term forecasts.” | claim_relations[1] |
| builds-on | Hersbach et al. (2020), the ERA5 reanalysis dataset — “For the proposed benchmark, we use the ERA5 reanalysis dataset (Hersbach et al., 2020) for training and testing.” | claim_relations[2] |
| builds-on | Ebert-Uphoff et al. (2017), criteria for benchmark datasets — “Loosely following Ebert-Uphoff et al. (2017), the key features of this benchmark are:” | claim_relations[3] |
| sense | training samples - AS RAW COUNT: around 350,000, being 40 years of hourly data — “Another potential issue is the limited amount of training data available. 40 years of hourly data amounts to around 350,000 samples.” | senses[0] |
| sense | training samples - AS EFFECTIVE COUNT: around 15,000, once temporal correlation is taken into account under the assumption of one new weather situation per day — “If one assumes that a new weather situation occurs every day, then the number of samples is reduced to around 15,000.” | senses[1] |
| sense | success on this benchmark - defined by a SINGLE metric over two fields, which the paper presents as a virtue of the design — “Clear metric for success: We defined a single metric (RMSE) for two fields (500 hPa geopotential and 850 hPa temperature). These scores provide a simple measure of success” | senses[2] |
| sense | primary metric - chosen for computational convenience and for matching the ML community's loss function, not for meteorological importance; the paper states both reasons — “We chose the root mean squared error (RMSE) as our primary metric because it is easy to compute and mirrors the loss used for most ML applications” | senses[3] |
| sense | `extremes` - LEFT UNDEFINED ON PURPOSE, AND THE BENCHMARK DROPS THE PHENOMENON RATHER THAN PICK A DEFINITION: `defining extremes is ambiguous which is why there is no standard metric for evaluating extremes ... we decided to omit extremes for now but users are encouraged to chose their own verification`. THE AMBIGUITY IS UPSTREAM OF THE METRIC AND THE MISSING METRIC IS UPSTREAM OF THE BENCHMARK'S SCOPE - a definitional problem propagating into what a whole field will be measured on. — “But defining extremes is ambiguous which is why there is no standard metric for evaluating extremes. The goal of this benchmark is to provide a simple, clear problem. Therefore, we decided to omit extremes for now but users are encouraged to chose their own verification of extremes.” | senses[4] |
| sense | Post-processing - ML correction of physical-model output bias — “ML can be applied to weather prediction in many different ways. Two long-standing applications of ML are post-processing – the correction of statistical biases in the output of physical models – and statistical forecasting – the prediction of variables not directly output by the physical model. Traditionally, this has been done using simple linear techniques but more recently modern machine learning approaches like random forests or neural networks have been explored (Gagne et al., 2014; Taillardat et al., 2016; McGovern et al., 2017; Lagerquist et al., 2017; Rasp and Lerch, 2018). Typically, these approaches target very specific variables or locations whereas the general evolution of the atmosphere is still predicted by a physical model” | senses[5] |
| sense | Statistical forecasting - ML prediction of variables absent from physical-model output — “ML can be applied to weather prediction in many different ways. Two long-standing applications of ML are post-processing – the correction of statistical biases in the output of physical models – and statistical forecasting – the prediction of variables not directly output by the physical model. Traditionally, this has been done using simple linear techniques but more recently modern machine learning approaches like random forests or neural networks have been explored (Gagne et al., 2014; Taillardat et al., 2016; McGovern et al., 2017; Lagerquist et al., 2017; Rasp and Lerch, 2018). Typically, these approaches target very specific variables or locations whereas the general evolution of the atmosphere is still predicted by a physical model” | senses[6] |
| sense | Nowcasting - radar-only precipitation extrapolation up to six hours — “Another application that has recently been explored using ML is nowcasting, which describes the short range (up to 6 hours) prediction of precipitation by directly extrapolating radar observation without a physical model involved” | senses[7] |
| sense | Hybrid weather model - physical dynamics with selected uncertain or expensive components replaced by ML emulators — “Yet another direction for ML research is hybrid modeling, in which a physical model is combined with data-driven components, for example replacing heuristic cloud or radiation parameterizations (Chevallier et al., 1998; Krasnopolsky et al., 2005; Rasp et al., 2018; Brenowitz and Bretherton, 2018; Yuval and O’Gorman, 2020). The key idea behind these approaches is to only replace uncertain (e.g. clouds) or computationally expensive (e.g. line-by-line radiation) model components with machine learning emulators and leave other model components (e.g. large-scale dynamics) untouched” | senses[8] |
| sense | WeatherBench medium range - direct global atmospheric-flow prediction at three- and five-day leads — “Here we focus on purely-data driven prediction of the global atmospheric flow in the medium-range. Specifically, we select lead times of 3 and 5 days, for which the atmosphere is still reasonably deterministic but also exhibits complex nonlinear behaviour, such as baroclinic instabilities and tropical cyclogenesis. This forecast range is important from a societal point of view because it delivers crucial information for disaster preparation, for example for flooding, cold and hot spells or damaging winds (Lazo et al., 2009). Creating a good medium-range forecast requires understanding complex atmospheric dynamics and the interplay between several variables across a range of scales. This sets this challenge apart from post-processing and statistical forecasting, in which the large-scale dynamics are predicted by a physical model, and nowcasting, in which the considered evolution is univariate and short-term. In other words, this benchmark closely emulates the task performed by physical NWP models” | senses[9] |
| sense | ERA5 reanalysis - forecast-model and observation synthesis used as atmospheric truth proxy — “For the proposed benchmark, we use the ERA5 reanalysis dataset (Hersbach et al., 2020) for training and testing. Reanalysis datasets provide the best guess of the atmospheric state at any point in time by combining a forecast model with the available observations. The raw data is available hourly for 40 years from 1979 to 2018 on a 0.25°latitude-longitude grid (721×1440 grid points) with 37 vertical levels” | senses[10] |
| sense | Train / validation / test - fitting data, tuning data, and held final-evaluation data — “However, there is the danger that through continued tuning of hyperparameters one unwillingly overfits to the validation dataset. Therefore it is advisable to keep a third, testing, dataset for final evaluations of model performance” | senses[11] |
| sense | Persistence forecast - tomorrow’s field set equal to today’s field — “The two simplest possible forecasts are a) a persistence forecast in which the fields at initialization time are used as forecasts ("tomorrow’s weather is today’s weather"), and b) a climatological forecast” | senses[12] |
| sense | Direct / iterative forecast - lead-specific model versus recursively applied short-lead model — “As a first purely data-driven baseline we fit a simple linear regression model. For the direct predictions a separate model was trained for each of the four variables. For this purpose the 2D fields were flattened from 32×64 → 2048. This was done for 3 d and 5 d forecast time. In addition an iterative model for Z500 and T850 was trained. Here we use a single linear regression to predict 6 hours ahead where the two fields are concatenated (2×32×64 → 4096). The advantage of iterative forecasts is that a single model is able to make predictions for any forecast time rather than having to train several models. For iterative forecasts the model takes its previous output as input for the next step. To create a 5 day iterative forecast the model trained to predict 6 hour forecasts is called 20 times” | senses[13] |
| sense | CNN weather baseline - fully convolutional Z500/T850 model with longitude-periodic kernels — “As our deep learning baseline we chose a simple fully-convolutional neural network. CNNs are the natural choice for spatial data since they exploit translational invariances in images/fields. Here we train a CNN with 5 layers. Each hidden layer has 64 channels with a convolutional kernel of size 5 and ELU activations (Clevert et al., 2015). The input and output layers have two channels, representing Z500 and T850. The model was trained using the Adam optimizer (Kingma and Ba, 2014) and a mean squared error loss function. The total number of trainable parameters is 313,858. We implemented periodic convolutions in the longitude direction but not the latitude direction” | senses[14] |
| sense | Transfer learning - pretrain on related simulations, then fine-tune on target observations — “Should overfitting be a problem, one could try transfer learning. In transfer learning, the network is pretrained on a similar task or dataset, for example, climate model simulations, and then finetuned on the actual data. This is common practice in computer vision and has been successfully applied to seasonal ENSO forecasting” | senses[15] |
| sense | Physical-field augmentation - overfitting control constrained by non-interchangeable spatial directions — “Another common method to prevent overfitting is data augmentation, which in traditional computer vision is done by e.g. randomly rotating or flipping the image. However, many of the traditional data augmentation techniques are questionable for physical fields. Random rotations, for example, will likely not work for this dataset since the x and y directions are physically distinct. Thus, finding good data augmentation techniques for physical fields is an outstanding problem” | senses[16] |
| sense | Probabilistic forecast - ensemble or parametric distribution expressing uncertainty — “One important aspect that is not currently addressed by this benchmark is probabilistic forecasting. Because of the chaotic evolution of the atmosphere, it is very important to also have an estimate of the uncertainty of a forecast. In physical NWP this is done by running several forecasts, called an ensemble, from slightly different initial conditions and potentially with different or stochastic model physics (Palmer, 2019). From this Monte Carlo forecast one can then estimate a probability distribution. A different approach, which is often taken in statistical post-processing of NWP forecasts, is to directly estimate a parametric distribution” | senses[17] |
| sense | Forecast reliability - uncertainty accurately indicates error — “For a probabilistic forecast to be reliable the forecast uncertainty has to be an accurate indicator of the error. A good first order approximation for this is the spread (ensemble standard deviation) to error (RMSE) ratio which should be one (Leutbecher and Palmer, 2008). A more stringent test is to use a proper probabilistic scoring rule, for example the continuous ranked probability score (CRPS)” | senses[18] |
| sense | Weather / climate prediction - future state before the horizon versus long-term statistics — “Even though weather and climate deal with the same underlying physical system, they pose different forecasting challenges. In weather forecasting the goal is to predict the state of the atmosphere at a specific time into the future. This is only possible up to the prediction horizon of the atmosphere, which is thought to be at roughly two weeks. Climate models, on the other hand, are evaluated by comparing long-term statistics to observations” | senses[19] |
| sense | Climate-change extrapolation - future states outside the historical training manifold — “future climates will produce atmospheric states that lie outside of the historical manifold of states. Plain neural networks are very bad at extrapolating to climates beyond what they have seen in the training dataset (Rasp et al., 2018). For this reason, climate change simulations with current data-driven methods are likely not a good idea” | senses[20] |
| control | baseline scores from linear regression, deep learning and purely physical forecasting models, published with the benchmark — held — “We propose simple and clear evaluation metrics which will enable a direct comparison between different methods.” — rules out that a data-driven forecast's apparent skill exceeds what simple statistical or existing physical models already achieve — “Further, we provide baseline scores from simple linear regression techniques, deep learning models, as well as purely physical forecasting models.” | controls[0] |
| note | deep learning —is-a→ machine learning — Defined in the paper's first sentence of body text, and the definition carries the mechanism with it - multi-layered artificial neural networks. — “Deep learning, a branch of machine learning based on multi-layered artificial neural networks, has proven to be a powerful tool for a wide range of tasks, most notably image recognition and natural language processing” | concept_edges[0] |
| weight | deep learning —is-a→ machine learning — 0.9 — “Deep learning, a branch of machine learning based on multi-layered artificial neural networks, has proven to be a powerful tool for a wide range of tasks, most notably image recognition and natural language processing” | concept_edges[0] |
| audit | root mean squared error —operationalised-as→ benchmark — Audit 2026-09-28: DOUBTFUL - read against its span and kept, but RMSE is the benchmark's primary metric; operationalised-as runs from the benchmark, not to it. — “We chose the root mean squared error (RMSE) as our primary metric because it is easy to compute and mirrors the loss used for most ML applications” | concept_edges[1] |
| note | root mean squared error —operationalised-as→ benchmark — THE CHOICE IS MADE ON GROUNDS THAT ARE NOT ABOUT WEATHER: the metric was picked 'because it is easy to compute and mirrors the loss used for most ML applications'. Convenience and alignment with the modelling community decide what counts as success on this benchmark - and the paper says so outright. The record already carries a sense noting success here is a SINGLE metric over two fields. — “We chose the root mean squared error (RMSE) as our primary metric because it is easy to compute and mirrors the loss used for most ML applications” | concept_edges[1] |
| weight | root mean squared error —operationalised-as→ benchmark — 0.9 — “We chose the root mean squared error (RMSE) as our primary metric because it is easy to compute and mirrors the loss used for most ML applications” | concept_edges[1] |
| note | iterative model —contrasted-with→ direct predictions — The paper contrasts a single recursively applied model with separate lead-specific models. — “As a first purely data-driven baseline we fit a simple linear regression model. For the direct predictions a separate model was trained for each of the four variables. For this purpose the 2D fields were flattened from 32×64 → 2048. This was done for 3 d and 5 d forecast time. In addition an iterative model for Z500 and T850 was trained. Here we use a single linear regression to predict 6 hours ahead where the two fields are concatenated (2×32×64 → 4096). The advantage of iterative forecasts is that a single model is able to make predictions for any forecast time rather than having to train several models. For iterative forecasts the model takes its previous output as input for the next step. To create a 5 day iterative forecast the model trained to predict 6 hour forecasts is called 20 times” | concept_edges[4] |
| audit | simple RMSE loss —causes→ unrealistically smooth fields — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says the smoothness is LIKELY a result of two factors; hedged. — “For 5 days forecast time the CNN model predicts unrealistically smooth fields. This is likely a result of two factors: first, the two input fields used in this baseline CNN contain insufficient information to create a skillful 5 day forecast; and second, at 5 days the atmosphere already shows some chaotic behavior which causes a model trained with a simple RMSE loss to predict smooth fields” | concept_edges[5] |
| note | simple RMSE loss —causes→ unrealistically smooth fields — Weight reflects the paper’s two-factor, likely-cause explanation for over-smoothing. — “For 5 days forecast time the CNN model predicts unrealistically smooth fields. This is likely a result of two factors: first, the two input fields used in this baseline CNN contain insufficient information to create a skillful 5 day forecast; and second, at 5 days the atmosphere already shows some chaotic behavior which causes a model trained with a simple RMSE loss to predict smooth fields” | concept_edges[5] |
| weight | simple RMSE loss —causes→ unrealistically smooth fields — 0.8 — “For 5 days forecast time the CNN model predicts unrealistically smooth fields. This is likely a result of two factors: first, the two input fields used in this baseline CNN contain insufficient information to create a skillful 5 day forecast; and second, at 5 days the atmosphere already shows some chaotic behavior which causes a model trained with a simple RMSE loss to predict smooth fields” | concept_edges[5] |
| note | weather forecasting —contrasted-with→ Climate models — The paper explicitly contrasts future-state and long-term-statistics objectives. — “Even though weather and climate deal with the same underlying physical system, they pose different forecasting challenges. In weather forecasting the goal is to predict the state of the atmosphere at a specific time into the future. This is only possible up to the prediction horizon of the atmosphere, which is thought to be at roughly two weeks. Climate models, on the other hand, are evaluated by comparing long-term statistics to observations” | concept_edges[6] |
| outcome | control — held — “Further, we provide baseline scores from simple linear regression techniques, deep learning models, as well as purely physical forecasting models.” — “We propose simple and clear evaluation metrics which will enable a direct comparison between different methods.” | controls[0] |
| for which use | limitation — the motivation for the benchmark: NUMERICAL WEATHER PREDICTION (NWP) MODELS STILL HAVE SHORTCOMING FOR MANY IMPORTANT APPLICATIONS — “numerical weather prediction (NWP) models still have shortcoming for many important applications” | limitations[0] |
| for which use | limitation — any probabilistic use of the benchmark: ONE IMPORTANT ASPECT THAT IS NOT CURRENTLY ADDRESSED BY THIS BENCHMARK IS PROBABILISTIC FORECASTING — “One important aspect that is not currently addressed by this benchmark is probabilistic forecasting.” | limitations[1] |
| for which use | limitation — any climate-timescale use: ANOTHER ASPECT THAT IS UNTOUCHED BY THE BENCHMARK CHALLENGE PROPOSED HERE IS CLIMATE PREDICTION — “Another aspect that is untouched by the benchmark challenge proposed here is climate prediction.” | limitations[2] |
| for which use | limitation — every score computed on this benchmark, which is a score on non-extremes: WE DECIDED TO OMIT EXTREMES FOR NOW BUT USERS ARE ENCOURAGED TO CHOSE THEIR OWN VERIFICATION OF EXTREMES — “Therefore, we decided to omit extremes for now but users are encouraged to chose their own verification of extremes.” | limitations[3] |
| for which use | limitation — the iterative CNN baseline result: THE POOR SKILL OF THE ITERATIVE FORECAST COULD EASILY BE A RESULT OF USING AN OVERLY SIMPLISTIC NETWORK ARCHITECTURE — “However, the poor skill of the iterative forecast could easily be a result of using an overly simplistic network architecture.” | limitations[4] |
| for which use | limitation — a ranking on this benchmark as a ranking of forecast quality: the primary metric was chosen because it is easy to compute and mirrors the loss used for most ML applications — “We chose the root mean squared error (RMSE) as our primary metric because it is easy to compute and mirrors the loss used for most ML applications.” | limitations[5] |
| for which use | limitation — the training-set size for this benchmark: 40 years of hourly data is around 350,000 samples, which the authors reduce to about 15,000 independent ones under a stated assumption, with sufficiency unknown without empirical evidence — “However, the samples are correlated in time. If one assumes that a new weather situation occurs every day, then the number of samples is reduced to around 15,000. Without empirical evidence it is hard to estimate whether this number is sufficient to train complex networks” | limitations[6] |
| for which use | limitation — the feasibility of a competitive data-driven weather model: the authors state high resolutions would be needed and that GPU RAM quickly becomes a limitation — “one can assume that to create a competitive data-driven NWP model, high resolutions have to be used, for which GPU RAM quickly becomes a limitation.” | limitations[7] |
| weight | Conditional GANs → unrealistically smooth fields — 0.4 — “For synthesis tasks, generative adverserial networks (GANs) (Goodfellow et al., 2014) were shown to be particularly powerful for creating realistic natural images and fluid flows (Xie et al., 2018). This might be attractive since minimizing a mean loss, such as the MSE, for random or stochastic data leads to unrealistically smooth predictions as seen in Fig. 3. Conditional GANs (Mirza and Osindero, 2014; Isola et al., 2016) could potentially alleviate this issue but it is still unclear to what extent GAN predictions are able to recover the multi-variate distribution of the training samples” | unhoused_relations[5] |
| replication | untested | |
| no type for this | ERA5 reanalysis dataset → forecast model with the available observations — ERA5 combines a forecast model with observations; composed-from is absent from the edge registry. — “For the proposed benchmark, we use the ERA5 reanalysis dataset (Hersbach et al., 2020) for training and testing. Reanalysis datasets provide the best guess of the atmospheric state at any point in time by combining a forecast model with the available observations. The raw data is available hourly for 40 years from 1979 to 2018 on a 0.25°latitude-longitude grid (721×1440 grid points) with 37 vertical levels” | unhoused_relations[0] |
| no type for this | probabilistic forecasting → ensemble — Physical probabilistic forecasts are implemented by an ensemble; implemented-by is absent from the edge registry. — “One important aspect that is not currently addressed by this benchmark is probabilistic forecasting. Because of the chaotic evolution of the atmosphere, it is very important to also have an estimate of the uncertainty of a forecast. In physical NWP this is done by running several forecasts, called an ensemble, from slightly different initial conditions and potentially with different or stochastic model physics (Palmer, 2019). From this Monte Carlo forecast one can then estimate a probability distribution. A different approach, which is often taken in statistical post-processing of NWP forecasts, is to directly estimate a parametric distribution” | unhoused_relations[1] |
| no type for this | ensemble → Monte Carlo forecast — The ensemble is treated as a Monte Carlo forecast; method-role is absent from the edge registry. — “One important aspect that is not currently addressed by this benchmark is probabilistic forecasting. Because of the chaotic evolution of the atmosphere, it is very important to also have an estimate of the uncertainty of a forecast. In physical NWP this is done by running several forecasts, called an ensemble, from slightly different initial conditions and potentially with different or stochastic model physics (Palmer, 2019). From this Monte Carlo forecast one can then estimate a probability distribution. A different approach, which is often taken in statistical post-processing of NWP forecasts, is to directly estimate a parametric distribution” | unhoused_relations[2] |
| no type for this | forecast uncertainty → spread (ensemble standard deviation) to error (RMSE) ratio — The spread-to-error ratio is a first-order approximation for reliability; diagnostic-for is absent from the edge registry. — “For a probabilistic forecast to be reliable the forecast uncertainty has to be an accurate indicator of the error. A good first order approximation for this is the spread (ensemble standard deviation) to error (RMSE) ratio which should be one (Leutbecher and Palmer, 2008). A more stringent test is to use a proper probabilistic scoring rule, for example the continuous ranked probability score (CRPS)” | unhoused_relations[3] |
| no type for this | forecast uncertainty → continuous ranked probability score (CRPS) — CRPS is a more stringent reliability test; diagnostic-for is absent from the edge registry. — “For a probabilistic forecast to be reliable the forecast uncertainty has to be an accurate indicator of the error. A good first order approximation for this is the spread (ensemble standard deviation) to error (RMSE) ratio which should be one (Leutbecher and Palmer, 2008). A more stringent test is to use a proper probabilistic scoring rule, for example the continuous ranked probability score (CRPS)” | unhoused_relations[4] |
| no type for this | Conditional GANs → unrealistically smooth fields — Conditional GANs could potentially alleviate smooth mean-loss predictions; may-remedy is absent from the edge registry. — “For synthesis tasks, generative adverserial networks (GANs) (Goodfellow et al., 2014) were shown to be particularly powerful for creating realistic natural images and fluid flows (Xie et al., 2018). This might be attractive since minimizing a mean loss, such as the MSE, for random or stochastic data leads to unrealistically smooth predictions as seen in Fig. 3. Conditional GANs (Mirza and Osindero, 2014; Isola et al., 2016) could potentially alleviate this issue but it is still unclear to what extent GAN predictions are able to recover the multi-variate distribution of the training samples” | unhoused_relations[5] |
| no type for this | machine learning emulators → hybrid modeling — Machine-learning emulators are combined with physical components in hybrid modeling; combined-with is absent from the edge registry. — “Yet another direction for ML research is hybrid modeling, in which a physical model is combined with data-driven components, for example replacing heuristic cloud or radiation parameterizations (Chevallier et al., 1998; Krasnopolsky et al., 2005; Rasp et al., 2018; Brenowitz and Bretherton, 2018; Yuval and O’Gorman, 2020). The key idea behind these approaches is to only replace uncertain (e.g. clouds) or computationally expensive (e.g. line-by-line radiation) model components with machine learning emulators and leave other model components (e.g. large-scale dynamics) untouched” | unhoused_relations[6] |
| no type for this | purely-data driven prediction → post-processing — The paper sets pure medium-range prediction apart from post-processing; set-apart-from is absent from the cue-governed registry. — “Here we focus on purely-data driven prediction of the global atmospheric flow in the medium-range. Specifically, we select lead times of 3 and 5 days, for which the atmosphere is still reasonably deterministic but also exhibits complex nonlinear behaviour, such as baroclinic instabilities and tropical cyclogenesis. This forecast range is important from a societal point of view because it delivers crucial information for disaster preparation, for example for flooding, cold and hot spells or damaging winds (Lazo et al., 2009). Creating a good medium-range forecast requires understanding complex atmospheric dynamics and the interplay between several variables across a range of scales. This sets this challenge apart from post-processing and statistical forecasting, in which the large-scale dynamics are predicted by a physical model, and nowcasting, in which the considered evolution is univariate and short-term. In other words, this benchmark closely emulates the task performed by physical NWP models” | unhoused_relations[7] |
| no type for this | purely-data driven prediction → nowcasting — The paper sets pure medium-range prediction apart from nowcasting; set-apart-from is absent from the cue-governed registry. — “Here we focus on purely-data driven prediction of the global atmospheric flow in the medium-range. Specifically, we select lead times of 3 and 5 days, for which the atmosphere is still reasonably deterministic but also exhibits complex nonlinear behaviour, such as baroclinic instabilities and tropical cyclogenesis. This forecast range is important from a societal point of view because it delivers crucial information for disaster preparation, for example for flooding, cold and hot spells or damaging winds (Lazo et al., 2009). Creating a good medium-range forecast requires understanding complex atmospheric dynamics and the interplay between several variables across a range of scales. This sets this challenge apart from post-processing and statistical forecasting, in which the large-scale dynamics are predicted by a physical model, and nowcasting, in which the considered evolution is univariate and short-term. In other words, this benchmark closely emulates the task performed by physical NWP models” | unhoused_relations[8] |
| no type for this | training dataset → split the data into three parts — Training is one partition of the three-way data split; partition-of is absent from the cue-governed registry. — “In machine learning it is good practice to split the data into three parts: the training, validation and test sets. The training dataset is used to actually fit the model” | unhoused_relations[9] |
| no type for this | validation dataset → split the data into three parts — Validation is one partition of the three-way data split; partition-of is absent from the cue-governed registry. — “However, there is the danger that through continued tuning of hyperparameters one unwillingly overfits to the validation dataset. Therefore it is advisable to keep a third, testing, dataset for final evaluations of model performance” | unhoused_relations[10] |
| no type for this | third, testing, dataset → split the data into three parts — Testing is the held final-evaluation partition; partition-of is absent from the cue-governed registry. — “However, there is the danger that through continued tuning of hyperparameters one unwillingly overfits to the validation dataset. Therefore it is advisable to keep a third, testing, dataset for final evaluations of model performance” | unhoused_relations[11] |
| limitation | current numerical weather prediction models still fall short for many applications — “numerical weather prediction (NWP) models still have shortcoming for many important applications” | limitations[0] |
| limitation | The benchmark does not cover probabilistic forecasting, only deterministic scores. — “One important aspect that is not currently addressed by this benchmark is probabilistic forecasting.” | limitations[1] |
| limitation | Climate-timescale prediction is outside the benchmark's scope. — “Another aspect that is untouched by the benchmark challenge proposed here is climate prediction.” | limitations[2] |
| limitation | Extreme weather events are deliberately excluded from the evaluation, with no standard metric offered for them. — “Therefore, we decided to omit extremes for now but users are encouraged to chose their own verification of extremes.” | limitations[3] |
| limitation | The poor performance of the iterative CNN baseline may reflect the deliberately simple architecture rather than a property of iterative forecasting. — “However, the poor skill of the iterative forecast could easily be a result of using an overly simplistic network architecture.” | limitations[4] |
| limitation | The benchmark's primary metric was chosen for ease of computation and for matching the usual machine-learning loss, not for being the right measure of forecast quality. — “We chose the root mean squared error (RMSE) as our primary metric because it is easy to compute and mirrors the loss used for most ML applications.” | limitations[5] |
| limitation | The usable amount of training data is far smaller than the raw count, by the authors' own assumption, and whether it suffices is stated to be unknown. — “However, the samples are correlated in time. If one assumes that a new weather situation occurs every day, then the number of samples is reduced to around 15,000. Without empirical evidence it is hard to estimate whether this number is sufficient to train complex networks” | limitations[6] |
| limitation | Scaling the approach to the resolutions likely required runs into hardware, which the authors name as the binding constraint. — “one can assume that to create a competitive data-driven NWP model, high resolutions have to be used, for which GPU RAM quickly becomes a limitation.” | limitations[7] |
| limitation | THE BENCHMARK EXCLUDES THE CASE IT CALLS MOST IMPORTANT TO SOCIETY, AND SAYS SO: extreme events `will contribute little to regular verification metrics like the RMSE` yet `for society these events are highly important`. The exclusion is declared, reasoned and handed to the user - `users are encouraged to chose their own verification of extremes` - which means any two WeatherBench results on extremes are not comparable BY THE BENCHMARK'S OWN DESIGN. — “But defining extremes is ambiguous which is why there is no standard metric for evaluating extremes. The goal of this benchmark is to provide a simple, clear problem. Therefore, we decided to omit extremes for now but users are encouraged to chose their own verification of extremes.” | limitations[8] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | benchmark datasets make algorithms quantitatively inter-comparable — “they make different algorithms quantitatively inter-comparable and foster constructive competition” | content_tags.findings[0] |
| finding | absence of a common dataset made prior studies hard to compare — “the lack of a common dataset and evaluation metrics make inter-comparison between studies difficult.” | content_tags.findings[1] |
| evidence status note | finding — QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. | content_tags.findings[1] |
| finding | a geopotential CNN performed best yet could not beat the physical baseline — “CNN predicting only geopotential performed best but was unable to beat the low-resolution physical baseline” | content_tags.findings[2] |
| finding | directly-trained networks outperformed iterative forecasts — “Interestingly, networks trained to directly predict a certain forecast time, e.g. 5 days, outperformed iterative networks” | content_tags.findings[3] |
| quantity | 40 years (1979-2018) — time span of hourly ERA5 raw data — “available hourly for 40 years from 1979 to 2018” | content_tags.quantities[0] |
| quantity | 721x1440 grid points — native 0.25-degree ERA5 grid with 37 levels — “The raw data is available hourly for 40 years from 1979 to 2018 on a 0.25°latitude-longitude grid (721×1440 grid points) with 37 vertical levels.” | content_tags.quantities[1] |
| quantity | 700 GB — size of a single vertical level over the full period — “a single vertical level for the entire time period amounts to almost 700GB of data” | content_tags.quantities[2] |
| quantity | 3 and 5 days — medium-range forecast lead times targeted — “lead times of 3 and 5 days” | content_tags.quantities[3] |
| quantity | 50 ensemble members — typical cap on probabilistic NWP forecasts — “usually limited to 50 ensemble members or less” | content_tags.quantities[4] |
| quantity | 191 GB — size of the full 5.625-degree benchmark dataset — “The entire dataset at 5.625° resolution has a size of 191GB” | content_tags.quantities[5] |
| variable | 500 hPa geopotential (primary predicted upper-level field) — “trained a neural network to predict 500 hPa geopotential (Z500; see Section 4 for details on commonly used fields)” | content_tags.variables[0] |
| variable | forecast lead time (prediction horizon) — “lead times of 3 and 5 days” | content_tags.variables[1] |
| variable | root mean squared error (forecast evaluation metrics) — “evaluated using the root mean squared error and the anomaly correlation coefficient” | content_tags.variables[4] |
| variable | anomaly correlation coefficient (forecast evaluation metrics) — “evaluated using the root mean squared error and the anomaly correlation coefficient” | content_tags.variables[5] |
| definition | numerical weather prediction (NWP) — “numerical weather prediction (NWP) models still have shortcoming for many important applications” | content_tags.definitions[0] |
| definition | nowcasting — “nowcasting, which describes the short range (up to 6 hours) prediction” | content_tags.definitions[1] |
| definition | hybrid modeling — “hybrid modeling, in which a physical model is combined with data-driven components” | content_tags.definitions[2] |
| evidence status note | definition — QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. | content_tags.definitions[2] |
| definition | reanalysis — “Reanalysis datasets provide the best guess of the atmospheric state at any point in time by combining a forecast model with the available observations” | content_tags.definitions[3] |
| key-terms | weather forecasting, data-driven, deep learning, benchmark dataset, ERA5, numerical weather prediction, neural networks, geopotential, medium-range, evaluation metrics, reanalysis |
| concept | numerical weather prediction — “However, today’s numerical weather prediction (NWP) models still have shortcoming for many important applications, for example forecasting mesoscale convective systems over Africa” | concepts[4] |
| concept | root mean squared error — “We chose the root mean squared error (RMSE) as our primary metric because it is easy to compute and mirrors the loss used for most ML applications.” | concepts[5] |
| concept | training data — “Another potential issue is the limited amount of training data available.” | concepts[6] |
| concept | extremes — not defined in this paper's own words; rare high-impact weather situations; the benchmark declines to define them — “But defining extremes is ambiguous which is why there is no standard metric for evaluating extremes. The goal of this benchmark is to provide a simple, clear problem. Therefore, we decided to omit extremes for now but users are encouraged to chose their own verification of extremes.” | concepts[7] |
| concept | post-processing — not defined in this paper's own words; ML correction of statistical biases in physical-model output — “ML can be applied to weather prediction in many different ways. Two long-standing applications of ML are post-processing – the correction of statistical biases in the output of physical models – and statistical forecasting – the prediction of variables not directly output by the physical model. Traditionally, this has been done using simple linear techniques but more recently modern machine learning approaches like random forests or neural networks have been explored (Gagne et al., 2014; Taillardat et al., 2016; McGovern et al., 2017; Lagerquist et al., 2017; Rasp and Lerch, 2018). Typically, these approaches target very specific variables or locations whereas the general evolution of the atmosphere is still predicted by a physical model” | concepts[8] |
| concept | statistical forecasting — not defined in this paper's own words; ML prediction of variables not directly output by a physical model — “ML can be applied to weather prediction in many different ways. Two long-standing applications of ML are post-processing – the correction of statistical biases in the output of physical models – and statistical forecasting – the prediction of variables not directly output by the physical model. Traditionally, this has been done using simple linear techniques but more recently modern machine learning approaches like random forests or neural networks have been explored (Gagne et al., 2014; Taillardat et al., 2016; McGovern et al., 2017; Lagerquist et al., 2017; Rasp and Lerch, 2018). Typically, these approaches target very specific variables or locations whereas the general evolution of the atmosphere is still predicted by a physical model” | concepts[9] |
| concept | nowcasting — not defined in this paper's own words; up-to-six-hour precipitation prediction by radar extrapolation without a physical model — “Another application that has recently been explored using ML is nowcasting, which describes the short range (up to 6 hours) prediction of precipitation by directly extrapolating radar observation without a physical model involved” | concepts[10] |
| concept | hybrid modeling — not defined in this paper's own words; physical model combined with data-driven replacements for selected components — “Yet another direction for ML research is hybrid modeling, in which a physical model is combined with data-driven components, for example replacing heuristic cloud or radiation parameterizations (Chevallier et al., 1998; Krasnopolsky et al., 2005; Rasp et al., 2018; Brenowitz and Bretherton, 2018; Yuval and O’Gorman, 2020). The key idea behind these approaches is to only replace uncertain (e.g. clouds) or computationally expensive (e.g. line-by-line radiation) model components with machine learning emulators and leave other model components (e.g. large-scale dynamics) untouched” | concepts[11] |
| concept | machine learning emulators — not defined in this paper's own words; learned replacements for uncertain or expensive physical parameterizations — “Yet another direction for ML research is hybrid modeling, in which a physical model is combined with data-driven components, for example replacing heuristic cloud or radiation parameterizations (Chevallier et al., 1998; Krasnopolsky et al., 2005; Rasp et al., 2018; Brenowitz and Bretherton, 2018; Yuval and O’Gorman, 2020). The key idea behind these approaches is to only replace uncertain (e.g. clouds) or computationally expensive (e.g. line-by-line radiation) model components with machine learning emulators and leave other model components (e.g. large-scale dynamics) untouched” | concepts[12] |
| concept | purely-data driven prediction — not defined in this paper's own words; global atmospheric-flow prediction without a physical forecast component — “Here we focus on purely-data driven prediction of the global atmospheric flow in the medium-range. Specifically, we select lead times of 3 and 5 days, for which the atmosphere is still reasonably deterministic but also exhibits complex nonlinear behaviour, such as baroclinic instabilities and tropical cyclogenesis. This forecast range is important from a societal point of view because it delivers crucial information for disaster preparation, for example for flooding, cold and hot spells or damaging winds (Lazo et al., 2009). Creating a good medium-range forecast requires understanding complex atmospheric dynamics and the interplay between several variables across a range of scales. This sets this challenge apart from post-processing and statistical forecasting, in which the large-scale dynamics are predicted by a physical model, and nowcasting, in which the considered evolution is univariate and short-term. In other words, this benchmark closely emulates the task performed by physical NWP models” | concepts[13] |
| concept | medium-range — not defined in this paper's own words; three- and five-day forecast regime chosen for the benchmark — “Here we focus on purely-data driven prediction of the global atmospheric flow in the medium-range. Specifically, we select lead times of 3 and 5 days, for which the atmosphere is still reasonably deterministic but also exhibits complex nonlinear behaviour, such as baroclinic instabilities and tropical cyclogenesis. This forecast range is important from a societal point of view because it delivers crucial information for disaster preparation, for example for flooding, cold and hot spells or damaging winds (Lazo et al., 2009). Creating a good medium-range forecast requires understanding complex atmospheric dynamics and the interplay between several variables across a range of scales. This sets this challenge apart from post-processing and statistical forecasting, in which the large-scale dynamics are predicted by a physical model, and nowcasting, in which the considered evolution is univariate and short-term. In other words, this benchmark closely emulates the task performed by physical NWP models” | concepts[14] |
| concept | baroclinic instabilities — not defined in this paper's own words; nonlinear atmospheric behavior present at the selected lead times — “Here we focus on purely-data driven prediction of the global atmospheric flow in the medium-range. Specifically, we select lead times of 3 and 5 days, for which the atmosphere is still reasonably deterministic but also exhibits complex nonlinear behaviour, such as baroclinic instabilities and tropical cyclogenesis. This forecast range is important from a societal point of view because it delivers crucial information for disaster preparation, for example for flooding, cold and hot spells or damaging winds (Lazo et al., 2009). Creating a good medium-range forecast requires understanding complex atmospheric dynamics and the interplay between several variables across a range of scales. This sets this challenge apart from post-processing and statistical forecasting, in which the large-scale dynamics are predicted by a physical model, and nowcasting, in which the considered evolution is univariate and short-term. In other words, this benchmark closely emulates the task performed by physical NWP models” | concepts[15] |
| concept | tropical cyclogenesis — not defined in this paper's own words; tropical storm-development behavior present in the benchmark task — “Here we focus on purely-data driven prediction of the global atmospheric flow in the medium-range. Specifically, we select lead times of 3 and 5 days, for which the atmosphere is still reasonably deterministic but also exhibits complex nonlinear behaviour, such as baroclinic instabilities and tropical cyclogenesis. This forecast range is important from a societal point of view because it delivers crucial information for disaster preparation, for example for flooding, cold and hot spells or damaging winds (Lazo et al., 2009). Creating a good medium-range forecast requires understanding complex atmospheric dynamics and the interplay between several variables across a range of scales. This sets this challenge apart from post-processing and statistical forecasting, in which the large-scale dynamics are predicted by a physical model, and nowcasting, in which the considered evolution is univariate and short-term. In other words, this benchmark closely emulates the task performed by physical NWP models” | concepts[16] |
| concept | common benchmark challenge — not defined in this paper's own words; shared challenge intended to make forecasting algorithms comparable — “we currently lack a common benchmark challenge to accelerate progress. Benchmark datasets can have a huge impact because they make different algorithms quantitatively inter-comparable and foster constructive competition, particularly in a nascent direction of research. Famous examples are the computer vision datasets MNIST (LeCun et al., 1998) and ImageNet (Russakovsky et al., 2015). Further, well-curated benchmark datasets make it easier for people from different fields to work on a problem” | concepts[17] |
| concept | Benchmark datasets — not defined in this paper's own words; curated shared datasets that enable comparison, competition, and cross-field entry — “we currently lack a common benchmark challenge to accelerate progress. Benchmark datasets can have a huge impact because they make different algorithms quantitatively inter-comparable and foster constructive competition, particularly in a nascent direction of research. Famous examples are the computer vision datasets MNIST (LeCun et al., 1998) and ImageNet (Russakovsky et al., 2015). Further, well-curated benchmark datasets make it easier for people from different fields to work on a problem” | concepts[18] |
| concept | ERA5 reanalysis dataset — not defined in this paper's own words; 1979–2018 hourly reanalysis source used for benchmark training and testing — “For the proposed benchmark, we use the ERA5 reanalysis dataset (Hersbach et al., 2020) for training and testing. Reanalysis datasets provide the best guess of the atmospheric state at any point in time by combining a forecast model with the available observations. The raw data is available hourly for 40 years from 1979 to 2018 on a 0.25°latitude-longitude grid (721×1440 grid points) with 37 vertical levels” | concepts[19] |
| concept | Reanalysis datasets — not defined in this paper's own words; best estimate of atmospheric state formed from a forecast model and observations — “For the proposed benchmark, we use the ERA5 reanalysis dataset (Hersbach et al., 2020) for training and testing. Reanalysis datasets provide the best guess of the atmospheric state at any point in time by combining a forecast model with the available observations. The raw data is available hourly for 40 years from 1979 to 2018 on a 0.25°latitude-longitude grid (721×1440 grid points) with 37 vertical levels” | concepts[20] |
| concept | forecast model — not defined in this paper's own words; physical model component combined with observations in reanalysis — “For the proposed benchmark, we use the ERA5 reanalysis dataset (Hersbach et al., 2020) for training and testing. Reanalysis datasets provide the best guess of the atmospheric state at any point in time by combining a forecast model with the available observations. The raw data is available hourly for 40 years from 1979 to 2018 on a 0.25°latitude-longitude grid (721×1440 grid points) with 37 vertical levels” | concepts[21] |
| concept | observations — not defined in this paper's own words; measured atmospheric evidence assimilated into reanalysis — “For the proposed benchmark, we use the ERA5 reanalysis dataset (Hersbach et al., 2020) for training and testing. Reanalysis datasets provide the best guess of the atmospheric state at any point in time by combining a forecast model with the available observations. The raw data is available hourly for 40 years from 1979 to 2018 on a 0.25°latitude-longitude grid (721×1440 grid points) with 37 vertical levels” | concepts[22] |
| concept | split the data into three parts — not defined in this paper's own words; partition of examples into fitting, tuning, and final-evaluation subsets — “In machine learning it is good practice to split the data into three parts: the training, validation and test sets. The training dataset is used to actually fit the model” | concepts[23] |
| concept | training dataset — not defined in this paper's own words; data partition used to fit the model — “In machine learning it is good practice to split the data into three parts: the training, validation and test sets. The training dataset is used to actually fit the model” | concepts[24] |
| concept | validation dataset — not defined in this paper's own words; experiment-time partition susceptible to hyperparameter overfitting — “However, there is the danger that through continued tuning of hyperparameters one unwillingly overfits to the validation dataset. Therefore it is advisable to keep a third, testing, dataset for final evaluations of model performance” | concepts[25] |
| concept | third, testing, dataset — not defined in this paper's own words; held third partition used for final evaluation — “However, there is the danger that through continued tuning of hyperparameters one unwillingly overfits to the validation dataset. Therefore it is advisable to keep a third, testing, dataset for final evaluations of model performance” | concepts[26] |
| concept | persistence forecast — not defined in this paper's own words; baseline carrying initialization fields forward as the forecast — “The two simplest possible forecasts are a) a persistence forecast in which the fields at initialization time are used as forecasts ("tomorrow’s weather is today’s weather"), and b) a climatological forecast” | concepts[27] |
| concept | climatological forecast — not defined in this paper's own words; baseline predicting climatological averages — “The two simplest possible forecasts are a) a persistence forecast in which the fields at initialization time are used as forecasts ("tomorrow’s weather is today’s weather"), and b) a climatological forecast” | concepts[28] |
| concept | direct predictions — not defined in this paper's own words; separate model trained for a specified forecast lead time — “As a first purely data-driven baseline we fit a simple linear regression model. For the direct predictions a separate model was trained for each of the four variables. For this purpose the 2D fields were flattened from 32×64 → 2048. This was done for 3 d and 5 d forecast time. In addition an iterative model for Z500 and T850 was trained. Here we use a single linear regression to predict 6 hours ahead where the two fields are concatenated (2×32×64 → 4096). The advantage of iterative forecasts is that a single model is able to make predictions for any forecast time rather than having to train several models. For iterative forecasts the model takes its previous output as input for the next step. To create a 5 day iterative forecast the model trained to predict 6 hour forecasts is called 20 times” | concepts[29] |
| concept | iterative model — not defined in this paper's own words; single short-lead model repeatedly applied to reach arbitrary lead times — “As a first purely data-driven baseline we fit a simple linear regression model. For the direct predictions a separate model was trained for each of the four variables. For this purpose the 2D fields were flattened from 32×64 → 2048. This was done for 3 d and 5 d forecast time. In addition an iterative model for Z500 and T850 was trained. Here we use a single linear regression to predict 6 hours ahead where the two fields are concatenated (2×32×64 → 4096). The advantage of iterative forecasts is that a single model is able to make predictions for any forecast time rather than having to train several models. For iterative forecasts the model takes its previous output as input for the next step. To create a 5 day iterative forecast the model trained to predict 6 hour forecasts is called 20 times” | concepts[30] |
| concept | previous output as input — not defined in this paper's own words; feedback step that advances an iterative forecast — “As a first purely data-driven baseline we fit a simple linear regression model. For the direct predictions a separate model was trained for each of the four variables. For this purpose the 2D fields were flattened from 32×64 → 2048. This was done for 3 d and 5 d forecast time. In addition an iterative model for Z500 and T850 was trained. Here we use a single linear regression to predict 6 hours ahead where the two fields are concatenated (2×32×64 → 4096). The advantage of iterative forecasts is that a single model is able to make predictions for any forecast time rather than having to train several models. For iterative forecasts the model takes its previous output as input for the next step. To create a 5 day iterative forecast the model trained to predict 6 hour forecasts is called 20 times” | concepts[31] |
| concept | fully-convolutional neural network — not defined in this paper's own words; five-layer spatial deep-learning baseline for Z500 and T850 — “As our deep learning baseline we chose a simple fully-convolutional neural network. CNNs are the natural choice for spatial data since they exploit translational invariances in images/fields. Here we train a CNN with 5 layers. Each hidden layer has 64 channels with a convolutional kernel of size 5 and ELU activations (Clevert et al., 2015). The input and output layers have two channels, representing Z500 and T850. The model was trained using the Adam optimizer (Kingma and Ba, 2014) and a mean squared error loss function. The total number of trainable parameters is 313,858. We implemented periodic convolutions in the longitude direction but not the latitude direction” | concepts[32] |
| concept | translational invariances — not defined in this paper's own words; spatial regularity exploited by CNNs — “As our deep learning baseline we chose a simple fully-convolutional neural network. CNNs are the natural choice for spatial data since they exploit translational invariances in images/fields. Here we train a CNN with 5 layers. Each hidden layer has 64 channels with a convolutional kernel of size 5 and ELU activations (Clevert et al., 2015). The input and output layers have two channels, representing Z500 and T850. The model was trained using the Adam optimizer (Kingma and Ba, 2014) and a mean squared error loss function. The total number of trainable parameters is 313,858. We implemented periodic convolutions in the longitude direction but not the latitude direction” | concepts[33] |
| concept | periodic convolutions — not defined in this paper's own words; longitude-only wraparound convolution used in the baseline — “As our deep learning baseline we chose a simple fully-convolutional neural network. CNNs are the natural choice for spatial data since they exploit translational invariances in images/fields. Here we train a CNN with 5 layers. Each hidden layer has 64 channels with a convolutional kernel of size 5 and ELU activations (Clevert et al., 2015). The input and output layers have two channels, representing Z500 and T850. The model was trained using the Adam optimizer (Kingma and Ba, 2014) and a mean squared error loss function. The total number of trainable parameters is 313,858. We implemented periodic convolutions in the longitude direction but not the latitude direction” | concepts[34] |
| concept | unrealistically smooth fields — not defined in this paper's own words; over-smoothed five-day CNN output attributed partly to RMSE loss under chaos — “For 5 days forecast time the CNN model predicts unrealistically smooth fields. This is likely a result of two factors: first, the two input fields used in this baseline CNN contain insufficient information to create a skillful 5 day forecast; and second, at 5 days the atmosphere already shows some chaotic behavior which causes a model trained with a simple RMSE loss to predict smooth fields” | concepts[35] |
| concept | simple RMSE loss — not defined in this paper's own words; mean-oriented objective implicated in smoothing chaotic five-day predictions — “For 5 days forecast time the CNN model predicts unrealistically smooth fields. This is likely a result of two factors: first, the two input fields used in this baseline CNN contain insufficient information to create a skillful 5 day forecast; and second, at 5 days the atmosphere already shows some chaotic behavior which causes a model trained with a simple RMSE loss to predict smooth fields” | concepts[36] |
| concept | chaotic behavior — not defined in this paper's own words; forecast uncertainty emerging by five days — “For 5 days forecast time the CNN model predicts unrealistically smooth fields. This is likely a result of two factors: first, the two input fields used in this baseline CNN contain insufficient information to create a skillful 5 day forecast; and second, at 5 days the atmosphere already shows some chaotic behavior which causes a model trained with a simple RMSE loss to predict smooth fields” | concepts[37] |
| concept | image-to-image translation — not defined in this paper's own words; computer-vision analogy for mapping one atmospheric state to another — “From a ML perspective, state-to-state weather prediction is similar to image-to-image translation. For this sort of problem many deep learning techniques have been developed in recent years (Kaji and Kida, 2019). However, forecasting weather differs in some important ways from typical image-to-image applications and raises several open questions. First, the atmosphere is three-dimensional. So far, this aspect has not been taken into account. In the networks of Scher and Messori (2019a), for example, the different levels have been treated as separate channels of the CNN. However, simply using a three-dimensional CNN might not work either because atmospheric dynamics and grid spacings change in the vertical, thereby violating the assumption of translation invariance which underlies the effectiveness of CNNs” | concepts[38] |
| concept | assumption of translation invariance — not defined in this paper's own words; CNN premise violated by vertically changing dynamics and grid spacing — “From a ML perspective, state-to-state weather prediction is similar to image-to-image translation. For this sort of problem many deep learning techniques have been developed in recent years (Kaji and Kida, 2019). However, forecasting weather differs in some important ways from typical image-to-image applications and raises several open questions. First, the atmosphere is three-dimensional. So far, this aspect has not been taken into account. In the networks of Scher and Messori (2019a), for example, the different levels have been treated as separate channels of the CNN. However, simply using a three-dimensional CNN might not work either because atmospheric dynamics and grid spacings change in the vertical, thereby violating the assumption of translation invariance which underlies the effectiveness of CNNs” | concepts[39] |
| concept | transfer learning — not defined in this paper's own words; pretraining on a related task or climate simulation before fine-tuning observations — “Should overfitting be a problem, one could try transfer learning. In transfer learning, the network is pretrained on a similar task or dataset, for example, climate model simulations, and then finetuned on the actual data. This is common practice in computer vision and has been successfully applied to seasonal ENSO forecasting” | concepts[40] |
| concept | data augmentation — not defined in this paper's own words; overfitting control whose usual rotations/flips may be invalid for physical fields — “Another common method to prevent overfitting is data augmentation, which in traditional computer vision is done by e.g. randomly rotating or flipping the image. However, many of the traditional data augmentation techniques are questionable for physical fields. Random rotations, for example, will likely not work for this dataset since the x and y directions are physically distinct. Thus, finding good data augmentation techniques for physical fields is an outstanding problem” | concepts[41] |
| concept | CPU RAM — not defined in this paper's own words; host-memory constraint producing a possible data-loading bottleneck — “Finally, there are technical challenges. Data for a single variable with ten levels at 5.625° resolution take up around 30 GB of data. For a network with several variables or even at higher resolution, the data might not fit into CPU RAM any more and data loading could become a bottleneck. For image files, efficient data loaders have been created6. For netCDF files, however, so far no efficient solution exists to our knowledge. Further, one can assume that to create a competitive data-driven NWP model, high resolutions have to be used, for which GPU RAM quickly becomes a limitation. This suggests that multi-GPU training might be necessary to scale up this approach” | concepts[42] |
| concept | GPU RAM — not defined in this paper's own words; accelerator-memory limit motivating multi-GPU training — “Finally, there are technical challenges. Data for a single variable with ten levels at 5.625° resolution take up around 30 GB of data. For a network with several variables or even at higher resolution, the data might not fit into CPU RAM any more and data loading could become a bottleneck. For image files, efficient data loaders have been created6. For netCDF files, however, so far no efficient solution exists to our knowledge. Further, one can assume that to create a competitive data-driven NWP model, high resolutions have to be used, for which GPU RAM quickly becomes a limitation. This suggests that multi-GPU training might be necessary to scale up this approach” | concepts[43] |
| concept | probabilistic forecasting — not defined in this paper's own words; forecasting that estimates uncertainty as well as a predicted state — “One important aspect that is not currently addressed by this benchmark is probabilistic forecasting. Because of the chaotic evolution of the atmosphere, it is very important to also have an estimate of the uncertainty of a forecast. In physical NWP this is done by running several forecasts, called an ensemble, from slightly different initial conditions and potentially with different or stochastic model physics (Palmer, 2019). From this Monte Carlo forecast one can then estimate a probability distribution. A different approach, which is often taken in statistical post-processing of NWP forecasts, is to directly estimate a parametric distribution” | concepts[44] |
| concept | ensemble — not defined in this paper's own words; multiple forecasts from varied initial conditions or model physics — “One important aspect that is not currently addressed by this benchmark is probabilistic forecasting. Because of the chaotic evolution of the atmosphere, it is very important to also have an estimate of the uncertainty of a forecast. In physical NWP this is done by running several forecasts, called an ensemble, from slightly different initial conditions and potentially with different or stochastic model physics (Palmer, 2019). From this Monte Carlo forecast one can then estimate a probability distribution. A different approach, which is often taken in statistical post-processing of NWP forecasts, is to directly estimate a parametric distribution” | concepts[45] |
| concept | Monte Carlo forecast — not defined in this paper's own words; ensemble-based route to a predictive probability distribution — “One important aspect that is not currently addressed by this benchmark is probabilistic forecasting. Because of the chaotic evolution of the atmosphere, it is very important to also have an estimate of the uncertainty of a forecast. In physical NWP this is done by running several forecasts, called an ensemble, from slightly different initial conditions and potentially with different or stochastic model physics (Palmer, 2019). From this Monte Carlo forecast one can then estimate a probability distribution. A different approach, which is often taken in statistical post-processing of NWP forecasts, is to directly estimate a parametric distribution” | concepts[46] |
| concept | parametric distribution — not defined in this paper's own words; alternative direct representation of a predictive distribution — “One important aspect that is not currently addressed by this benchmark is probabilistic forecasting. Because of the chaotic evolution of the atmosphere, it is very important to also have an estimate of the uncertainty of a forecast. In physical NWP this is done by running several forecasts, called an ensemble, from slightly different initial conditions and potentially with different or stochastic model physics (Palmer, 2019). From this Monte Carlo forecast one can then estimate a probability distribution. A different approach, which is often taken in statistical post-processing of NWP forecasts, is to directly estimate a parametric distribution” | concepts[47] |
| concept | forecast uncertainty — not defined in this paper's own words; quantity that must indicate forecast error for reliability — “For a probabilistic forecast to be reliable the forecast uncertainty has to be an accurate indicator of the error. A good first order approximation for this is the spread (ensemble standard deviation) to error (RMSE) ratio which should be one (Leutbecher and Palmer, 2008). A more stringent test is to use a proper probabilistic scoring rule, for example the continuous ranked probability score (CRPS)” | concepts[48] |
| concept | spread (ensemble standard deviation) to error (RMSE) ratio — not defined in this paper's own words; first-order reliability diagnostic expected to equal one — “For a probabilistic forecast to be reliable the forecast uncertainty has to be an accurate indicator of the error. A good first order approximation for this is the spread (ensemble standard deviation) to error (RMSE) ratio which should be one (Leutbecher and Palmer, 2008). A more stringent test is to use a proper probabilistic scoring rule, for example the continuous ranked probability score (CRPS)” | concepts[49] |
| concept | proper probabilistic scoring rule — not defined in this paper's own words; more stringent probabilistic forecast test — “For a probabilistic forecast to be reliable the forecast uncertainty has to be an accurate indicator of the error. A good first order approximation for this is the spread (ensemble standard deviation) to error (RMSE) ratio which should be one (Leutbecher and Palmer, 2008). A more stringent test is to use a proper probabilistic scoring rule, for example the continuous ranked probability score (CRPS)” | concepts[50] |
| concept | continuous ranked probability score (CRPS) — not defined in this paper's own words; proper scoring rule proposed for probabilistic forecasts — “For a probabilistic forecast to be reliable the forecast uncertainty has to be an accurate indicator of the error. A good first order approximation for this is the spread (ensemble standard deviation) to error (RMSE) ratio which should be one (Leutbecher and Palmer, 2008). A more stringent test is to use a proper probabilistic scoring rule, for example the continuous ranked probability score (CRPS)” | concepts[51] |
| concept | Climate models — not defined in this paper's own words; models evaluated by long-term statistical agreement rather than a future atmospheric state — “Even though weather and climate deal with the same underlying physical system, they pose different forecasting challenges. In weather forecasting the goal is to predict the state of the atmosphere at a specific time into the future. This is only possible up to the prediction horizon of the atmosphere, which is thought to be at roughly two weeks. Climate models, on the other hand, are evaluated by comparing long-term statistics to observations” | concepts[52] |
| concept | prediction horizon of the atmosphere — not defined in this paper's own words; roughly two-week limit for state-specific weather forecasts — “Even though weather and climate deal with the same underlying physical system, they pose different forecasting challenges. In weather forecasting the goal is to predict the state of the atmosphere at a specific time into the future. This is only possible up to the prediction horizon of the atmosphere, which is thought to be at roughly two weeks. Climate models, on the other hand, are evaluated by comparing long-term statistics to observations” | concepts[53] |
| concept | historical manifold of states — not defined in this paper's own words; range of atmospheric states represented in historical training data — “future climates will produce atmospheric states that lie outside of the historical manifold of states. Plain neural networks are very bad at extrapolating to climates beyond what they have seen in the training dataset (Rasp et al., 2018). For this reason, climate change simulations with current data-driven methods are likely not a good idea” | concepts[54] |
| concept | Plain neural networks — not defined in this paper's own words; models described as poor at extrapolating to unseen future climates — “future climates will produce atmospheric states that lie outside of the historical manifold of states. Plain neural networks are very bad at extrapolating to climates beyond what they have seen in the training dataset (Rasp et al., 2018). For this reason, climate change simulations with current data-driven methods are likely not a good idea” | concepts[55] |
| concept | generative adverserial networks (GANs) — not defined in this paper's own words; generative models proposed for more realistic stochastic field synthesis — “For synthesis tasks, generative adverserial networks (GANs) (Goodfellow et al., 2014) were shown to be particularly powerful for creating realistic natural images and fluid flows (Xie et al., 2018). This might be attractive since minimizing a mean loss, such as the MSE, for random or stochastic data leads to unrealistically smooth predictions as seen in Fig. 3. Conditional GANs (Mirza and Osindero, 2014; Isola et al., 2016) could potentially alleviate this issue but it is still unclear to what extent GAN predictions are able to recover the multi-variate distribution of the training samples” | concepts[56] |
| concept | Conditional GANs — not defined in this paper's own words; conditional generator suggested as a possible remedy for smooth mean-loss predictions — “For synthesis tasks, generative adverserial networks (GANs) (Goodfellow et al., 2014) were shown to be particularly powerful for creating realistic natural images and fluid flows (Xie et al., 2018). This might be attractive since minimizing a mean loss, such as the MSE, for random or stochastic data leads to unrealistically smooth predictions as seen in Fig. 3. Conditional GANs (Mirza and Osindero, 2014; Isola et al., 2016) could potentially alleviate this issue but it is still unclear to what extent GAN predictions are able to recover the multi-variate distribution of the training samples” | concepts[57] |
| correlation r | as written: correlation coefficient — “The forecasts were evaluated using the root mean squared error and the anomaly correlation coefficient of Z500” | content_tags.statistics[0] |
| regression β | as written: linear regression — “Solid lines for linear regression and CNN indicate iterative forecasts, while dots represent direct forecasts for 3 and 5 days lead time.” | content_tags.statistics[1] |
| standard-deviation SD/σ | — “A good first order approximation for this is the spread (ensemble standard deviation) to error (RMSE) ratio which should be one (Leutbecher and Palmer, 2008).” | content_tags.statistics[2] |
Full text — tagged
The reference list is not part of the text we hold for this paper. Our copy was extracted from the publisher's page and the bibliography did not survive that extraction, so the article below ends before its references. Nothing has been removed by us.
0 2 0 2 n u J 1 1 ] h p -[ Passage withheld — in our stored copy of this paper these characters are set one per token and run backwards, which is what a PDF extractor returns for a table or a caption printed sideways. It is not what the page of the article looks like, so we do not reproduce it. The tags cut from this paper are unaffected. ] WeatherBench: A benchmark datasetterm·key for data-driventerm·key weather forecastingterm·dom Stephan Rasp1, Peter D. Dueben2, Sebastian Scher3, Jonathan A. Weyn4, Soukayna Mouatadid5, and Nils Thuerey1 1Technical University of Munich, Germany 2European Centre for Medium-rangeterm Weather Forecasts, Reading, UK 3Department of Meteorology and Bolin Centre for Climate Research, Stockholm University, Sweden 4Department of Atmospheric Sciences, University of Washington, Seattle, USA 5Department of Computer Science, University of Toronto, Canada Correspondence: Stephan RasppersName (stephan.rasppersName@tum.de) Abstract. Data-driventerm·key approaches, most prominently deep learningterm·ai, have become powerful tools for prediction in many domains. A natural question to ask is whether data-driventerm·key methods could also be used to predict global weather patterns days in advance. First studies show promise but the lack of a common dataset and evaluation metricsterm·key make inter-comparison between studies difficult. Here we present a benchmark datasetterm·key for data-driventerm·key medium-rangeterm weather forecastingterm·dom, a topic of high scientific interest for atmospheric and computer scientists alike. We provide data derived from the ERA5term·key archive that has been processed to facilitate the use in machine learningterm·ai models. We propose simple and clear evaluation metricsterm·key which will enable a direct comparison between different methods. Further, we provide baseline scores from simple linear regressionterm·stat techniques, deep learningterm·ai models, as well as purely physical forecasting models. The dataset is publicly available at https://github.com/pangeo-data/WeatherBench and the companion code is reproducible with tutorials for getting started. We hope that this dataset will accelerate research in data-driventerm·key weather forecastingterm·dom.
Introduction Deep learningterm·ai, a branch of machine learningterm·ai based on multi-layered artificial neural networksterm·key, has proven to be a powerful tool for a wide range of tasks, most notably image recognition and natural language processing (LeCun et al., 2015). More recently, deep learningterm·ai has also been used in many fields of natural science. Much of the success of deep learningterm·ai is based on the ability of neural networksterm·key to recognize patterns in high-dimensional spaces. A natural question to ask then is whether deep learningterm·ai can also be used to predict future weather patterns.
Currently, weather (and climate) predictions are based on purely physical computer models, in which the governing equations, or our best approximation thereof, of the atmosphere and ocean are solved on a discrete numerical grid (Bauer et al., 2015). Overall, this approach has been very successful. However, today’s numerical weather prediction (NWP)term·def models still have shortcoming for many important applications, for example forecasting mesoscale convective systems over Africa (Vogel et al., 2018). Furthermore, huge amounts of computing power are required, especially for creating probabilistic forecasts which are usually limited to 50 ensembleterm members or less. For these reasons and the growing popularity of machine learningterm·ai (ML) there has been a growing interest to improve and speed up NWP with data-driventerm·key approaches.
ML can be applied to weather prediction in many different ways. Two long-standing applications of ML are post-processingterm – the correction of statistical biases in the output of physical models – and statistical forecastingterm – the prediction of variables not directly output by the physical model. Traditionally, this has been done using simple linear techniques but more recently modern machine learningterm·ai approaches like random forests or neural networksterm·key have been explored (Gagne et al., 2014; Taillardat et al., 2016; McGovern et al., 2017; Lagerquist et al., 2017; RasppersName and Lerch, 2018). Typically, these approaches target very specific variables or locations whereas the general evolution of the atmosphere is still predicted by a physical model. Another application that has recently been explored using ML is nowcastingterm, which describes the short range (up to 6 hours) prediction of precipitation by directly extrapolating radar observationterm without a physical model involved (Shi et al., 2015, 2017; Agrawal et al., 2019; Grönquist et al., 2020).
Yet another direction for ML research is hybrid modelingterm, in which a physical model is combined with data-driventerm·key components, for example replacing heuristic cloud or radiation parameterizations (Chevallier et al., 1998; Krasnopolsky et al., 2005; RasppersName et al., 2018; Brenowitz and Bretherton, 2018; Yuval and O’Gorman, 2020). The key idea behind these approaches is to only replace uncertain (e.g. clouds) or computationally expensive (e.g. line-by-line radiation) model components with machine learning emulatorsterm and leave other model components (e.g. large-scale dynamics) untouched. However such hybrid models also have drawbacks. First, the interaction between physical and machine learningterm·ai components are poorly understood and can lead to unexpected instabilities and biases (Brenowitz and Bretherton, 2019). Second, they are difficult to implement from a technical perspective because one has to interface the machine learningterm·ai components with complex climate modelterm code, typically written in Fortran.
Here we focus on purely-data driven predictionterm of the global atmospheric flow in the medium-rangeterm. Specifically, we select lead times of 3 and 5 days, for which the atmosphere is still reasonably deterministic but also exhibits complex nonlinear behaviour, such as baroclinic instabilitiesterm and tropical cyclogenesisterm. This forecast range is important from a societal point of view because it delivers crucial information for disaster preparation, for example for flooding, cold and hot spells or damaging winds (Lazo et al., 2009). Creating a good medium-rangeterm forecast requires understanding complex atmospheric dynamics and the interplay between several variables across a range of scales. This sets this challenge apart from post-processingterm and statistical forecastingterm, in which the large-scale dynamics are predicted by a physical model, and nowcastingterm, in which the considered evolution is univariate and short-term. In other words, this benchmarkterm·mth closely emulates the task performed by physical NWP models.
There are several motivations for considering a purely-data driven approach. As mentioned above current NWP is computationally expensive and, nevertheless, has low skill for certain applications. If data-driventerm·key models were able to learn a more efficient representation of the underlying dynamical and physical equations, they might enable computationally cheaper forecasts. This can be useful for many applications, for example creating very large ensemblesterm to better estimate the probability of extremeterm·dom events. It is also possible that by learning from a diverse set of data sources, data-driventerm·key models can outperform physical models in areas where the latter struggle. While in this benchmarkterm·mth challenge the focus is on upper-level fields of pressure and temperature – for which physical models perform very well – the hope is that the insights gained from this task can be leveraged for more impactful application. Further, recent research into interpretable machine learningterm·ai might provide scientists with new analysis tools (McGovern et al., 2019; Toms et al., 2019). Finally, there is the basic scientific question to what extent purely data-driventerm·key models can learn the underlying dynamics of the atmosphere.
Note also that while this benchmarkterm·mth is framed as a data-driventerm·key prediction challenge, the proposed framework can also be applied to post-processingterm using the same metrics.
In machine learningterm·ai research, the data-driventerm·key prediction of future states is an active area of research with applications from language translation (Sutskever et al., 2014), over audio signals (Oord et al., 2016), to numerical simulations (Morton et al., 2018). In this context, weather forecasts are a particularly challenging task. The behavior is highly complex and non-linear, but also exhibits some recurring patterns, albeit only on local scales (Hamill and Whitaker, 2006). As such the the proposed benchmarkterm·mth poses interesting challenges for deep learningterm·ai algorithms, e.g., to evaluate different architectures (Ronneberger et al., 2015; He et al., 2015; Huang et al., 2016), regularization methods (Krogh and Hertz, 1992; Srivastava et al., 2014; Xie et al., 2017) or optimizers (Graves, 2013; Kingma and Ba, 2014).
In the last couple of years, several studies (summarized in Section 2) have pioneered data-driventerm·key, global, medium-rangeterm weather prediction. All of them show that there is some potential in this approach but also highlight the need for further research. In particular, we currently lack a common benchmark challengeterm to accelerate progress. Benchmark datasetsterm can have a huge impact because they make different algorithms quantitatively inter-comparable and foster constructive competition, particularly in a nascent direction of research. Famous examples are the computer vision datasets MNIST (LeCun et al., 1998) and ImageNet (Russakovsky et al., 2015). Further, well-curated benchmark datasetsterm make it easier for people from different fields to work on a problem (Ebert-Uphoff et al., 2017).
Here, we propose a benchmarkterm·mth problem for data-driventerm·key weather forecastingterm·dom. We provide a ready-to-use dataset for download along with specific metrics to compare different approaches. In this paper, we start by reviewing the previous work done on this topic (Section 2), describe the dataset (Section 3) and the evaluation metricsterm·key (Section 4), and provide several baseline models (Section 5). Finally, we will highlight several promising directions for further research (Section 6) and conclude with a big picture view (Section 8).
2 Overview of previous work In this section, we briefly describe the four existing studies on predicting the large-scale atmospheric state in the medium-rangeterm with a focus on the data, methods and evaluation.
2.1 DuebenpersName and Bauer (2018) In this study, the authors trained a neural networkterm·key to predict 500 hPa geopotentialterm·var (Z500; see Section 4 for details on commonly used fields), and in some experiments 2 meter-temperature, 1 hour ahead. The training dataterm·dom was taken from the ERA5term·key archive for the time period from 2010 to 2017 and regridded to a 6 degree latitude-longitude grid. Two neural networkterm·key variants were Figure 1. Schematic of data-driventerm·key weather forecastingterm·dom. a) Example of direct weather prediction for 5 days lead time. The input to the neural networkterm·key are fields on a latitude-longitude grid. The fields can be several levels of the same variable and/or different variables. The goal is to predict the same fields some time ahead. b) Iterative forecasts are created from data-driventerm·key models trained on a shorter lead time, for example 6 hours, which are then iteratively called up to the required forecast lead timeterm·var.
used, a fully connected neural networkterm·key and a spatially localized network, similar to a convolutional neural networkterm·key (CNN).
After training they then created iterative forecasts up to 120 h lead time for 10 month validation period. They compared their data-driventerm·key forecasts to an operational NWP model and the same model run at a spatial resolution comparable to the datadriventerm·key method. One interesting detail is that their networks predict the difference from one time step to the next, instead of the absolute field. To create these iterative forecasts, they use a third-order Adams-Bashford explicit time-stepping scheme. The CNN predicting only geopotentialterm·key performed best but was unable to beat the low-resolution physical baseline.
2.2 ScherpersName (2018) and ScherpersName and Messori (2019b) These two studies addressed the issue of data-driventerm·key weather forecastingterm·dom in a simplified reality setting. Long runs of simplified General Circulation Models (GMCs) were used as “reality”. Neural networksterm·key were trained to predict the model fields several days ahead. The neural networkterm·key architecture are CNNs with an encoder-decoder setup. They take as input the instantaneous 3D model fields at one timestep, and output the same model fields at some time later. In ScherpersName (2018), a separate network was trained for each lead-time up to 14 days. ScherpersName and Messori (2019b) trained only on 1-day forecasts, and constructed longer forecasts iteratively. Interestingly, networks trained to directly predict a certain forecast time, e.g. 5 days, outperformed iterative networks. The forecasts were evaluated using the root mean squared errorterm and the anomaly correlation coefficientterm·var of Z500 and Longitudet = 0t = 5 daysa) Direct predictionb) Iterative predictiont = 0t = 6 hourst = 5 daysLatitudeChannels = Variables xLevels 800 hPa temperature. ScherpersName (2018) used a highly simplified GCM without hydrological cycle, and achieved very high predictive skill. Additionally, they were able to create stable "climate" runs (long series of consecutive forecasts) with the network. ScherpersName and Messori (2019b) used several more realistic and complex GCMs. The data-driventerm·key model achieved relatively good shortterm forecast skill, but was unable to generate stable and realistic “climate” runs. In terms of neural-network architectures they showed that architectures tuned on simplified GCMs also work on more complex GCMs, and that the same architecture also has some prediction skill on single-level reanalysisterm·key data.
2.3 WeynpersName et al. (2019) In this study, reanalysis-derived Z500 and 700-300 hPa thickness at 6-hourly time steps are predicted with deep CNNs. The data are from the Climate Forecast System (CFS) Reanalysisterm·key from 1979–2010 with 2.5-degree horizontal resolution and cropped to the northern hemisphere. The authors used similar encoder-decoder convolutional networks as those used by ScherpersName (2018) and ScherpersName and Messori (2019b) but also experimented with adding a convolution long short-term memory (LSTM; Hochreiter and Schmidhuber, 1997) hidden layer. As in ScherpersName and Messori (2019b), forecasts are generated iteratively by feeding the model’s outputs back in as inputs. The authors found that using two input time steps, 6 h apart, and predicting two output time steps, performed better than using a single step. Their best CNN forecast outperforms a climatology benchmarkterm·mth at up to 120 h lead time, and appears to correctly asymptote towards persistence forecaststerm at longer lead times up to 14 days.
These three approaches outline promising first steps towards data-driventerm·key forecasting. The differences of the proposed methods already highlight the importance of a common benchmarkterm·mth case to compare prediction skill.
3 Dataset For the proposed benchmarkterm·mth, we use the ERA5 reanalysis datasetterm (Hersbach et al., 2020) for training and testing. Reanalysis datasetsterm provide the best guess of the atmospheric state at any point in time by combining a forecast modelterm with the available observationsterm. The raw data is available hourly for 40 years from 1979 to 2018 on a 0.25°latitude-longitude grid (721×1440 grid points) with 37 vertical levels.
Since this raw dataset is very large (a single vertical level for the entire time period amounts to almost 700GB of data), we regrid the data to lower resolutions. This is also a more realistic use case, since very high resolutions are still hard to handle for deep learningterm·ai models because of GPU memory constraints and I/O speed. In particular, we chose 5.625° (32×64 grid points), 2.8125° (64×128 grid points) and 1.40525° (128×256 grid points) resolution for our data. The regridding was done with the xesmf Python package (Zhuang, 2019) using a bilinear interpolation. Powers of two for the grid are used since this is common for many deep learningterm·ai architectures where image sizes are halved in the algorithm. Further, for 3D fields we selected 13 vertical levels: 50, 100, 150, 200, 250, 300, 400, 500, 600, 700, 850, 925, 1000 hPa. Note that it is common to use pressure in hecto-Pascals as a vertical coordinate instead of physical height. The pressure at sea level is approximately 1000 hPa and decreases roughly exponentially with height. 850 hPa is at around 1.5 km height. 500 hPa is at around 5.5 km height. If the surface pressure is smaller than a given pressure level, for example at high altitudes, the pressure-level values are interpolated.
The selected pressure levels contain the seven pressure levels that are commonly used for 3D output by the climate modelsterm in the Coupled Model Intercomparison Project Phase 6 (CMIP6, Eyring et al., 2016) which could be useful for pretraining. One regridded historical climate run is also available from the data repository with a template workflow for downloading further CMIP data on the Github repository.
The processed data (see Table 1) are available at https://mediatum.ub.tum.de/1524895 (RasppersName et al., 2020). The data are split into yearly NetCDF files for each variable and resolution, packed in a zip file. The entire dataset at 5.625° resolution has a size of 191GB. Individual variables amount to around 25GB three-dimensional and 2GB for two-dimensional fields. File sizes for 2.8125° and 1.40525° resolutions are a factor 4 and 16 times larger. Data processing was organized using Snakemake (Koster and Rahmann, 2012). For further instructions on data downloading visit the Github page1. The available variables were chosen based on meteorological consideration. Geopotentialterm·key, temperature, humidity and wind are prognostic state variables in most physical NWP and climate modelsterm. Geopotentialterm·key at a certain pressure level p, typically denoted as Φ with units of m2s−2, defined as z at p (cid:90) g dz(cid:48) Φ = (1) where z describes height in meters and g = 9.81 m s−2 is the gravitational acceleration. Horizontal relative vorticity, defined as ∂v/∂x − ∂u/∂y, describes the rotation of air at a given point in space. Potential vorticity (Hoskins et al., 1985; Holton, 2004) is a commonly used quantity in synoptic meteorology which combines the rotation (vorticity) and vertical temperature gradient of the atmosphere. It is defined as P V = ρ−1ζa · ∇θ, where ρ is the density, ζa is the absolute vorticity (relative plus the Earth’s rotation) and θ is the potential temperature. In addition to the three-dimensional fields, we also include several twodimensional fields: 2 meter-temperature is often used as an impact variable because of its relevance for human activities and is directly affected by the diurnal solar cycle; 10 meter-wind is also an important impact-related forecast variable, for example for wind energy; similarly, total cloud cover is an essential variable for solar energy forecasting. We also included precipitation but urge caution since precipitation in reanalysis datasetsterm often shows large deviation from observationsterm (e.g. Betts et al., 2019; Xu et al., 2019). Finally, we added the top-of-atmosphere incoming solar radiation as it could be a useful input variable to encode the diurnal cycle. Further, there are several potentially important time-invariant fields, which are contained in the constants file. The first three variables enclose information about the surface: the land-sea mask is a binary field with ones for land points; the soil type consists of seven different soil categories2; orography is simply the surface height. In addition, we included two-dimensional fields with the latitude and longitude values at each point. Particularly the latitude values could become important for the network to learn latitude-specific information such as the grid structure or the Coriolis effect (see Section 6). The Github code repository includes all scripts for downloading and processing of the data. This enables users to download additional variables or regrid the data to a different resolution.
1https://github.com/pangeo-data/WeatherBench 2Coarse = 1, Medium = 2, Medium fine = 3, Fine = 4, Very fine = 5, Organic = 6, Tropical organic = 7, see https://apps.ecmwf.int/codes/grib/param-db?
id=43 Table 1. List of variables contained in the benchmark datasetterm·key.
Short name Description t2m u10 v10 tcc tisr lsm slt Proportional to the height of a pressure level Temperature Mixing ratio of water vapor Humidity relative to saturation Wind in x/longitude-direction Wind in y/latitude direction Relative horizontal vorticity Potential vorticity Temperature at 2 m height above surface Wind in x/longitude-direction at 10 m height Wind in y/latitude-direction at 10 m height Fractional cloud cover Hourly precipitation Accumulated hourly incident solar radiation File containing time-invariant fields Land-sea binary mask Soil-type categories orography Height of surface lat2d lon2d 2D field with latitude at every grid point 2D field with longitude at every grid point Unit [m2s−2] [K] [kg kg−1] [%] [m s−1] [m s−1] [1 s−1] [K m2 kg−1 s−1] [K] [m s−1] [m s−1] (0–1) [m] [J m−2] (0/1) see text [m] [°] [°] Levels 13 levels 13 levels 13 levels 13 levels 13 levels 13 levels 13 levels 13 levels Single level Single level Single level Single level Single level Single level Single level Single level Single level Single level Single level Long name geopotentialterm·key temperature specific_humidity relative_humidity u_component_of_wind v_component_of_wind vorticity potential_vorticity 2m_temperature 10m_u_component_of_wind 10m_v_component_of_wind total_cloud_cover total_precipitation toa_incident_solar_radiation constants land_binary_mask soil_type orography latitude longitude 4 Evaluation Evaluation is done for the years 2017 and 2018. To make sure no overlap exists between the training and test dataset, the first test date is 1 January 2017 00UTC plus forecast time (i.e. for a three day forecast the first test date would be 4 January 2017 00UTC) while the last training target is 31 December 2016 23UTC. Further, the evaluation presented here is done on 5.625° resolution3. This means that predictions at higher resolutions have to be downscaled to the evaluation resolution. We also evaluated some baselines at higher resolutions and found that the scores were almost identical with differences smaller than 1%. Therefore we are reassured that little information is lost by evaluating at a coarser resolution.
A note on validation and testing: In machine learningterm·ai it is good practice to split the data into three partsterm: the training, validation and test sets. The training datasetterm is used to actually fit the model. The validation datasetterm is used during experimentation 3The evaluation of all baselines in this paper are done in this Jupyter notebook: https://github.com/pangeo-data/WeatherBench/blob/master/notebooks/ 4-evaluation.ipynb to check the model performance on data not seen during training. However, there is the danger that through continued tuning of hyperparameters one unwillingly overfits to the validation datasetterm. Therefore it is advisable to keep a third, testing, datasetterm for final evaluations of model performance. For this benchmarkterm·mth this final evaluation is done for the years 2017 and 2018.
Therefore, we strongly encourage users of this dataset to pick a period from 1979 to 2016 for validation of their models for hyperparameter tuning. Because meteorological fields are highly correlated in time, it is advisable to choose a longer contiguous chunk of data for validation instead of a completely random split. Here we chose the year 2016 for validation.
We chose 500 hPa geopotentialterm·var and 850 hPa temperature as primary verification fields. Geopotentialterm·key at 500 hPa pressure, often abbreviated as Z500, is a commonly used variable that encodes the synoptic-scale pressure distribution. It is the standard verification variable for most medium-rangeterm NWP models. Note that geopotentialterm·key height, also commonly used, is defined as Φ/g with units of meters. We picked 850 hPa temperature as our secondary verification field because temperature is a more impact-related variable. 850 hPa is usually above the planetary boundary layer and therefore not affected by diurnal variations but provides information about broader temperature trends, including cold spells and heat waves. In addition we also provide some baseline scores for total 6-hourly accumulated precipitation (TP; but noting the dubious quality mentioned above) and 2-meter temperature (T2M). However, they will not be further discussed here.
We chose the root mean squared errorterm (RMSE) as our primary metric because it is easy to compute and mirrors the loss used for most ML applications. We define the RMSE as the mean latitude-weighted RMSE over all forecasts: RMSE = 1 Nforecasts Nforecasts(cid:88) (cid:118) (cid:117) (cid:117) (cid:116) 1 NlatNlon Nlat(cid:88) Nlon(cid:88) L(j)(fi,j,k − ti,j,k)2 (2) where f is the model forecast and t is the ERA5term·key truth. L(j) is the latitude weighting factor for the latitude at the jth latitude index: L(j) = cos(lat(j)) (cid:80)Nlat j cos(lat(j)) 1 Nlat (3) In addition, we also evaluate the baselines using the latitude weighted anomaly correlation coefficientterm·var (ACC; see Section 7.6.4 of Wilks, 2006) and the mean absolute error (MAE). The tables and figures can be found in the Appendix A. For smooth fields like Z500 and T850 the qualitative differences between the metrics are small. For intermittent fields like precipitation the choice of metric matters a lot more.
5 Baselines To evaluate the skill of a forecasting model it is important to have baselines to compare to. In this section, we compute scores for several baselines. The results are summarized in Fig. A3 and Table 2.
5.1 Persistence and Climatology The two simplest possible forecasts are a) a persistence forecastterm in which the fields at initialization time are used as forecasts ("tomorrow’s weather is today’s weather"), and b) a climatological forecastterm. For the climatological forecastterm, two different Figure 2. RMSE of a) 500 hPa geopotentialterm·var and b) 850 hPa temperature for different baselines at 5.625° resolution. Solid lines for linear regressionterm·stat and CNN indicate iterative forecasts, while dots represent direct forecasts for 3 and 5 days lead time.
Table 2. Baseline RMSE for 3 and 5 days forecast time at 5.625° resolution. Best machine learningterm·ai baseline and physical model are highlighted. TP is 6 hourly accumulated precipitation.
Baseline Persistence Climatology Weekly climatology RMSE (3 days / 5 days) Z500 [m2 s−2] T850 [K] T2M [K] TP [mm] 936 / 1033 4.23 / 4.56 3.00 / 3.27 3.23 / 3.24 1075 816 5.51 3.50 6.07 3.19 2.36 2.32 Linear regressionterm·stat (direct) 693 / 783 3.19 / 3.44 2.39 / 2.60 2.37 / 2.37 Linear regressionterm·stat (iterative) 718 / 810 626 / 757 3.17 / 3.48 2.87 / 3.37 CNN (direct) CNN (iterative) IFS T42 IFS T63 Operational IFS 1114 / 1559 4.48 / 9.69 489 / 743 268 / 463 154 / 334 3.09 / 3.83 3.21 / 3.69 1.85 / 2.52 2.04 / 2.44 1.36 / 2.03 1.35 / 1.77 2.36 / 2.59 climatologies were computed from the training datasetterm (1979–2016): first, a single mean over all times in the training datasetterm and, second, a mean computed for each of the 52 calendar weeks. The weekly climatology is significantly better, approximately matching the persistence forecastterm between 1 and 2 days, since it takes into account the seasonal cycle. This means that to be useful, a forecast system needs to beat the weekly climatology and the persistence forecastterm.
012345Forecast time [days]020040060080010001200Z500 RMSE [m2 s2]a) Z500PersistenceClimatologyWeekly clim.OperationalIFS T42IFS T63LR (iterative)CNN (iterative)LR (direct)CNN (direct)012345Forecast time [days]0123456T850 RMSE [K]b) T850 5.2 Operational NWP model The gold standard of medium-rangeterm NWP is the operational IFS (Integrated Forecast System) model of the European Center for Medium-rangeterm Weather Forecastingterm·dom (ECMWF)4. We downloaded the forecasts for 2017 and 2018 from the THORPEX Interactive Grand Global Ensembleterm (TIGGE; Bougeault et al., 2010) archive5, which contains the operational forecasts, initialized at 00 and 12 UTC regridded to a 0.5° by 0.5° grid, which we further regridded to 5.625°. Note that the forecast error starts above zero because the operational IFS is initialized from a different analysis. Operational forecasting is computationally very expensive. The current IFS deterministic forecast is computed on a cluster with 11,664 cores. One 10 day forecast at 10 km resolution takes around 1 hour of real time to compute.
5.3 Physical NWP model run at coarser resolution To provide physical baselines more in line with the computational resources of a data-driventerm·key model, we ran the IFS model at two coarser horizontal resolutions, T42 (approximately 2.8° or 310 km resolution at the equator (NCAR)) with 62 vertical levels and T63 (approximately 1.9° or 210 km) with 137 vertical levels. The T42 run was initialized from ERA5term·key whereas the T63 run was initialized from the operational analysis. The gap in skill at t = 0 is caused by the conversion to spherical coordinates at coarse resolutions. For Z500 the skill for these two runs lies in-between the operational IFS and the machine learningterm·ai baselines. For T850, the T42 run is significantly worse. The likely reason for this is that temperature close to the ground is much more affected by the resolution and representation of topography within the model. Further, the model was not specifically tuned for these resolutions. Computationally, a single forecast takes 270 seconds for the T42 model and 503 seconds for the T64 model on a single XC40 node with 36 cores. Since the computational costs and resolutions of these runs are much closer to those of a data-driventerm·key method, beating those baselines should be a realistic target. note, however, that the model was not tuned to run at such coarse resolutions.
5.4 Linear regressionterm·stat As a first purely data-driventerm·key baseline we fit a simple linear regressionterm·stat model. For the direct predictionsterm a separate model was trained for each of the four variables. For this purpose the 2D fields were flattened from 32×64 → 2048. This was done for 3 d and 5 d forecast time. In addition an iterative modelterm for Z500 and T850 was trained. Here we use a single linear regressionterm·stat to predict 6 hours ahead where the two fields are concatenated (2×32×64 → 4096). The advantage of iterative forecasts is that a single model is able to make predictions for any forecast time rather than having to train several models. For iterative forecasts the model takes its previous output as inputterm for the next step. To create a 5 day iterative forecast the model trained to predict 6 hour forecasts is called 20 times. For this model, the iterative forecast performs just as well as the direct forecast due to its linear nature. At 5 days, the linear regressionterm·stat forecast is about as good as the weekly climatology.
4https://www.ecmwf.int/en/forecasts/documentation-and-support 5The TIGGE data for total precipitation and 2m temperature was damaged for 2017. For this reason the TIGGE evaluation for these variables is only done using the 2018 data.
5.5 Simple convolutional neural networkterm·key As our deep learningterm·ai baseline we chose a simple fully-convolutional neural networkterm. CNNs are the natural choice for spatial data since they exploit translational invariancesterm in images/fields. Here we train a CNN with 5 layers. Each hidden layer has 64 channels with a convolutional kernel of size 5 and ELU activations (Clevert et al., 2015). The input and output layers have two channels, representing Z500 and T850. The model was trained using the Adam optimizer (Kingma and Ba, 2014) and a mean squared error loss function. The total number of trainable parameters is 313,858. We implemented periodic convolutionsterm in the longitude direction but not the latitude direction. The implementation can be found in the Github repository. The direct CNN forecasts beat the linear regressionterm·stat forecasts for 3 and 5 days forecast time. However, at 5 days these forecasts are only marginally better than the weekly climatology (see Table 2). This baseline, however, is to be seen simply as a starting point for more sophisticated data driven methods. The iterative CNN forecast, which equivalently to the linear regressionterm·stat iterative forecast was created by chaining together 6 hourly predictions, performs well up to around 1.5 days but then the network’s errors grow quickly and diverge. This confirms the findings of ScherpersName and Messori (2019a) whose experiments showed that training with longer lead time yields better results than chaining together short-term forecasts. However, the poor skill of the iterative forecast could easily be a result of using an overly simplistic network architecture. The iterative forecasts of WeynpersName et al. (2019), who employ a more complex network structure, show stable long term performance up to two weeks with realistic statistics.
5.6 Example forecasts To further illustrate the prediction task, Fig. 3 shows example geopotentialterm·key and temperature fields. The ERA5term·key temporal differences show several interesting features. First, the geopotentialterm·key fields and differences are much smoother compared to the temperature fields. The differences in both fields are also much smaller in the tropics compared to the extratropics where propagating fronts can cause rapid temperature changes. An interesting feature is detectable in the 6h Z500 difference field in the tropics. These alternating patterns are signatures of atmospheric tides.
The CNN forecasts for 6h lead time are not able to capture these wave-like patterns which hints at a failure to capture the basic physics of the atmosphere. For 5 days forecast time the CNN model predicts unrealistically smooth fieldsterm. This is likely a result of two factors: first, the two input fields used in this baseline CNN contain insufficient information to create a skillful 5 day forecast; and second, at 5 days the atmosphere already shows some chaotic behaviorterm which causes a model trained with a simple RMSE lossterm to predict smooth fields (see Section 6). The IFS operational forecast has much smaller errors than the CNN forecast. It is able to capture the propagation of tropical waves. Its main errors appear at 5 days in the mid-latitudes where extratropical cycles are in slightly wrong positions.
Figure 3. Example fields for 2017-01-01 00UTC initialization time. The top two rows show the ERA5term·key "truth" fields for geopotentialterm·key (Z500) and temperature (T850) at initialization time (t=0h) and for 6h and 5d forecast time. In addition, the difference between the forecast times and the initialization time is shown. The third and fourth rows show the forecasts from the CNN model. Rows five and six show the IFS operational model. For the CNN forecasts the first column is identical to the ERA5term·key truth. We selected the 6h iterative CNN model for the 6h forecast but the 5d direct CNN model for the 5 day forecast. For the IFS the initial states (t=0h) differ slightly albeit not visibly. In addition to the forecast fields the error relative to the ERA5term·key "truth" is shown in the third and fifth columns. Please note that the colorbars for the difference fields change.
ERA5term·key “Truth”CNN forecastsIFS forecasts 6 Discussion 6.1 Weather-specific challenges From a ML perspective, state-to-state weather prediction is similar to image-to-image translationterm. For this sort of problem many deep learningterm·ai techniques have been developed in recent years (Kaji and Kida, 2019). However, forecasting weather differs in some important ways from typical image-to-image applications and raises several open questions.
First, the atmosphere is three-dimensional. So far, this aspect has not been taken into account. In the networks of ScherpersName and Messori (2019a), for example, the different levels have been treated as separate channels of the CNN. However, simply using a three-dimensional CNN might not work either because atmospheric dynamics and grid spacings change in the vertical, thereby violating the assumption of translation invarianceterm which underlies the effectiveness of CNNs. This directly leads to the next challenge: On a regular latitude-longitude grid, the dynamics also change with latitude because towards the poles the grid cells become increasingly stretched. This is in addition to the Coriolis effect, the deflection of wind caused by the rotation of Earth, which also depends on latitude. A possible solution in the horizontal could be to use spherical convolutions (Cohen et al., 2018; Perraudin et al., 2019; Jiang et al., 2019) or to feed in latitude information to the network.
Another potential issue is the limited amount of training dataterm·dom available. 40 years of hourly data amounts to around 350,000 samples. However, the samples are correlated in time. If one assumes that a new weather situation occurs every day, then the number of samples is reduced to around 15,000. Without empirical evidence it is hard to estimate whether this number is sufficient to train complex networks without overfitting. Should overfitting be a problem, one could try transfer learningterm.
In transfer learningterm, the network is pretrained on a similar task or dataset, for example, climate modelterm simulations, and then finetuned on the actual data. This is common practice in computer vision and has been successfully applied to seasonal ENSO forecasting (Ham et al., 2019). Another common method to prevent overfitting is data augmentationterm, which in traditional computer vision is done by e.g. randomly rotating or flipping the image. However, many of the traditional data augmentationterm techniques are questionable for physical fields. Random rotations, for example, will likely not work for this dataset since the x and y directions are physically distinct. Thus, finding good data augmentationterm techniques for physical fields is an outstanding problem. Using ensembleterm analyses and forecasts could provide more diversity in the training datasetterm.
Finally, there are technical challenges. Data for a single variable with ten levels at 5.625° resolution take up around 30 GB of data. For a network with several variables or even at higher resolution, the data might not fit into CPU RAMterm any more and data loading could become a bottleneck. For image files, efficient data loaders have been created6. For netCDF files, however, so far no efficient solution exists to our knowledge. Further, one can assume that to create a competitive data-driventerm·key NWP model, high resolutions have to be used, for which GPU RAMterm quickly becomes a limitation. This suggests that multi-GPU training might be necessary to scale up this approach (potentially similar to the technical achievement of Kurth et al. (2018)).
6See e.g. https://keras.io/preprocessing/image/ or https://pytorch.org/tutorials/beginner/data_loading_tutorial.html. One promising but so far unexplored option is to use Tensorflow’s TFRecords (https://www.tensorflow.org/tutorials/load_data/tfrecord) 6.2 Probabilistic forecasts and extremesterm·dom One important aspect that is not currently addressed by this benchmarkterm·mth is probabilistic forecastingterm. Because of the chaotic evolution of the atmosphere, it is very important to also have an estimate of the uncertainty of a forecast. In physical NWP this is done by running several forecasts, called an ensembleterm, from slightly different initial conditions and potentially with different or stochastic model physics (Palmer, 2019). From this Monte Carlo forecastterm one can then estimate a probability distribution. A different approach, which is often taken in statistical post-processingterm of NWP forecasts, is to directly estimate a parametric distributionterm (e.g. Gneiting et al., 2005; RasppersName and Lerch, 2018). For a probabilistic forecast to be reliable the forecast uncertaintyterm has to be an accurate indicator of the error. A good first order approximation for this is the spread (ensemble standard deviation) to error (RMSE) ratioterm which should be one (Leutbecher and Palmer, 2008). A more stringent test is to use a proper probabilistic scoring ruleterm, for example the continuous ranked probability score (CRPS)term (Gneiting and Raftery, 2007; Gneiting and Katzfuss, 2014). For deterministic forecast the CRPS reduces to the mean absolute error. Extending this benchmarkterm·mth to probabilistic forecastingterm simply requires computing a probabilistic score. How to produce probabilistic datadriventerm·key forecasts is a very interesting research question in its own right. We encourage users of this benchmarkterm·mth to explore this dimension.
A related issue is the question of extremeterm·dom weather situations, for example heat waves. These events are, by definition, rare, which means that they will contribute little to regular verification metrics like the RMSE. However, for society these events are highly important. For this reason, it would make sense to evaluate extremeterm·dom situations separately. But defining extremesterm·dom is ambiguous which is why there is no standard metric for evaluating extremesterm·dom. The goal of this benchmarkterm·mth is to provide a simple, clear problem. Therefore, we decided to omit extremesterm·dom for now but users are encouraged to chose their own verification of extremesterm·dom.
6.3 Climate simulations Another aspect that is untouched by the benchmarkterm·mth challenge proposed here is climate prediction. Even though weather and climate deal with the same underlying physical system, they pose different forecasting challenges. In weather forecastingterm·dom the goal is to predict the state of the atmosphere at a specific time into the future. This is only possible up to the prediction horizon of the atmosphereterm, which is thought to be at roughly two weeks. Climate modelsterm, on the other hand, are evaluated by comparing long-term statistics to observationsterm, for example the mean surface temperature (Stocker et al., 2013). ScherpersName and Messori (2019a) created iterative climate time scale runs with their data-driventerm·key models and compared first and second-order statistics. They found that the model sometimes produced a stable climate but with significant biases and a poor seasonal cycle. This indicates that, so far, iterative data-driventerm·key models have been unable to produce physically reasonable long-term predictions. This remains a key challenge for future research. While not specifically included in this benchmarkterm·mth, a good test for climate simulations is to look at long term mean statistics and the seasonal cycle as done in Figs. 6 and 7 of ScherpersName and Messori (2019a).
Climate change simulations represent another step up in complexity. To start with, external greenhouse gas forcing would have to be included. Further, future climates will produce atmospheric states that lie outside of the historical manifold of statesterm.
Plain neural networksterm are very bad at extrapolating to climates beyond what they have seen in the training datasetterm (RasppersName et al., 2018). For this reason, climate change simulations with current data-driventerm·key methods are likely not a good idea. However, research into physical machine learningterm·ai is ongoing and might offer new opportunities in the near future (e.g. Bar-Sinai et al., 2019; Beucler et al., 2019).
6.4 Promising research directions There is a wide variety of promising research directions for data-driventerm·key weather forecastingterm·dom. The most obvious direction is to increase the amount of data used for training and the complexity of the network architecture. This dataset provides a, so far, unexploited volume and diversity of data for training. It is up to future research to find out exactly which combination of variables will turn out to be useful. Further, this dataset offers a four times higher horizontal resolution than all previous studies. The hope is that this data will enable researcher to train more complex models than have previously been used.
With regards to model architecture, there is a huge variety of network architectures that can be explored. U-Nets (Ronneberger et al., 2015) have been used extensively for image segmentation tasks that require computations across several spatial scales. Resnets (He et al., 2015) are currently the state of the art for image classification and their residual nature could be a good fit for state-to-state forecasting tasks. For synthesis tasks, generative adverserial networks (GANs)term (Goodfellow et al., 2014) were shown to be particularly powerful for creating realistic natural images and fluid flows (Xie et al., 2018). This might be attractive since minimizing a mean loss, such as the MSE, for random or stochastic data leads to unrealistically smooth predictions as seen in Fig. 3. Conditional GANsterm (Mirza and Osindero, 2014; Isola et al., 2016) could potentially alleviate this issue but it is still unclear to what extent GAN predictions are able to recover the multi-variate distribution of the training samples.
7 Code and data availability The dataset is available at https://mediatum.ub.tum.de/1524895 (RasppersName et al., 2020). Code, instructions for dowloading the data and evaluating forecasts can be found at https://github.com/pangeo-data/WeatherBench.
8 Conclusions In this paper a benchmark datasetterm·key for data-driventerm·key weather forecastingterm·dom is presented. It focuses on global medium-rangeterm (roughly 2 days to 2 weeks) prediction. With the rise of deep learningterm·ai in physics, weather prediction is a challenging and interesting target because of the large overlap with traditional deep learningterm·ai tasks (Reichstein et al., 2019). While first attempts have been made in this direction, as discussed in Section 2, the field currently lacks a common dataset which enables the inter- comparison of different methods. We hope that this benchmarkterm·mth can provide a foundation for accelerated research in this area.
Loosely following Ebert-Uphoff et al. (2017), the key features of this benchmarkterm·mth are: – Scientific impact: Numerical weather forecastingterm·dom impacts many aspects of society. Currently, NWP model run on massive super-computers at very high computational cost. Building a capable data-driventerm·key model would be beneficial in many ways (see Section 1). In addition, there is the open, and highly debated, question whether fully data-driventerm·key methods are able to learn a good representation of atmospheric physics.
– Challenge for data science: While global weather prediction is conceptually similar an image-to-image task, and therefore allows for the application of many state-of-the-art deep learningterm·ai techniques, there are some unique challenges to this problem: the three-dimensional, anisotropic nature of the atmosphere; non-uniform-grids; potentially limited amounts of training dataterm·dom and the technical challenge of handling large data volumes.
– Clear metric for success: We defined a single metric (RMSE) for two fields (500 hPa geopotentialterm·var and 850 hPa temperature). These scores provide a simple measure of success for data-driventerm·key, medium-rangeterm forecast modelterm.
– Quick start: The code repository contains a quick-start Jupyter notebook for reading the data, training a neural networkterm·key and evaluating the predictions against the target data. In addition, the repository contains many functions which are likely to be used frequently, for example an implementation of periodic convolutionsterm in Keras.
– Reproducibility and citability: All baselines and results form this paper are fully reproducible from the code repository.
Further, the baseline predictions are all saved in the data repository. The data has been assigned a permanent DOI.
– Communication platform: We will use the Github code repository as an evolving hub for this project. We encourage users of this dataset to start by forking the repository and eventually merge code that might be useful for others back into the main branch. The main platform for communication, e.g. asking questions, about this project will be Github issues.
We hope that this benchmarkterm·mth will foster collaboration between atmospheric and data scientists in the ways we imagined and beyond.
(A1) (cid:80) tj,k.
Appendix A: Additional metrics The anomaly correlation coefficientterm·var (ACC) is defined as ACC = (cid:80) i,j,kt(cid:48) i,j,k L(j)f (cid:48) (cid:80) i,j,k L(j)f (cid:48)2 i,j,k i,j,k i,j,k L(j)t(cid:48)2 i,j,k (cid:113)(cid:80) where the prime (cid:48) denotes the difference to the climatology. Here the climatology is defined as climatologyj,k = 1 The mean absolute error is defined just like the MSE (Eq. 2) but with the absolute instead of the squared difference.
Ntime Figure A1. RMSE of a) 2-meter temperature and b) 6-hourly accumulated precipitation for different baselines at 5.625° resolution.
012345Forecast time [days]0123456T2M RMSE [K]a) T2MPersistenceClimatologyWeekly clim.OperationalIFS T42IFS T63LR (direct)012345Forecast time [days]0.00.51.01.52.02.53.03.5TP RMSE [mm]b) TP Figure A2. ACC of a) 500 hPa geopotentialterm·var, b) 850 hPa temperature, c) 2-meter temperature and d) 6-hourly accumulated precipitation for different baselines at 5.625° resolution.
012345Forecast time [days]0.50.60.70.80.91.0Z500 ACCa) Z500PersistenceClimatologyWeekly clim.OperationalIFS T42IFS T63LR (iterative)CNN (iterative)LR (direct)CNN (direct)012345Forecast time [days]0.600.650.700.750.800.850.900.951.00T850 ACCb) T850012345Forecast time [days]0.40.50.60.70.80.91.0T2M ACCc) T2M012345Forecast time [days]0.00.20.40.60.81.0TP ACCd) TP Figure A3. MAE of a) 500 hPa geopotentialterm·var, b) 850 hPa temperature, c) 2-meter temperature and d) 6-hourly accumulated precipitation for different baselines at 5.625° resolution.
012345Forecast time [days]0100200300400500600700Z500 MAE [m2 s2]a) Z500PersistenceClimatologyWeekly clim.OperationalIFS T42IFS T63LR (iterative)CNN (iterative)LR (direct)CNN (direct)012345Forecast time [days]0.00.51.01.52.02.53.03.54.0T850 MAE [K]b) T850012345Forecast time [days]0.00.51.01.52.02.53.03.5T2M MAE [K]a) T2M012345Forecast time [days]0.00.20.40.60.81.0TP MAE [mm]b) TP Table A1. Baseline ACC for 3 and 5 days forecast time at 5.625° resolution. Best machine learningterm·ai baseline and physical model are highlighted. TP is 6 hourly accumulated precipitation.
Baseline Persistence Climatology Weekly climatology ACC (3 days / 5 days) Z500 T850 T2M 0.62 / 0.53 0.69 / 0.65 0.88 / 0.85 0.06/0.06 0.65 0.77 0.85 0.16 Linear regressionterm·stat (direct) 0.76 / 0.68 0.81 / 0.78 0.92 / 0.90 0.15 / 0.13 Linear regressionterm·stat (iterative) 0.76 / 0.67 0.82 / 0.78 CNN (direct) CNN (iterative) IFS T42 IFS T63 0.81 / 0.71 0.85 / 0.79 0.61 / 0.41 0.72 / 0.31 0.90 / 0.78 0.86 / 0.78 0.87 / 0.83 0.97 / 0.91 0.94 / 0.90 0.94 / 0.92 Operational IFS 0.99 / 0.95 0.97 / 0.93 0.98 / 0.96 0.43 / 0.30 Table A2. Baseline MAE for 3 and 5 days forecast time at 5.625° resolution. Best machine learningterm·ai baseline and physical model are highlighted. TP is 6 hourly accumulated precipitation.
Baseline Persistence Climatology Weekly climatology MAE (3 days / 5 days) Z500 [m2 s−2] T850 [K] T2M [K] TP [mm] 572 / 634 2.91 / 3.12 1.81 / 1.95 1.07 / 1.08 708 525 3.84 2.48 3.77 2.00 0.92 0.88 Linear regressionterm·stat (direct) 431 / 489 2.23 / 2.41 1.48 / 1.60 0.96 / 0.97 Linear regressionterm·stat (iterative) 447 / 508 403 / 476 2.20 / 2.42 2.02 / 2.38 CNN (direct) CNN (iterative) IFS T42 IFS T63 Operational IFS 892 / 1263 3.49 / 7.49 295 / 449 188 / 307 97 / 198 1.99 / 2.53 1.83 / 2.12 1.30 / 1.73 1.28 / 1.53 0.93 / 1.35 0.84 / 1.08 0.69 / 0.81 Author contributions. SR, PD, SS and JW conceived the idea. SR prepared the data and baselines and led the writing. All authors contributed to the manuscript.
Competing interests. The authors declare no competing interests.
Acknowledgements. Stephan RasppersName acknowledges funding from the German Research Foundation (DFG). We thank the Copernicus Climate Change Service (C3S) for allowing us to redistribute the data. Peter D. DuebenpersName gratefully acknowledges funding from the Royal Society for his University Research Fellowship and the ESIWACE2 project. The ESIWACE2 project have received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 823988.