Faster And More Accurate Weather Forecasting With HPC In The Cloud

ADVERTORIAL The increasing need for more accurate weather modelling rears its head on an almost daily basis, as the changing global climate proves to be one of humanity's greatest challenges.

Complex weather modelling has been a mainstay of supercomputing for decades, but as weather extremes test these models a different approach is needed to meet the demands for faster, and more accurate predictions.

Numerical weather prediction (NWP) is used to improve our understanding of how processes work in the atmosphere, land masses and our oceans, both for long-term trends and near forecasts. Better results not only mean countries can better predict climate extremes to protect citizens, but helps them improve agricultural processes in preparation for changing conditions. That means the impacts of climate on our food production could also be lessened.

Major weather services around the world continuously develop and incorporate new meteorological research into complex operational forecasting models – compute-intensive workloads that are shifting to the cloud for scale, cost, and flexibility benefits.

Fully-fitted cloud HPC

As cloud computing matured its offerings, diversified and value-added customization entered the market.

Larger cloud service providers like Amazon Web Services (AWS) have led the way with offerings like its Amazon Elastic Compute Cloud (EC2) service on the AWS cloud. Through EC2, user-organizations rent virtual servers (or virtual server instances) to run their own applications, as opposed to buying and owning their own physical IT infrastructure – often the traditional domain of HPC workloads.

Various kinds of Amazon EC2 instancesoptimized for different configurations of CPU/GPU type, memory, storage and networking resources are available to match a specific IT/workload requirement. The thinking behind this model is that user organizations consume only as much IT resource as they need for a specific application or workload, thus minimizing costly overprovisioning. Additional resources can be accessed as required on-demand.

Amazon EC2 instance types have their own designations, Hpc6a, and are purpose-built for tightly-coupled, compute-intensive HPC workloads like weather modelling. These are powered by two 48-core 3 Gen AMD EPYC "Milan" processors built on 7nm technology (for increased efficiency), and supporting 384GB of memory across the 96 cores (or 4GB memory per core).

Cloud HPC's impact on weather predictions

Frequency of data crunching in weather modelling is key to speeding up the process of getting results that are still accurate, or even more so. Typically, models are run twice a day due to the large volumes of data involved. Crunching more data more frequently gets organizations closer to the goal of improving accuracy and getting near real-time weather intelligence.

EC2 Hpc6a instances are the perfect platform to run these models on. EC2 Hpc6a instances offer up to 65% better price performance over comparable Amazon EC2 x-86 based compute-optimized instances. In practice that means much more can be done, in a shorter timeframe and at less cost.

The key to the Hpc6a instances' performance lies partly in their silicon architecture with AMD's EPYC processors. EPYC's system-on-chip design does not require any additional chipset on the motherboard. Built on an efficient 7nm process, the third-generation EPYC processor family was just the kind of compact, high-performance silicon that AWS was looking for when it developed Hpc6a instances.

The performance spike is also partly due to the Elastic Fabric Adaptor (EFA) inter-node bus, which manages the data-intensive communications between nodes in an HPC architecture. This low-latency capability is especially important for the kinds of tightly-coupled workloads that weather modelling deals with.

The proof of course is how all this adds up in terms of real-world scientific analysis. Weather data firm DTN used Hpc6a instances to double its weather forecasting performance (increasing frequency from two to four models each day). DTN utilized Hpc6a instances to process huge volumes of climate data integrated with satellite imagery, more quickly. The company needed an HPC system that could quickly consume a flood of incoming data, using it to create new models at speed and scale. This was a tough ask. DTN was not just dealing with atmospheric data modelling, which would have been difficult enough. It also models oceanic data, which is intrinsically linked with weather patterns.

The cloud, with its elastic computing and storage capability, was the obvious route as DTN pursued its holy grail of hourly weather models. In 2020, the weather company enlisted Amazon to help build its HPC infrastructure in the public cloud. It spent the next 18 months with the AWS team creating a proof-of-concept using test data collected during Hurricane Laura (2020). In 2022, by using Hpc6a instances, DTN enhanced its results and have now moved its entire global forecasting solution to AWS.

Dealing with data… lots of it!

Weather modelling organizations can churn through petabytes of meteorological data every day. But in DTN's case, the company was further able to utilize AWS tools. It managed the processing of data through a complex array of EPYC-based nodes using AWS ParallelCluster. This is an open-source cluster management tool targeting HPC applications. AWS ParallelCluster 3.5 added a GUI, enabling it to scale up instance numbers, and also integrates with a variety of operating systems and schedulers, making it easier for HPC customers to migrate or burst into the cloud from existing on-premises systems.

Faster forecasts – the new norm

With AWS and AMD proving that cloud HPC can slash the forecast times of many incumbent systems, the future of NWP looks bright. Another great example saw an NWP forecast generated via a cloud HPC cluster in about 53 minutes – just slightly more than half the time incumbent systems take to complete the same forecast. The customer was mapping firm Maxar, which had an ambitious target to cut forecast times by more than half. With this achieved using Hpc6a instances, it was mission accomplished, but further gains could be made – this time with AWS' Elastic Fabric Adapter (EFA). By using EFA—a network interface for Amazon EC2 instances—Maxar shortened forecast times from 53 to 42 minutes. Further testing and optimization with AWS meant Maxar could complete a forecast in less than 30 minutes eliminating more of the barriers to commercial solutions.

Forecasting the future

Weather forecasting using HPC in the cloud with Amazon EC2 instances, is meeting the demands of a fast-changing climate, where scientists must provide ever more accurate predictions of dramatic weather events to protect lives and livelihoods.

Sponsored by AWS & AMD.