Analyzing Weather Patterns

There are numerous types of weather patterns analyses that are closely correlated to socio-economic value. We have decided to focus on just a few that have broad relevance.

Most of our initial analyses have been made on familiar weather parameters such as air temperature, dew point, rainfall, rainfall rate, wind,  lightning and snow. 

Most of our insights are produced at specified granularities in space and time. We think of spatial granularity as the size of a grid (e.g. 25km x 25km) on the surface of earth or a reference to  a specific station (often an airport/municipal weather station). Analyses at different time granularities lend themselves to different applications. Typically, those at months or year granularities are appropriate for medium to long term planning while those at hour or minute granularities are useful for operational decisions.

While all of the pattern analyses require historical data at the appropriate granularity , we call out the type of data on which the analysis is performed. The first two classes are intuitive , Observations (measurements recorded by weather stations, satellites or radar)  and Forecasts (conditions that are predicted for n days in the future). The third class is somewhat related to observations and is called Reanalysis. While this is a familiar term for those in the meteorology/climatology domain, it can be confusing for the uninitiated. They are a "blend of observations with past short-range weather forecasts rerun with modern weather forecasting models". The last category are the Climate Projections datasets. These are datasets that contain the projected climate based on climate models that are run on many emission scenarios for the earth till the end of this century. Unsurprisingly, the IPCC report and products of numerous climate analytics companies are based on data derived from them. It is important to note that projections are not predictions,  in the sense that they are not 'this will be the expected weather in 2028, or that will be the expected weather in Jun 2025'  . They are more like for a given location X, 'between years 2025-2035 , the distribution of max temperatures informs us that summer days are likely to be 0.7 C warmer on average with extremes 3.5C higher than the mean, which we can compare to earlier projections say from 2015-2025 to get a sense of the shift in patterns.' . This does not tell you if there is going to be a particularly warm summer in 2028 or 2032. 

It is also worth noting that each of these classes of data has data available at a defined granularity of space and time and tradeoffs and limitations are associated with each of them. (ex: Hourly Forecasts' accuracy falls significantly after 2 days into the future, reanalysis at 25km will miss extreme rainfall recorded at a station nearby)

At the time of writing, we have spent most of our time processing Observations and Forecasts to produce a subset of these patterns. While there are interactive websites that compute anomalies and produce station-level analytics, we are not aware of alternatives that provide weather-patterns-as-a-service (via an API) . 

Data Collection 

In order to start analyzing weather patterns, historical weather data needs to be collected and processed. In addition, current observations or latest forecasts are required to perform pattern analysis. Of course, none of what we are trying to accomplish is possible if it were not for the vast open datasets that are published by government agencies such as NOAA or ECMWF.  We also benefit from private API providers for specific kinds of weather data.

However, it is still non-trivial to ingest and make sense of all that data. At the very least, datasets need to be updated at least monthly to detect most patterns we have discussed. For any 'daily' use case, the ingestion has to be more frequent, often sub-daily. We use the term latency to characterize the expected delay in receiving data from a source. (We also use it to define delays for the insights we produce, but that is for another post). When consuming data from public sources, it is necessary to understand operational aspects of the datasets. It is not enough to merely understand data specifications such as spatial/time granularity or coverage without accounting for latency. e.g. We noticed that a key observation dataset (NOAA GSOD) we consume had a reasonably complex lineage and pre-processing steps that contributed to a significantly varying latency (2-7 days). This led us to directly consume some of its upstream (ex: SYNOP, to obtain global daily precipitation data  within 24hrs) and then eventually reconcile them with the main data source. We summarize some of the data sources we use and highlight the characteristics that led us to make changes to our architecture and product roadmap.

We believe that this kind of work involving data collection, ingestion and transformation is repeated across many companies to derive similar first level insights and analytics to understand weather patterns. While it is tempting (and sometimes not appropriate) to rely on 'just one or two gridded datasets' - collecting, transforming , processing those is non-trivial and requires specific expertise.  That is why we believe that providing APIs on top of these detected patterns and ensuring an exceptional level of transparency could be the way forward in effectively utilizing climate data. 

Commercial  Weather APIs

There are some excellent options to pay for weather data access in convenient data formats (including access to gridded data from ERA5) for recent and forecast weather. However , if you plan on analyzing large historical datasets, not all providers have easy ways to download large datasets. One of our aims is to provide insights and analytics that are very affordable (or close to free) which becomes challenging if we pay enterprise level prices as opposed to pay-as-you-go APIs where it's possible to dial-up/down usage based on affordability or need.

Reanalysis Data

Even as we increase our use of gridded data (Reanalysis and Climate Projection) in our analyses, we remain committed to obtaining  high quality (and low latency) observational data wherever possible. We have encountered regions where the reanalysis data is quite far from actual observations.  These are probably confined to stations where highly localized events such as heavy rain were averaged across a wider grid. That said, the continuity that reanalysis data offers is appropriate for trend analysis of non-urban locations (where density of stations is low) or stations where historical data is patchy or unreliable (e.g. precipitation data of non-US locations) .

We need more help

We are looking for climatologists/data scientists to join the founding team to help us define and scale how we detect and understand weather trends. Please get in touch if you are curious.