Close Menu
    Trending
    • Agentic AI in Finance: Opportunities and Challenges for Indonesia
    • Dispatch: Partying at one of Africa’s largest AI gatherings
    • Topp 10 AI-filmer genom tiderna
    • OpenAIs nya webbläsare ChatGPT Atlas
    • Creating AI that matters | MIT News
    • Scaling Recommender Transformers to a Billion Parameters
    • Hidden Gems in NumPy: 7 Functions Every Data Scientist Should Know
    • Is RAG Dead? The Rise of Context Engineering and Semantic Layers for Agentic AI
    ProfitlyAI
    • Home
    • Latest News
    • AI Technology
    • Latest AI Innovations
    • AI Tools & Technologies
    • Artificial Intelligence
    ProfitlyAI
    Home » Decoding Nonlinear Signals In Large Observational Datasets
    Artificial Intelligence

    Decoding Nonlinear Signals In Large Observational Datasets

    ProfitlyAIBy ProfitlyAISeptember 24, 2025No Comments28 Mins Read
    Share Facebook Twitter Pinterest LinkedIn Tumblr Reddit Telegram Email
    Share
    Facebook Twitter LinkedIn Pinterest Email



    In latest a long time, world local weather monitoring has made important strides, resulting in the creation of latest, in depth observational datasets (Karpatne et al., 2019). These datasets are important for bettering numerical climate predictions and refining distant sensing retrievals by offering detailed insights into complicated bodily processes (Alizadeh, 2022). Nevertheless, as the amount and complexity of the info grows, figuring out patterns throughout the observations turns into more and more difficult (Zhou et al., 2021). Extracting key options from these datasets may result in necessary developments in our understanding of phenomena like convection and precipitation, additional enhancing our data of the altering world local weather.

    On this submit, we’ll discover a few of these complicated information patterns by the lens of precipitation, which has been highlighted as a critically necessary space of examine beneath warming world temperatures (IPCC, 2023). Fairly than counting on randomly generated or simulated information for this challenge, we’ll work with real-world observations from throughout the globe, which are publicly accessible for you, my reader, to explore and experiment with as well. Let this submit function a analysis information, beginning with the significance of excellent high quality information, and concluding with insights on linear and nonlinear interpretations of stated information. 

    Should you’d wish to observe together with some code, take a look at our interactive Google Colab notebook.

    This evaluation unfolds in three elements, every of which is a separate, printed analysis article:

    1. Curating a strong, multidimensional dataset
    2. Analyzing linear embeddings
    3. Exploring nonlinear options

    1. The Microphysical Dataset

    https://doi.org/10.1029/2024EA003538

    After we discuss understanding the options of precipitation, what are we actually asking? How complicated can one thing as widespread as rain or snow be? It’s straightforward to look outdoors on a stormy day and say, “It’s raining” or “It’s snowing”. However what’s truly occurring in these moments? Can we be extra exact? For instance, how intense is the rainfall? Are the raindrops giant or small? If it’s snowing, what do the snowflakes appear to be ? Are they fluffy, dendritic crystals, or are they composed of a number of, fused particles in giant combination clumps (e.g., Fig. 1)? If the temperature hovers close to zero levels Celsius (C), do the snowflakes change into dense and slushy? How briskly are they falling? These variations may have a big effect on what occurs when the particles attain the bottom, and categorizing these processes into distinct teams is non-trivial (Pettersen et al., 2021).

    Determine 1: Macro picture of an combination snow particle composed of a number of fused snowflakes —  Photograph by Writer

    Understanding these processes is essential for higher monitoring and mitigating the impacts of flooding, runoff, freezing rain and excessive precipitation, all of that are doubtlessly harmful occasions with billions of {dollars} of related world damages annually (Sturm et al., 2017). However with hundreds of particles falling over just some sq. meters in a matter of minutes, how can we quantify this complicated course of? It’s not nearly counting the particles, we additionally must seize key traits like dimension and form. As a substitute of making an attempt this manually (an unimaginable activity, go attempt for your self), we sometimes depend on distant sensing devices to do the heavy lifting. One such device is the NASA Precipitation Imaging Package deal (PIP), a video disdrometer that gives detailed observations of falling rain drop and snow particles (Pettersen et al., 2020), as proven under in Fig. 2.

    Determine 2: Photograph of the PIP instrument setup (digicam and bulb) outdoors Marquette, MI — Photograph courtesy of Claire Pettersen

    This comparatively cheap instrument consists of a 150-watt halogen bulb and a high-speed video digicam (capturing at 380 frames per second) positioned two meters aside (King et al., 2024). As particles fall between the bulb and the digicam, they block the sunshine, creating silhouettes that may be analyzed for variations in dimension and form. By monitoring the identical particle throughout a number of frames, the PIP software program may also decide its fall pace (Fig. 3). With further assumptions about particle movement within the air, the PIP information permit us to additionally derive minute-scale particle dimension distributions (PSDs), fall speeds, and efficient particle density distributions (Newman et al., 2009). These microphysical measurements, when mixed with close by meteorological observations of floor variables like temperature, relative humidity, stress, and wind pace, provide a complete snapshot of the surroundings on the time of commentary.

    Determine 3: PIP video with falling snowflakes, recorded Christmas Eve, 2021 outdoors Storrs, Conneticuit (1/twentieth full pace). Particles are falling sideways attributable to wind interference — Video by Larry Bliven [Source: https://www.youtube.com/@larrybliven832]

    Over a span of 10 years, we collected greater than 1 million minutes of particle microphysical observations, alongside collocated floor meteorological variables, throughout 10 totally different websites (Fig. 4). Gathering information from a number of regional climates over such an extended interval was essential to constructing a strong database of precipitation occasions. To make sure consistency, all microphysical observations have been recorded utilizing the identical kind of instrument with equivalent calibration settings and software program variations. We then performed an in depth high quality assurance (QA) course of to get rid of faulty information, appropriate timing drifts, and take away any unphysical outliers. This curated info was then standardized, packaged into Network Common Data Form (NetCDF) recordsdata, and made publicly available through the University of Michigan’s DeepBlue data repository.

    You’re welcome to obtain and discover the dataset your self! For extra particulars on the websites included, the QA course of, and the microphysical variations noticed between places, please discuss with our associated data paper published in the journal of Earth and Space Science.

    Study site locations and data coverage periods
    Determine 4: a) Areas of measurement websites; and b) Gantt chart of observational protection for every website — Picture by Writer

    To explain the PSD, we calculate a pair of parameters (n0, and λ) representing the intercept and slope of an inverse exponential match (Eq. 1). This match was chosen because it has been extensively utilized in earlier literature to precisely describe snowfall PSDs (Cooper et al., 2017; Wooden and L’Ecuyer, 2021). Nevertheless, different suits (e.g., a gamma distribution) is also thought-about in future work to raised seize giant combination particles (Duffy et al., 2022).

    n0-λ joint 2D histograms are proven under in Fig. 5 for every website, demonstrating the big variety of precipitation PSDs occurring throughout totally different regional climates. Word how some websites show bimodal distributions (OLY) evaluate to very slender distributions at others (NSA). We’ve got additionally put collectively a Python API for interacting with and visualizing this information referred to as pipdb. Please see our documentation on readthedocs for extra info on the best way to set up and use this bundle in your personal challenge.

    Determine 5: 2D joint histograms of inverse exponential intercept (n0) and slope (λ) PSD parameters for every website — Picture by Writer

    In abstract, we’ve compiled a high-quality, multidimensional dataset of precipitation microphysical observations, capturing particulars such because the particle dimension distributions, fall speeds, and efficient densities. These measurements are complemented by a spread of close by floor meteorological variables, offering essential context in regards to the particular forms of precipitation occurring throughout every minute (e.g., was it heat out, or chilly?). A full checklist of the variables we’ve collected for this challenge is proven in Desk 1 under.

    Desk 1: Abstract of all microphysical and floor meteorologic variables collected and made accessible within the DeepBlue dataset — Picture by Writer

    Now, what can we do with this information? 


    2. Inspecting Linear Embeddings with PCA

    https://doi.org/10.1175/JAS-D-24-0076.1

    With our information collected, it’s time to place it to make use of. We start by exploring linear embeddings by Principal Part Evaluation (PCA), following the methodology of Dolan et al. (2019). Their work targeted on uncovering the latent options in rainfall drop dimension distributions (DSDs), figuring out six key modes of variability linked to the bodily processes that govern drop formation throughout quite a lot of places. Constructing on this, we purpose to increase the evaluation to snowfall occasions utilizing our customized dataset from Half 1. I received’t delve into the mechanics of PCA right here, as there are already many excellent resources on TDS that cover its implementation in detail.

    Earlier than making use of PCA, we section the complete dataset into discrete 5-minute intervals. This segmentation permits us to calculate the PSD parameters with a sufficiently giant pattern dimension. We then filter these intervals, choosing solely these with efficient density values under 0.4 g/cm³ (i.e., values sometimes related to snowfall and characterised by much less dense particles). This filtering leads to a dataset of 210,830 five-minute durations prepared for evaluation. For the variables used to suit the PCA, we select a subset from Desk 1 associated to snowfall, derived from the PIP. These variables embody n0, λ, Fs, Rho, Nt, and Sr (see Desk 1 for particulars). We targeted on a smaller subset of observations from the disdrometer alone right here, as a result of future websites may not have collocated floor variables and we have been considering what may very well be extracted from simply this six-dimensional dataset.

    Earlier than diving into the evaluation, it’s necessary to first examine the info to make sure every thing seems as anticipated. Keep in mind the outdated GIGO addage, rubbish in, rubbish out. We need to mitigate the affect of dangerous information if we will. By analyzing the worth distributions of every variable, we confirmed they fall throughout the anticipated ranges. Moreover, we reviewed the covariance matrix of the enter variables to realize some preliminary insights into their joint conduct (Fig. 6). As an example, variables like n0 and Nt, each tightly coupled to the variety of particles current, present excessive correlation as anticipated, whereas variables like efficient density (Rho) and Nt show much less of a relationship. After scaling and normalizing the inputs, we proceed by feeding them into scikit-learn’s PCA implementation.

    Determine 6: Covariance matrix heatmap of PCA enter variables — Picture by Writer

    Making use of PCA to the inputs leads to three Empirical Orthogonal Capabilities (EOFs) that collectively account for 95% of the variability within the dataset (Fig. 7). The primary EOF is essentially the most important, capturing roughly 55% of the dataset’s variance, as evidenced by its broad distribution in Fig. 6.a. When analyzing the usual anomalies of the EOF values for every enter variable, EOF1 exhibits a powerful unfavourable relationship with all inputs. The second EOF accounts for about 20% of the variance, with a barely narrower distribution (Fig. 6.b) and is most strongly related to the Fallspeed and Rho (density) inputs. Lastly, EOF3, which explains round 15% of the variance, is primarily associated to λ and snowfall fee variables (Fig. 6.c).

    Determine 7: 2D joint histograms displaying normalized counts of a) EOF1 v. EOF2; b) EOF2 v. EOF3; c) EOF3 v. EOF1; and d) the EOF worth for every normalized enter function (observe that the signal of every anomaly is unfair) — Picture by Writer 

    On their very own, these EOFs are difficult to interpret in bodily phrases. What underlying options are they capturing? Are these embeddings bodily significant? One option to simplify the interpretation is by specializing in essentially the most excessive values in every distribution, as these are most strongly related to every EOF. Whereas this guide clustering method leaves a lot of the distribution close to the origin ambiguous, it permits us to separate the info into distinct teams that may be analyzed extra intently. By making use of a σ > 2 threshold (represented by the skinny white dashed traces in Fig. 6.a-c), we will divide this 3D distribution of factors into six distinct teams of equal sampling quantity. Since visualizing this separation in 2D is especially difficult, we’ve offered an interactive information viewer (Fig. 8), created with Plotly, to make this distinction clearer. Be happy to click on on the determine under to explore the data yourself.

    Determine 8: Interactive 3D plotly scatter of PCA EOF embeddings. Factors are coloured by their respective guide clustering teams (with ambiguous in grey). Click on to work together with the info your self — Picture by Writer

    With essentially the most excessive EOF clusters chosen, we will now plot these in bodily variable areas to start deciphering them. That is demonstrated in Fig. 9 throughout totally different variable areas: n0-λ (panel a), Fs-Rho (panel b), λ-Dm (panel c), and Sr-Dm (panel d). Beginning with the pink and blue clusters in Fig. 9.a (representing the optimistic and unfavourable EOF1 values), we see a transparent separation in n0-λ area. The pink cluster, characterised by a excessive PSD intercept and slope, signifies a high-intensity grouping, suggestive of many small particles, whereas the blue cluster exhibits the other conduct. That is indicative of a possible depth embedding.

    In panel b, there’s a definite separation between the purple and light-weight blue clusters (similar to the optimistic and unfavourable EOF2 values). The purple cluster, related to excessive fall pace and density, contrasts with the sunshine blue cluster, which exhibits the other traits. This sample doubtless represents a particle temperature/wetness embedding, describing the “stickiness” of the snow because it falls. Hotter, denser particles (comparable to partially melted or frozen particles) are inclined to fall quicker, very like how a slushy pellet falls quicker than a dry snowflake.

    Lastly, in panels c and d, the yellow and magenta clusters are separated based mostly on PSD slope and mass-weighted imply diameter. Whereas much less clear, this means a possible relationship with particle dimension and the underlying snowfall regime, comparable to variations between complicated shallow programs and deep programs.

    Determine 9: Bodily variable area comparisons for every of the PC teams, together with: a) n0-λ, b) Fs-Rho, c) λ-Dm and d) Sr-Dm, with the coloured contours highlighting every PC group utilizing a smoothed kernel density approximation — Picture by Writer

    One other option to strengthen our confidence in these attributions is by evaluating the teams to impartial observations. We will do that by cross-referencing the PCA-based snowfall classifications from the PIP with close by floor radar observations (i.e., a Micro Rain Radar) and reanalysis (i.e., ERA5) estimates to judge bodily consistency. That is one motive we suggest not at all times utilizing all accessible information within the dimensionality discount, because it limits the flexibility to later assess the robustness of the embeddings. To validate our method, we examined a collection of case research at Marquette (MQT), Michigan, to see how properly these classifications align. As an example, in Fig. 10, we observe a transition from a high-intensity snowstorm (pink) to partially melted mixed-phase snow crystals (sleet) as temperatures briefly rise above zero levels C (panel h), after which again to high-intensity snow as temperatures drop under zero later within the day. This additionally aligns with the adjustments we see in reflectivity (panel a) and we will see this transition within the n0-λ plot in panel i.

    Determine 10: PC teams 2 and 4 with ancillary observations at MQT on 2018-12-02. PC teams for a single day are highlighted throughout the highest determine panels in 5-minute time steps towards collocated a) MRR reflectivity (Ze) observations; b) ERA5 atmospheric temperature (T) estimates (zero diploma isotherm highlighted within the black dashed line); c) MRR Doppler velocity (DV) observations; d) ERA5 atmospheric relative humidity (RH) estimates; e) MRR spectral width (SW) observations; f) ERA5 atmospheric vertical velocity (ω) estimates; g) EOF1, EOF2 and EOF3 values from the PCA; h) Floor meteorologic observations of 2-meter air T in black, RH in grey and stress (P) in brown; and that i) every PIP commentary plotted in n0-λ area in grey for ambiguous factors, pink for PC group 2, and purple for PC group 4 (black circles signify all different factors throughout all websites). Dashed black traces point out the placement of the dendritic progress zone — Picture by Writer

    Constructing on our PCA evaluation and the consistency noticed with collocated observations, we additionally created Fig. 11, which summarizes how the first linear embeddings recognized by PCA are distributed throughout totally different bodily variable areas. These classifications provide vital microphysical insights that may improve a priori datasets, in the end bettering the accuracy of state-of-the-art fashions and snowfall retrievals. 

    Determine 11: PCA-derived snowfall attribute conceptual mannequin in a) n0-λ; and b) Fs-Rho areas. Black factors signify all PIP observations from all websites, with the coloured contours depicting every of the PC teams produced from a smoothed kernel density approximation. Every of the inferred bodily attributes are labeled on every contour in white — Picture by Writer

    Nevertheless, since PCA is proscribed to linear embeddings, this raises an necessary query: are there nonlinear patterns inside this dataset that we’ve got but to discover? Moreover, what new insights would possibly emerge if we prolong this evaluation past snow to incorporate different forms of precipitation?

    Let’s sort out these questions within the subsequent part!


    3. Nonlinear dimensionality discount utilizing UMAP

    https://doi.org/10.1126/sciadv.adu0162

    So as to look at extra complicated, nonlinear embeddings, we have to take into account a unique kind of unsupervised studying that loosens the linearity assumptions of strategies like PCA. This brings us to the idea of manifold studying. The concept behind manifold studying is that high-dimensional information typically lie on a lower-dimensional, curved manifold throughout the unique information area (McInnes et al., 2020). By mapping this manifold, we will uncover the underlying construction and relationships that linear strategies would possibly miss. Strategies like t-SNE, UMAP, VAEs, or Isomap can reveal these intricate patterns, offering a extra nuanced understanding of the dataset’s latent options. Making use of manifold studying to our dataset may uncover nonlinear embeddings that additional distinguish precipitation sorts, doubtlessly providing even deeper insights into the microphysical processes at play. As slightly trace as to what’s to come back, see Fig. 12. As talked about earlier than, I received’t go into the implementation particulars of such strategies, as this has been lined many instances here on TDS.

    Determine 12: 3D UMAP embedding of our precipitation dataset — Picture by Writer

    Moreover, we wish to use our whole dataset of each disdrometer observations and collocated floor metoeologic variables this time round to see if the extra dimensions present helpful context for higher differentiating between extremely complicated bodily processes. For instance, can we detect various kinds of mixed-phase precipitation if we knew extra in regards to the temperature and humidity on the time of commentary? So, in contrast to the earlier part the place we restricted the inputs to simply PIP information and simply snowfall, we now embody all 12 dimensions for the complete dataset. This additionally considerably reduces our complete pattern all the way down to 128,233 5-minute durations at 7 places, since not all websites have working floor meteorologic stations to tug information from. As is at all times the case with these kind of issues, as we add extra dimensions, we run up towards the dreaded curse of dimentionality.

    Because the dimensionality of the function area will increase, the variety of configurations can develop exponentially, and thus the variety of configurations lined by an commentary decreases — Richard Bellman

    This tradeoff within the variety of inputs and have sparsity is a problem we could have to bear in mind transferring ahead. Fortunately for us, we solely have 12 dimensions which can seem to be lots, however is actually fairly small in comparison with many different tasks within the Pure Sciences with doubtlessly hundreds of dimensions (Auton et al., 2015).

    As talked about earlier, we explored quite a lot of nonlinear fashions for this part of the challenge (see Desk 2). In any Machine Studying (ML) challenge we undertake, we favor to begin with easier, extra interpretable strategies and steadily progress to extra subtle strategies, as much less complicated approaches are sometimes extra environment friendly and simply understood. 

    With this technique in thoughts, we started by constructing on the outcomes from Half 2, utilizing PCA as soon as once more as a baseline for this bigger dataset of rain and snow particles. We then in contrast PCA to nonlinear strategies comparable to Isomap, VAEs, t-SNE, and UMAP. After conducting a collection of sensitivity analyses, we discovered that UMAP outperformed the others in producing clear embeddings in a extra computationally environment friendly method, making it the main target of our dialogue right here. Moreover, with UMAP’s improved world separation of information throughout the manifold, we will transfer past guide clustering, using a extra goal methodology like Hierarchical Density-Based mostly Spatial Clustering of Functions with Noise (HDBSCAN) to group comparable instances collectively (McInnes et al., 2017).

    Desk 2: Overview of strategies examined within the nonlinear comparability portion of the challenge — Picture by Writer

    Making use of UMAP to this 12-dimensional dataset resulted within the identification of three major latent embeddings (LEs). We experimented with varied hyperparameters, together with the variety of embeddings, and located that, just like PCA, the primary two embeddings have been essentially the most important. The third embedding additionally displayed some separation between sure teams, however past this third degree, further embeddings offered little separation and have been due to this fact excluded from the evaluation (though these is likely to be attention-grabbing to have a look at extra in future work). The primary two LEs, together with a case examine instance from Marquette, Michigan, illustrating discrete information factors over a 24-hour interval, are proven under in Fig. 13.

    Determine 13: Overview of precipitation course of clusters derived from UMAP+HDBSCAN. a) UMAP coordinates for all observations throughout all websites (grey factors), overlaid with coloured HDBSCAN-derived density clusters (centroids proven as white circles), and annotated with attributed bodily precipitation processes and key abstract traits. b) Instance day displaying the particle behavior evolution all through the day at MQT, MI. c) Ancillary observations for the occasion in b) — Picture by Writer

    Instantly, we discover a number of key variations from the earlier snowfall-focused examine utilizing PCA. With the addition of rain and mixed-phase information, the primary and second empirical orthogonal features (EOF1 and EOF2) have now swapped locations. The first embedding now encodes details about particle part slightly than depth. Depth shifts to the second latent embedding (LE2), remaining important however now secondary. The third LE nonetheless seems to narrate to particle dimension and form, significantly throughout the snowfall portion of the manifold. 

    Making use of HDBSCAN to the manifold teams generated by UMAP resulted in 9 distinct clusters, plus one ambiguous cluster (Fig. 13.a). The separation between clusters is far clearer in comparison with PCA, and these teams appear to signify distinct bodily precipitation processes, starting from snowfall to mixed-phase to rainfall at varied depth ranges. Curiously, the ambiguous factors and the connections between nodes within the graph kind distinct pathways of particle behavior evolution. This discovering is especially intriguing because it outlines clear particle evolutionary pathways, displaying how a raindrop can remodel right into a frozen snow crystal beneath the best atmospheric circumstances.

    An actual-world instance of this phenomenon is proven in Fig. 13.b, noticed in Marquette on February 15, 2023. Every coloured ring represents a person (5-minute) information level all through the day, with an arrow indicating the route of time. In Fig. 13.c, we overlay ancillary radar observations with floor temperatures. Up till round 12:00 UTC, a transparent brightband in reflectivity might be seen at roughly 1 km, indicative of a melting layer the place temperatures are heat sufficient for snow to soften into rain. This era was accurately labeled as rainfall utilizing our UMAP+HDBSCAN (UH) clustering methodology. Then, round 17:00 UTC, temperatures quickly dropped properly under freezing, resulting in the classification of particles as mixed-phase and ultimately as snowfall. A lot of these checks are critically necessary for ensuring what your manifold form suggests is sensible bodily.

    Should you’d wish to discover this manifold your self, analyzing totally different websites and seeing how varied variables map to the embedding, take a look at our interactive data analysis tool, or click on Fig. 14 under.

    Determine 14: Interactive 3D plotly scatter of UMAP LE embeddings. Factors are coloured by their respective HDBSCAN cluster teams (with ambiguous in black). Click on to work together with the info your self — Picture by Writer

    Whenever you discover the device talked about above, you’ll discover that mapping varied enter options to the manifold embedding leads to clean gradients. These gradients point out that the overall world construction of the info is probably going being captured in a significant means, providing worthwhile insights into what the embeddings are encoding.

    Evaluating the separation of factors utilizing UMAP to that of PCA (the place PCA is utilized to the very same dataset as UMAP) reveals considerably higher separation with UMAP, particularly regarding precipitation part. Whereas PCA can broadly distinguish between “liquid” and “stable” particles, it struggles with the extra complicated mixed-phase particles. This limitation is clear within the distributions proven in Fig. 15.d-e. PCA typically suffers from variance overcrowding close to the origin, resulting in a tradeoff between the variety of clusters we will establish and the dimensions of the ambiguous supercluster. Though HDBSCAN might be utilized to PCA in the identical method as UMAP, it solely generates two clusters (rain and snow) which isn’t significantly helpful by itself, and might be achieved with a easy linear threshold. In distinction, UMAP gives significantly better separation, leading to 37% fewer ambiguous factors and a +0.14 greater silhouette score for the clusters in comparison with PCA (0.51).

    Determine 15: Comparability of PCA to UMAP educated utilizing the identical dataset, displaying the PCA and UMAP teams in a) and b), respectively; c) exhibits the full variety of ambiguous factors between every approach; d) and e) present 1D KDEs of LE1/EOF1 and LE2/EOF2 for every approach — Picture by Writer

    As we did beforehand with PCA, we will conduct a collection of case examine comparisons when utilizing UMAP to bolster our bodily cluster attributions. By evaluating these with collocated MRR observations, we will assess whether or not the circumstances reported within the astmosphere above the PIP align with the attributions produced by the UH clusters, and the way these evaluate to the clusters from PCA. In Fig. 16 under, we look at a number of of those instances at Marquette.

    Determine 16: Case examine comparability of UMAP and PCA classifications towards collocated floor radar information for 3 days at MQT — Picture by Writer

    Within the first column (a), we current an instance of a chronic mixed-phase occasion, emphasizing LE1, which we all know occurred at MQT from recorded climate reviews. Alongside the highest panel, each PCA and UMAP establish the interval up till 19:00 UTC as rain. Nevertheless, after this era, the PCA groupings change into sparse and largely ambiguous, whereas UMAP efficiently maps the post-19:00 UTC interval as mixed-phase, distinguishing between moist sleet (inexperienced) and colder, slushy pellets (purple).

    In panel (b), we spotlight a case specializing in depth adjustments (LE2), the place circumstances shift from high-intensity mixed-phase to low-intensity mixed-phase, after which again to high-intensity snowfall as temperatures cool. Once more, UMAP gives a extra detailed and constant classification in comparison with the sparser outcomes from PCA.

    Lastly, in panel (c), we discover an LE3 case involving a shallow system till 15:00 UTC, adopted by a deep convective system transferring over the positioning, resulting in a rise within the dimension, form complexity, and depth of the snow particles. Right here too, UMAP demonstrates a extra complete mapping of the occasion. Word that these are just a few handpicked case research nonetheless, and we suggest testing our full paper for multi-year comparisons.

    General, we discovered that the nonlinear 3D manifold generated utilizing UMAP offered a clean and correct approximation of precipitation part, depth, and particle dimension/form (Fig. 17). When mixed with hierarchical density-based clustering, the ensuing teams have been distinct and bodily according to impartial observations. Whereas PCA was capable of seize the overall embedding construction (with EOFs 1-3 largely analogous to LEs 1-3), it struggled to signify the worldwide construction of the info, as many of those processes are inherently nonlinear.

    Determine 17: 3D Visualization of the ultimate UMAP manifold for our precipitation dataset — Picture by Writer

    So what does this all imply?


    Conclusions

    You’ve made it to the top! 

    I notice this has been a prolonged submit, so I’ll maintain this part transient. In abstract, we’ve developed a high-quality dataset of precipitation observations from a number of websites over a number of years and used this information to use each linear and nonlinear dimensionality discount strategies, aiming to be taught extra in regards to the construction of the info itself! Throughout all strategies, embeddings associated to particle part, precipitation depth, and particle dimension/form have been essentially the most dominant. Nevertheless, solely the nonlinear strategies have been capable of seize the complicated world construction of the info, revealing distinct precipitation teams that aligned properly with impartial observations.

    We imagine these teams (and particle transitionary pathways) can be utilized to enhance present satellite tv for pc precipitation retrievals in addition to numerical mannequin microphysical parameterizations. With this in thoughts, we’ve got constructed an operational parameter matrix (the lookup area is illustrated in Fig. 18) which produces a clean conditional chance vector for every group based mostly on temperature (T) and particle counts (Nt). Please see the related manuscript for entry/API particulars to this desk.

    Determine 18: UMAP+HDBSCAN lookup desk histogram mapped in 2D — Picture by Writer

    Nonlinear dimensionality discount strategies like UMAP are nonetheless comparatively new and have but to be extensively utilized to the massive datasets rising within the Geosciences. It must be famous that these techniques are imperfect, and there are tradeoffs based on your problem context, so maintain that in thoughts. Nevertheless, our findings right here, constructing first on PCA, counsel that these strategies might be extremely efficient, emphasizing the worth of rigorously curated and complete observational databases, which we hope to see extra of within the coming years. 

    Thanks once more for studying, and tell us within the feedback how you might be occupied with studying extra out of your giant observational datasets!


    Knowledge and Code

    PIP and floor meteorologic observations used as enter to the PCA and UMAP are publicly accessible for obtain on the College of Michigan’s DeepBlue information repository (https://doi.org/10.7302/37yx-9q53). This dataset is offered as a collection of folders containing NetCDF recordsdata for every website and yr, with standardized CF metadata naming conventions. For extra detailed info, please see our information paper (https://doi.org/10.1029/2024EA003538). ERA5 information might be downloaded from the Copernicus Climate Data Store. 

    PIP information preprocessing code is out there on our public GitHub repository (https://github.com/frasertheking/pip_processing), and we’ve got offered a customized API for interacting with the particle microphysics information in Python referred to as pipdb (https://github.com/frasertheking/pipdb). The snowfall PCA challenge code is out there on Github (https://github.com/frasertheking/snowfall_pca). Moreover, the code used to suit the DR strategies, cluster instances, analyze inputs and generate figures can also be accessible for obtain on a separate, public GitHub repository (https://github.com/frasertheking/umap).


    References

    Alizadeh, O. (2022). Advances and challenges in local weather modeling. Climatic Change, 170(1), 18. https://doi.org/10.1007/s10584-021-03298-4

    Auton, A., Abecasis, G. R., Altshuler, D. M., Durbin, R. M., Abecasis, G. R., Bentley, D. R., Chakravarti, A., Clark, A. G., Donnelly, P., Eichler, E. E., Flicek, P., Gabriel, S. B., Gibbs, R. A., Inexperienced, E. D., Hurles, M. E., Knoppers, B. M., Korbel, J. O., Lander, E. S., Lee, C., … Nationwide Eye Institute, N. (2015). A worldwide reference for human genetic variation. Nature, 526(7571), 68-74. https://doi.org/10.1038/nature15393

    Cooper, S. J., Wooden, N. B., & L’Ecuyer, T. S. (2017). A variational approach to estimate snowfall fee from coincident radar, snowflake, and fall-speed observations. Atmospheric Measurement Strategies, 10(7), 2557-2571. https://doi.org/10.5194/amt-10-2557-2017

    Dolan, B., Fuchs, B., Rutledge, S. A., Barnes, E. A., & Thompson, E. J. (2018). Major Modes of International Drop Dimension Distributions. Journal of the Atmospheric Sciences, 75(5), 1453-1476. https://doi.org/10.1175/JAS-D-17-0242.1

    Duffy, G., & Posselt, D. J. (2022). A Gamma Parameterization for Precipitating Particle Dimension Distributions Containing Snowflake Aggregates Drawn from 5 Subject Experiments. Journal of Utilized Meteorology and Climatology, 61(8), 1077-1085. https://doi.org/10.1175/JAMC-D-21-0131.1

    IPCC, 2023: Local weather Change 2023: Synthesis Report. Contribution of Working Teams I, II and III to the Sixth Evaluation Report of the Intergovernmental Panel on Local weather Change [Core Writing Team, H. Lee and J. Romero (eds.)]. IPCC, Geneva, Switzerland, pp. 35-115, doi: 10.59327/IPCC/AR6-9789291691647.

    Karpatne, A., Ebert-Uphoff, I., Ravela, S., Babaie, H. A., & Kumar, V. (2019). Machine Studying for the Geosciences: Challenges and Alternatives. IEEE Transactions on Data and Knowledge Engineering, 31(8), 1544-1554. https://doi.org/10.1109/TKDE.2018.2861006

    King, F., Pettersen, C., Bliven, L. F., Cerrai, D., Chibisov, A., Cooper, S. J., L’Ecuyer, T., Kulie, M. S., Leskinen, M., Mateling, M., McMurdie, L., Moisseev, D., Nesbitt, S. W., Petersen, W. A., Rodriguez, P., Schirtzinger, C., Stuefer, M., von Lerber, A., Wingo, M. T., … Wooden, N. (2024). A Complete Northern Hemisphere Particle Microphysics Knowledge Set From the Precipitation Imaging Package deal. Earth and Area Science, 11(5), e2024EA003538. https://doi.org/10.1029/2024EA003538

    McInnes, L., Healy, J., & Astels, S. (2017). hdbscan: Hierarchical density based mostly clustering. The Journal of Open Supply Software program, 2(11), 205. https://doi.org/10.21105/joss.00205

    McInnes, L., Healy, J., & Melville, J. (2020). UMAP: Uniform Manifold Approximation and Projection for Dimension Discount (arXiv:1802.03426). arXiv. https://doi.org/10.48550/arXiv.1802.03426

    Newman, A. J., Kucera, P. A., & Bliven, L. F. (2009). Presenting the Snowflake Video Imager (SVI). Journal of Atmospheric and Oceanic Know-how, 26(2), 167-179. https://doi.org/10.1175/2008JTECHA1148.1

    Pettersen, C., Bliven, L. F., von Lerber, A., Wooden, N. B., Kulie, M. S., Mateling, M. E., Moisseev, D. N., Munchak, S. J., Petersen, W. A., & Wolff, D. B. (2020). The Precipitation Imaging Package deal: Evaluation of Microphysical and Bulk Traits of Snow. Ambiance, 11(8), Article 8. https://doi.org/10.3390/atmos11080785

    Pettersen, C., Bliven, L. F., Kulie, M. S., Wooden, N. B., Shates, J. A., Anderson, J., Mateling, M. E., Petersen, W. A., von Lerber, A., & Wolff, D. B. (2021). The Precipitation Imaging Package deal: Part Partitioning Capabilities. Distant Sensing, 13(11), Article 11. https://doi.org/10.3390/rs13112183

    Sturm, M., Goldstein, M. A., & Parr, C. (2017). Water and life from snow: A trillion greenback science query. Water Sources Analysis, 53(5), 3534-3544. https://doi.org/10.1002/2017WR020840

    Wooden, N. B., & L’Ecuyer, T. S. (2021). What millimeter-wavelength radar reflectivity reveals about snowfall: An information-centric evaluation. Atmospheric Measurement Strategies, 14(2), 869-888. https://doi.org/10.5194/amt-14-869-2021

    Zhou, C., Wang, H., Wang, C., Hou, Z., Zheng, Z., Shen, S., Cheng, Q., Feng, Z., Wang, X., Lv, H., Fan, J., Hu, X., Hou, M., & Zhu, Y. (2021). Geoscience data graph within the massive information period. Science China Earth Sciences, 64(7), 1105-1114. https://doi.org/10.1007/s11430-020-9750-4



    Source link

    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
    Previous ArticleRAG Explained: Reranking for Better Answers
    Next Article New AI system could accelerate clinical research | MIT News
    ProfitlyAI
    • Website

    Related Posts

    Artificial Intelligence

    Agentic AI in Finance: Opportunities and Challenges for Indonesia

    October 22, 2025
    Artificial Intelligence

    Creating AI that matters | MIT News

    October 21, 2025
    Artificial Intelligence

    Scaling Recommender Transformers to a Billion Parameters

    October 21, 2025
    Add A Comment
    Leave A Reply Cancel Reply

    Top Posts

    AI/ML for Smarter Enterprise Document Workflows

    September 8, 2025

    AGI by 2035? Google DeepMind CEO Warns “Society’s Not Ready”

    April 25, 2025

    Nya Mercedes GLC EQ 2026 utrustad med båda ChatGPT och Gemini

    September 9, 2025

    Apple pulls AI-generated news from its devices after backlash

    April 4, 2025

    ACP: The Internet Protocol for AI Agents

    May 9, 2025
    Categories
    • AI Technology
    • AI Tools & Technologies
    • Artificial Intelligence
    • Latest AI Innovations
    • Latest News
    Most Popular

    Help Your Model Learn the True Signal

    August 20, 2025

    A Chinese firm has just launched a constantly changing set of AI benchmarks

    June 23, 2025

    Shaip Launches Generative AI Platform for Experimentation, Evaluation, & Monitoring of AI Applications

    April 7, 2025
    Our Picks

    Agentic AI in Finance: Opportunities and Challenges for Indonesia

    October 22, 2025

    Dispatch: Partying at one of Africa’s largest AI gatherings

    October 22, 2025

    Topp 10 AI-filmer genom tiderna

    October 22, 2025
    Categories
    • AI Technology
    • AI Tools & Technologies
    • Artificial Intelligence
    • Latest AI Innovations
    • Latest News
    • Privacy Policy
    • Disclaimer
    • Terms and Conditions
    • About us
    • Contact us
    Copyright © 2025 ProfitlyAI All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.