2.1. Data selection, filtering, and preprocessing

We select a subset of meteorological, chemical, and microphysical variables from the ACTIVATE campaign relevant to aerosol–cloud interactions. We balance including as many variables as possible for scientific discovery while including as much data as possible for data-driven machine learning. This tradeoff exists due to gaps in observational data records. Our subset selection was somewhat subjective but guided by literature as follows.

Initial variable selection was to limit analyses to warm clouds in the boundary layer, which was what was dominantly observed by the HU-25 Falcon. We removed aircraft-state observations (pitch, roll, heading, etc.), as well as local time and geolocation information (latitude and longitude). We further refined our variable list to focus on observations that were most widely studied in similar related papers (e.g., Fountoukis et al., Reference Fountoukis, Nenes, Meskhidze, Bahreini, Conant, Jonsson, Murphy, Sorooshian, Varutbangkul, Brechtel, Flagan and Seinfeld2007; McComiskey et al., Reference McComiskey, Feingold, Frisch, Turner, Miller, Chiu, Min and Ogren2009; Jones and Christopher, Reference Jones and Christopher2010). As a consequence of this literature search, we focused on using bulk aerosol and cloud droplet variables instead of their full observed size distributions. In accordance with guidance from instrument PIs on the ACTIVATE team, individual measurements marked as missing or otherwise flagged were removed.

Once a large subset of initial variables were identified, we evaluated the density of available data within each category. This is key because most modern machine learning methods require complete data records (i.e., complete rows in a tabular dataset). For the time period considered in this work, we calculate the percentage of data available for each of these variables. We observed a critical lack of data availability for some of the variables of interest, and so they were removed. This data density issue is a hallmark challenge of using observational data for machine learning instead of model data as in-situ measurements frequently have missing data. The final observational data we use is listed in Table 1.

Table 1. Variables used in the machine learning techniques

Pressure altitude is measured by the Applanix 610 (Sorooshian et al., Reference Sorooshian, Alexandrov, Bell, Bennett, Betito, Burton, Buzanowicz, Cairns, Chemyakin, Chen, Choi, Collister, Cook, Corral, Crosbie, van Diedenhoven, DiGangi, Diskin, Dmitrovic and Zuidema2023). Static air temperature is measured with a Rosemount Model 102 non-deiced total air temperature sensor with a fast-response platinum sensing element (E102E4AL) (Thornhill et al., Reference Thornhill, Anderson, Barrick, Bagwell, Friesen and Lenschow2003). The NASA Langley Turbulent Air Motion Measurement System (TAMMS) supplies measurements of vertical velocity (Thornhill et al., Reference Thornhill, Anderson, Barrick, Bagwell, Friesen and Lenschow2003). Relative humidity was derived from water vapor mixing ratio measurements made by the Diode Laser Hygrometer (DLH) (Diskin et al., Reference Diskin, Podolske, Sachse and Slate2002). The DLH measures water vapor along an open external optical path using wavelength-modulated laser absorption. Chemical concentrations of carbon monoxide and ozone concentration were measured with the PICARRO G2401-m (DiGangi et al., Reference DiGangi, Choi, Nowak, Halliday, Diskin, Feng, Barkley, Lauvaux, Pal, Davis, Baier and Sweeney2021) and 2B Tech Model 205 (Wei et al., Reference Wei, Shrestha, Pal, Gerken, Feng, McNelis, Singh, Thornton, Boyer, Shook, Chen, Baier, Barkley, Barrick, Bennett, Browell, Campbell, Campbell, Choi and Davis2021), respectively. The Langley Aerosol Research Group (LARGE) instrument team attached probes to the aircraft to measure aerosol and cloud droplet size distributions (i.e., size-resolved concentrations). Measurements from the Fast Cloud Droplet Probe (FCDP) give aerosol and cloud number concentration and effective diameter (divided by 2 here to give effective radius) (Kirschler et al., Reference Kirschler, Voigt, Anderson, Campos Braga, Chen, Corral, Crosbie, Dadashazar, Ferrare, Hahn, Hendricks, Kaufmann, Moore, Pöhlker, Robinson, Scarino, Schollmayer, Shook, Thornhill and Sorooshian2022) while liquid water content (LWC) is measured by the SPEC two-dimensional stereo instrument (2DS) (Kirschler et al., Reference Kirschler, Voigt, Anderson, Chen, Crosbie, Ferrare, Hahn, Hair, Kaufmann, Moore, Painemal, Robinson, Sanchez, Scarino, Shingler, Shook, Thornhill, Winstead, Ziemba and Sorooshian2023). Cloud condensation nuclei (CCN) number concentration was measured with a DMT Scanning Flow CCN Counter (Moore and Nenes, Reference Moore and Nenes2009). More details about the products provided in the ACTIVATE dataset can be found in Sorooshian et al. (Reference Sorooshian, Alexandrov, Bell, Bennett, Betito, Burton, Buzanowicz, Cairns, Chemyakin, Chen, Choi, Collister, Cook, Corral, Crosbie, van Diedenhoven, DiGangi, Diskin, Dmitrovic and Zuidema2023).

With the list of variables listed in Table 1, we filter these observations following recommended data quality screening, including removing individual timestep measurements flagged by instrument PIs as missing (Sorooshian et al., Reference Sorooshian, Alexandrov, Bell, Bennett, Betito, Burton, Buzanowicz, Cairns, Chemyakin, Chen, Choi, Collister, Cook, Corral, Crosbie, van Diedenhoven, DiGangi, Diskin, Dmitrovic and Zuidema2023). We additionally filter data to remove periods marked as take-off or landing, see Supplementary Material (S1). To ensure a useful comparison of microphysical measurements, we filter and keep CCN measurements within a specific supersaturation range of 0.3–0.5%. We then removed rows where one or more of our chosen parameters had measurements that were missing. After filtering was complete, the clean dataset had 34,277 instances for further analysis.

All data are normalized before training following best practices for machine learning model training on atmospheric data (e.g., Geiss et al., Reference Geiss, Silva and Hardin2022). Many of the features investigated here have log-normal or zero-inflated distributions, which cannot be trivially normalized following traditional standardization methods (e.g., z-score or min-max). Instead, we normalize each variable individually. For CCN concentration, we apply a box-cox transformation to achieve a more Gaussian distribution since the original distribution was heavily skewed. To aid with model training, we binarize LWC to serve as a cloud flag: 1 if greater than or equal to 0.2 x 10⁻⁵ kg m⁻³ representing in cloud and 0 if less than 0.2 × 10⁻⁵ kg m⁻³ representing out of cloud under the guidance of instrument PIs (Table 1). In the final normalization step, we subtract the minimum value of the parameter and divide it by the standard deviation of the parameter. The post-processing distributions of the parameters are shown in Figure 1 where air temperature, vertical velocity, CO, O₃, and CCN concentration have a roughly normal distribution, altitude, effective radius, and number concentration are slightly right skewed, and LWC is binary.

Figure 1. Distribution of normalized parameters after post-processing, therefore values are unitless.

For model training, the data is randomly split into training (70%), validation (10%), and test (20%) sets. We use the training and validation sets to optimize the model parameters and hyperparameters, respectively, and withhold the test set for model evaluation.

2.2. Dimensionality reduction

2.2.2. Autoencoder

We further explore dimensionality reduction with an autoencoder, a non-linear dimensionality reduction method. An autoencoder is a feed-forward neural network that is trained to predict its own inputs (i.e., as an approximation to the identity function). Autoencoders are constructed such that information flows through a low-dimensional “bottleneck” layer. We explore the information in that bottleneck space as a reduced-dimensional representation of the input data.

Autoencoders have been used in a variety of studies to explore low-dimensional variability in atmospheric datasets (e.g., Behrens et al., Reference Behrens, Beucler, Gentine, Iglesias-Suarez, Pritchard and Eyring2022; Kurihana et al., Reference Kurihana, Moyer and Foster2022). For example, Behrens et al. use a type of autoencoder called a variational autoencoder and reduce the dimensionality of climate model output by a factor of 12 (Behrens et al., Reference Behrens, Beucler, Gentine, Iglesias-Suarez, Pritchard and Eyring2022). They found that this low-dimensional latent space effectively represents large-scale climate variability and convective regimes (Behrens et al., Reference Behrens, Beucler, Gentine, Iglesias-Suarez, Pritchard and Eyring2022). Kurihana et al. use an autoencoder to capture the most relevant features of a large dataset of cloud satellite imagery. They find that an autoencoder can compress their dataset size by a factor of 15,000 and produces more descriptive categories than the nine categories previously established by the International Satellite Cloud Climatology Project (ISCCP) (Kurihana et al., Reference Kurihana, Moyer and Foster2022).

An example of neural network architecture for an autoencoder is shown in Figure 2. The autoencoder is composed of an input layer, followed by encoding layers, which bottleneck into a hidden layer, ultimately reversing the structure back to the size of the input layer through decoding layers. An autoencoder of this nature is trained by optimizing a set of weights and biases in each layer. These are commonly optimized by stochastic gradient descent adjusting to target a reduction in training loss calculated by a loss function (e.g., Chollet, Reference Chollet2021). For our purposes, we use the variables listed in Table 1 as input, where the data are then passed through the encoding layers, compressing the data into nodes within the hidden bottleneck layer. The hidden bottleneck layer feeds into a decoder, which expands the data back out and reconstructs the original input in the output layer. In this way, the autoencoder learns how to non-linearly encapsulate the data into the reduced dimensional space of the bottleneck layer during training.

Figure 2. Example of an autoencoder structure. The input layer is shown in blue, hidden layers are in orange, the bottleneck layer is in yellow, and the output layer is in green.

Autoencoders have a set of hyperparameters that are set prior to model training, which are described as follows. We use ReLU (rectified linear) activation function for all nodes. The learning rate is set to 0.001. The optimization method in this case is rmsprop (Hinton et al., Reference Hinton, Srivastava and Swersky2012). We use a custom loss function, described in a later section, to assess the performance of the model. The batch size is also set to the default of 32. To help prevent overfitting, we use an early stopping of 25 epochs to halt training if validation loss does not improve.

To evaluate the autoencoder, we treat each node in the bottleneck layer as a new axis to plot the data and much like the PCA analysis, look for regimes and relationships between variables and nodes. A comparison of the two dimensionality reduction approaches in this study is included in the table below (Table 2).

Table 2. Comparison of PCA and Autoencoder technique features

3.1. Principal component analysis

Using the scikit-learn optimal number of principal components (Minka, Reference Minka2000), we find nine principal components can explain 98.4% of the variance in the dataset. Figure 3 shows the associated scree plot, which shows how much variance is described by each principal component. From Figure 3, the majority (63.5%) of the variance in the entire dataset can be explained by the first four principal components. As a result, we use the first four principal components from this PCA to investigate the reduced dimensional space of our dataset given by this method.

Figure 3. Scree plot of principal component analysis (PCA) explained variance.

3.2. Autoencoder

3.2.1. Network Architecture

We build an optimized autoencoder architecture specific to this learning task. This involves selecting an optimal number of layers and nodes in the encoder and decoder, as well as selecting an appropriate number of nodes in the hidden bottleneck layer. The encoding and decoding layers and the units in each layer are optimized using the KerasTuner Hyperband hyperparameter optimization method with a mean squared error loss (Li et al., Reference Li, Jamieson, DeSalvo, Rostamizadeh and Talwalkar2018). We optimize the architecture for performance on our dataset by first using KerasTuner to find a candidate number of units for each layer, ranging from 1 to 100, and the number of layers in the encoder and decoder ranging from 1 to 5. We then fine-tune the KerasTuner hyperparameters to be sure that performance on each individual variable is good.

We separately optimize the number of units in the latent space by assessing the validation mean squared error for the autoencoder as a function of the size of the reduced-dimensional space. We compare this error against the validation loss from a PCA baseline, with results shown in Figure 4. For PCA, encoding dimensions represent the number of new orthogonal axes of the latent space. In the autoencoder, the encoding dimensions are the number of nodes in the hidden bottleneck layer. We ultimately chose 4 encoding dimensions since there is an “elbow” in the PCA plot at 3 encoding dimensions and a decent drop in loss between 3 and 5 dimensions for the autoencoder. In the end, this choice of encoding dimensions is subjective and balances keeping the number of dimensions low for understandability against the need for accuracy (which increases with the size of the number of encoding dimensions), similar to previous works (e.g., Behrens et al., Reference Behrens, Beucler, Gentine, Iglesias-Suarez, Pritchard and Eyring2022; Mooers et al., Reference Mooers, Beucler, Pritchard and Mandt2024).

Figure 4. Assessing autoencoder hidden layer units and principal component analysis (PCA) encoding dimensions. Note the break in the y-axis.

The final architecture structure is illustrated in Table 3 and follows what is shown in Figure 2. The input layer of 10 units is followed by one dense encoding layer, the dense hidden layer of four units, three dense decoding layers, and one dense output layer of 10 units.

Table 3. Autoencoder architecture

3.2.2. Custom loss

A few of the metrics included in our analysis are heavily zero-inflated, especially parameters like LWC, number concentration, and effective radius (see Figure 1). With zero-inflated data, the model can achieve high predictive accuracy by always predicting values of 0, with incorrect predictions for all non-zero values. To account for this during training, we build our own custom loss that applies a fractional penalty to values above a threshold.

The equation of our custom loss follows these conditions:

(1) $$ if\;{y}_{true}> threshold: $$

$$ penalty\bullet \frac{1}{N}\sum \limits_{i=1}^n{\left({y}_{predicted}-{y}_{true}\right)}^2 $$

$$ else: $$

$$ \frac{1}{N}\sum \limits_{i=1}^n{\left({y}_{predicted}-{y}_{true}\right)}^2 $$

where $ {y}_{true} $ is the true value of a particular instance and $ {y}_{predicted} $ is the associated predicted value from the autoencoder, n is the n ^th instance in the dataset, and N is the total number of instances. If $ {y}_{true} $ is greater than the threshold, a penalty is applied to the mean-squared error.

For values below the threshold value, mean squared error is applied (Equation 1). In practice, this loss down-weights the relative importance of accurate prediction of low (near zero) values in the dataset and increases the relative importance of accurate prediction of high values. The value of the penalty and threshold is determined by a grid search ranging from 0 to 100 for the penalty and 0 to 10 for the threshold. With each combination of penalty and threshold in place, the mean squared error of each variable in reconstruction was calculated. The penalty and threshold values are chosen based on favorable values of mean squared error across all variables with a final threshold of 0.0 and a final penalty of 0.1.

3.2.3. Autoencoder training

For our autoencoder, we use the 70/20/10 split for training, testing, and validation, respectively, as described in the Data Filtering and Pre-processing section. Figure 5 shows the loss after each epoch for both the training and validation sets. This loss curve shows that there was no apparent overfitting of the validation data and that the model converges.

Figure 5. Autoencoder model training and validation loss as a function of epoch.

3.3. Performance assessment

We assess the quality of the dimensionality reduction methods by their respective errors in reconstructing the original data using the unseen test set. To compare the performance of the two different machine learning approaches, we evaluate the predictions and true values and calculate the mean squared error between the predicted and true values as shown in Equation 2:

(2) $$ \frac{1}{N}\sum \limits_{i=1}^n{\left({y}_{predicted}-{y}_{true}\right)}^2 $$

where $ {y}_{true} $ is the true value from the dataset and $ {y}_{predicted} $ is the predicted value from the autoencoder, n is the n ^th instance in the dataset, and N is the total number of instances.

The total reconstruction mean squared error for PCA is fairly small, being 1.67 x 10⁻¹ on the test set for the normalized variables. Mean squared errors for individual features with normalized units are summarized in Table 4 and range from 1.83 x 10⁻¹ to 6.73 x 10⁻¹. In terms of MSE, PCA reconstructs air temperature and carbon monoxide the best but number concentration and liquid water content poorly. The density plot results for each feature are shown for PCA in Figure 6 where the predicted value is plotted versus the true value. In general, there is a fair prediction of 6 of the 10 variables. The exceptions are vertical velocity, LWC, effective radius, and number concentration, where prediction errors are large in magnitude and skew off the 1:1 line. For vertical velocity, measurements with values in the extremes are not well predicted (above and below a value of 5) and there is a subset of values with 5 as the true value, which are predicted to have negative values in PCA transformation (Figure 6c). LWC does not have a good prediction for out-of-cloud data points (Figure 6h). For true values of R_eff and number concentration, there is an underprediction by the PCA, suggesting that the PCA is not able to reconstruct the higher values of R_eff and number concentration (Figure 6i, j).

Table 4. Individual mean squared error (MSE) from PCA

Figure 6. Principal component analysis (PCA) performance for test data a) altitude, b) air temperature, c) vertical velocity (w), d) relative humidity (RH), e) carbon monoxide (CO), f) ozone (O₃), g) cloud condensation nuclei (CCN), h) liquid water content (LWC), i) effective radius (R_eff), j) number concentration (N). All units are normalized following the procedure outlined in Section 2.1.

An analogous assessment of the autoencoder is shown in Figure 7. We find the total reconstruction mean squared error (MSE) for the autoencoder to be 8.67 x 10⁻³ for the test set, nearly two full orders of magnitude lower than the PCA approach. The MSE for each normalized variable is included in Table 5 and ranges from 1.65 x 10⁻² to 1.36 x 10⁻¹. Compared to PCA, the error for each individual metric predicted by the autoencoder is less than the PCA approach by at least a factor of two. There is marked improvement in the autoencoder MSE for altitude, air temperature, vertical velocity, relative humidity, liquid water content, and number concentration over the PCA approach where the reconstruction error improves by an order of magnitude. Reconstruction error of carbon monoxide, ozone, cloud condensation nuclei, and effective radius also improves but to a lesser extent.

Figure 7. Autoencoder performance for test data a) altitude, b) air temperature, c) vertical velocity (w), d) relative humidity (RH), e) carbon monoxide (CO), f) ozone (O₃), g) cloud condensation nuclei (CCN), h) liquid water content (LWC), i) effective radius (R_eff), j) number concentration (N). All units are normalized following the procedure outlined in Section 2.1.

Table 5. Individual mean squared error (MSE) from autoencoder

The density plots in Figure 7 demonstrate that there is generally a much better prediction of all variables for the autoencoder than PCA. Predicted values tend to cluster closer to the 1:1 line, with higher correlations and fewer outliers. In particular, the autoencoder better reproduces the extrema of the true values. This is particularly evident in R _eff and N (Figure 7i,j). Additionally, a subset of vertical velocity predictions near the mean of the true distribution are more accurately predicted along with a better prediction of LWC values of zero (Figure 7c,h). Overall, the autoencoder efficiently reconstructs the dataset over the range of their values, offering a robust and representative reduced dimensional space.

4.1. Principal component analysis

To investigate the data in the reduced dimensional space provided by PCA, we plot the test data on the first four principal components, shown in Figure 8. Some distinct regimes are found in the latent spaces as some of the principal components set groups of datapoints apart from the main data distribution. Although PC1 explains most of the variance in the dataset (Figure 3), we do not identify discernable structures in the first two panels (Figure 8a,b). However, we identify a specific regime, which is set apart by PC4 (Figure 8c,e,f). There is another cluster of points associated with low values of PC2 and high values of PC3 (Figure 8d), which will be explored in this section.

Figure 8. Test data plotted in principal component (PC) space a) PC2 vs PC1, b) PC3 vs PC1, c) PC4 vs PC1, d) PC3 vs PC2, e) PC4 vs PC2, f) PC4 vs PC3.

Ultimately, we find that two apparent regimes in Figure 8 can be attributed to “low pollution” (Figure 8d) and “in-cloud” (Figure 8c) conditions. We discuss the characteristics of these regimes below and illustrate the regimes through coloring points with the native measurements of metrics of interest.

The plot of PC3 vs PC2 (Figure 8d) is shown in Figure 9 with points colored by two gas-phase tracers of pollution, CO and O₃. The cluster of points present with low PC2 and high PC3 corresponds closely to low observed CO and O₃ concentrations, consistent with this regime being associated with lower pollution levels. There is some correlation between the regime highlighted and higher air temperatures, although this relationship is not as pronounced (Figure 9c).

Figure 9. Principal component analysis (PCA) for low pollution conditions. PC3 vs PC2 is colored by a) carbon monoxide concentration (CO), b) ozone concentration (O₃), and c) air temperature.

In addition to the low pollution regime evident in Figure 8, PCA shows distinct in-cloud and out-of-cloud regimes. This in-cloud and out-of-cloud distinction is most apparent in PC4. To illustrate this, we color-test data points by metrics plotted in the PC4 vs PC1 latent space in Figure 10. In Figure 10a, we see that there is a clear separation of LWC binary values, which correspond with in-cloud (above or equal to 0.2 x 10⁻⁵ kg m⁻³) and out-of-cloud (below 0.2 x 10⁻⁵ kg m⁻³). Vertical velocity decreases with PC4, with the decrease separately occurring within the two regimes delineated by LWC.

Figure 10. Principal component analysis (PCA) in-cloud regime. PC4 versus PC1 is colored by a) liquid water content (LWC) and b) vertical velocity (w).

While the PCA is able to successfully highlight low pollution and in-cloud regimes, it is important to recall that PCA performed poorly on reconstruction, and it is, therefore, useful to also evaluate a method that is better able to capture the nuances of this dataset.

4.2. Autoencoder

Next, we investigate test data in the latent space of the autoencoder. To do so, we treat each of the four encoding nodes of the hidden layer as a new axis on which to plot the test data, analogous to the projection onto the principal components in Section 4.1. The unseen test data plotted in log space on these new axes is illustrated in density plots in Figure 11. Here, we show all combinations of node relationships and identify regimes residing in this reduced dimensional space. Distinct structures are not identified in the first two nor fourth or sixth panels (Figure 11a,b or Figure 11d,f). We identify distinct regimes set apart in the third and fifth panels (Figure 11c,e). In Figure 11c, we see a distinct regime in high values of Node 4. In Figure 11e, there is a cluster of points set apart from the bulk of datapoints concentrated at high values of both Node 2 and Node 4. Overall, test data plotted in this latent space vary more continuously than the more obvious clustering evident in PCA latent spaces, but the latent spaces of the autoencoder represent specific states.

Figure 11. Test data plotted in autoencoder node relationship plots a) Node 2 vs Node 1, b) Node 3 vs Node 1, c) Node 4 vs Node 1, d) Node 3 vs Node 2, e) Node 4 vs Node 2, f) Node 4 vs Node 3.

Similar to PCA, there is also a low pollution condition relationship in the autoencoder latent space with the log relationship of Node 1 and Node 4 as shown in Figure 12. At values of approximately 1 for the log of Node 4 and 0 to 0.5 for the log of Node 1, we see the lowest values of CO and O₃ (Figure 12a and 12b). Whereas Figure 9c shows a strong relationship between PCs with CO and O₃ but a weak relationship between those same PCs and air temperature, we see that this low pollution regime is tightly delineated with a gradient in high temperatures as well as low pollution in this autoencoder regime (Figure 12c). This provides more insight into the meteorological conditions that go hand in hand with low levels of gas phase pollution in the ACTIVATE dataset.

Figure 12. Autoencoder low pollution conditions. Node 4 versus Node 1 is colored by a) carbon monoxide concentration (CO), b) ozone concentration (O₃), and c) air temperature.

As shown in Figure 13, the autoencoder highlights an in vs out of cloud regime with the relationship between Node 2 and Node 4, similar to PCA (Figure 10). Values of LWC indicating in-cloud measurements are set apart with high values of Node 2 and Node 4.

Figure 13. Autoencoder in-cloud regime highlighted by Node 4 versus Node 2 colored by liquid water content (LWC).

However, the autoencoder goes above a simple demarcation of LWC threshold the PCA showed (Figure 10). By focusing on the in-cloud regime located at high values of Node 4 and Node 2 (Figure 13), we find a correlation between this subset of datapoints arranged in this latent space and several variables shown in Figure 14. With PCA, we saw a linear relationship between vertical velocity and PC 4 (Figure 10). Here we see a correlation between Node 2 and Node 4 for both vertical velocity and R _eff (Figure 14a,b). There is also a potential correlation between Node 2 and Node 4 and number concentration, although this relationship is largely driven by a few instances of in-cloud datapoints (Figure 14c).

Figure 14. Autoencoder in-cloud regime magnified. Node 4 versus Node 2 is colored by a) vertical velocity (w), b) effective radius (R_eff), and c) number concentration (N).

We also investigated the out-of-cloud regime to see if there are any relationships found in the latent space of log Node 4 versus log Node 2 in Figure 15. Here, we see that there is a positive correlation between Node 2 and number concentration but there is an additional strong relationship between effective radius and Node 2 (Figure 15c,b). The linear relationships between the metrics effective radius and number concentration with Node 2 are consistent in both in-cloud and out-of-cloud regimes. However, this is not the case with vertical velocity where the linear relationship shown in the in-cloud regime is not nearly as strong in the out-of-cloud regime (Figure 14a and 15a). There also appears to be a potential relationship between Node 2 and the presence of particles as denoted by the measurement of effective radius and number concentration.

Figure 15. Autoencoder out of cloud regime. Node 4 versus Node 2 is colored by a) vertical velocity (w), b) effective radius (R_eff), and c) number concentration (N).

Ultimately, this relationship between nodes in- and out of cloud formations is consistent with our understanding that vertical velocity and aerosol size are important in aerosol-cloud processes (e.g., Twomey, Reference Twomey1959; Fountoukis et al., Reference Fountoukis, Nenes, Meskhidze, Bahreini, Conant, Jonsson, Murphy, Sorooshian, Varutbangkul, Brechtel, Flagan and Seinfeld2007).

There is utility in both the PCA and autoencoder approaches. The distinct regimes of cloudy and clean are consistent across both linear and non-linear methods. Both the encoded dimensions of PCA and the latent space of the autoencoder highlight low pollution conditions and in-cloud regimes. However, the autoencoder performs better than PCA by two orders of magnitude and represents the non-linearity of the data in the latent space. This is consistent even in applications of these tools in other applications (e.g., Hinton and Salakhutdinov, Reference Hinton and Salakhutdinov2006; Kiran et al., Reference Kiran, Thomas and Parakkal2018; Fournier and Aloise, Reference Fournier and Aloise2019).