AI-DRIVEN MAIZE YIELD FORECASTING USING UNMANNED AERIAL VEHICLE-BASED HYPERSPECTRAL AND LIDAR DATA FUSION

: The increased availability of remote sensing data combined with the wide-ranging applicability of artificial intelligence has enabled agriculture stakeholders to monitor changes in crops and their environment frequently and accurately. Applying cutting-edge technology in precision agriculture also enabled the prediction of pre-harvest yield from standing crop signals. Forecasting grain yield from standing crops benefits high-throughput plant phenotyping and agriculture policymaking with information on where crop production is likely to decline. Advanced developments in the Unmanned Aerial Vehicle (UAV) platform and sensor technologies aided high-resolution spatial, spectral, and structural data collection processes at a relatively lower cost and shorter time. In this study, UAV-based LiDAR and hyperspectral images were collected during the growing season of 2020 over a cornfield near Urbana Champaign, Illinois, USA. Hyperspectral imagery-based canopy spectral & texture features and LiDAR point cloud-based canopy structure features were extracted and, along with their combination, were used as inputs for maize yield prediction under the H2O Automated Machine Learning framework (H2O-AutoML). The research results are (1) UAV Hyperspectral imagery can successfully predict maize yield with relatively decent accuracies; additionally, LiDAR point cloud-based canopy structure features are found to be significant indicators for maize yield prediction, which produced slightly poorer, yet comparable results to hyperspectral data; (2) regardless of machine learning methods, integration of hyperspectral imagery-based canopy spectral and texture information with LiDAR-based canopy structure features outperformed the predictions when using a single sensor alone; (3) the H2O-AutoML framework presented to be an efficient strategy for machine learning-based data-driven model building.


INTRODUCTION
Ensuring food security for the ever-increasing population under changing climate is a global challenge that requires cutting-edge technologies in agriculture management practices. Precision agriculture aims to utilize up-to-date solutions to reduce financial and environmental costs and improve crop value with limited natural resources under variable weather conditions caused by global warming. Preharvest yield information at a fine scale is essential in precision agriculture and high-throughput plant phenotyping (Maimaitijiang et al. 2020c). Foreseeing where productivity is likely to decline can help prevent the decrease and improve productivity before harvest. Thus, reliable preharvest crop yield prediction is imperative for grain policymaking and food security. Precision agriculture benefited from the development of Unmanned Aerial Vehicles (UAV) based remote sensing systems. Up to date, advancements in UAV and sensor technologies enabled remote sensing data acquisition with high spatial, spectral, and temporal resolution at a relatively low cost and time. In addition to the flexible temporal resolution, flying at a low altitude facilitated UAV to gain crop images with a high spatial resolution, improving monitoring performance (Tsouros et al. 2019). When Machine * Corresponding author Learning (ML) techniques are applied to hyperspectral and LiDAR data acquired from remote sensing platforms, forecasting crop grain yield from canopy signals can be achieved with comparatively low cost and decent accuracy.
The developments in machine learning provide enormous opportunities to create remote sensing-based data-driven models concerning crop monitoring, plant traits estimation, and grain yield prediction (Bhadra et al. 2020;Maimaitijiang et al. 2020b;Sagan et al. 2021a;Sagan et al. 2021b). ML conducts efficient identification of complex-linear/nonlinear relationships and automatically extracts spatiotemporal features from various input variables, enabling the estimation of plant traits and grain yield more accurately (Babaeian et al. 2021). The recent development of automated machine learning frameworks, such as H2O Automated Machine Learning (H2O-AutoML) (LeDell and Poirier 2020) etc., accelerated the implementation of ML in remote sensing-based applications through automated and streamlined feature selection, hyperparameter optimization and model evaluation functions (Babaeian et al. 2021). However, it has not been broadly employed in remote sensing agricultural applications such as crop yield prediction.
This research aims to evaluate the potential of point cloud LiDAR data, hyperspectral data, and their combinations in predicting maize yield from canopy signals of standing crops captured during the reproductive stage. Specific study goals are 1) to evaluate the potential of canopy spectra, texture and structure information extracted from hyperspectral and LiDAR imagery data in predicting maize yield; 2) to investigate the contribution of LiDAR point cloud-based canopy structural features in maize yield prediction accuracy; 3) to introduce the application of a fully automated machine learning-based approach in yield prediction.

DATA
On August 26 th ,2020, high-resolution remote sensing images were acquired from the University of Illinois Urbana-Champaign's agriculture research field near Champaign, Illinois (IL), using LiDAR and Hyperspectral sensors (Figure1) mounted on UAV platforms. The details of the UAV and sensors are listed in Table 1. 369 plots were sampled as ground truth maize dataset during the harvest that was conducted in October of 2020.
LiDAR data processing pipeline LiDARMill (Phoenix LiDAR Systems, Los Angeles, California, USA) was provided with the raw point clouds, trajectory files and the Novatel GNSS ground reference station-based GPS information for LiDAR data processing. Point clouds data steps combine IMU and GNSS data to generate precise trajectory (SBET) files and then detect flight lines to minimize processing time by automatically excluding turns. Calibration maneuvers focus on data-collecting flight lines and a LiDAR snap process that compares geometric observations made across overlapping flight lines to optimize alignment parameters and reduce offsets from multiple flight lines (Maimaitijiang et al. 2020a). The LiDAR mill pipeline delivers classified (ground/nonground) point clouds with ± 2 cm position accuracy. The 2D and 3D visualization of LiDAR point clouds from the cornfield are displayed in Figure 2, where the blue color points represent the ground primarily. In contrast, the red color points are mostly the higher canopy areas. The height dimension values of the point clouds are the elevation values, not the canopy height values The raw hyperspectral image cubes were processed with the SpectralView software (Headwall Photonics, Fitchburg, MA, USA) by radiometric calibration, geometric correction and ortho mosaicking steps. To convert the raw data displayed as 12-bit digital numbers (DN) to radiance values, dark and white reference information, taken before each flight, were used along with the factory calibration files; then, by using the imaged reflectance tarp, the radiance values were converted to surface reflectance factor (Hartling et al. 2021); 3D sensor positional information recorded by onboard IMU, and high resolution (10 m) digital elevation model (D.E.M.) through relevant functions of the SpectralView tool were incorporated for Orthorectification and geometric correction. Lastly, the geometrically corrected and orthorectified reflectance data cubes were stitched into a single image cube covering the whole field . Atmospheric correction was unnecessary due to low flight height, where the effect of the atmosphere is inconsiderable.

Feature Extraction
A set of vegetation indices often used for plant traits estimation and grain yield prediction and 269 original reflectance bands, and the grey co-occurrence matrix (GLCM) canopy texture features, introduced by Haralick et al. (1973), were extracted from the processed UAV hyperspectral imagery. For the machine learning prediction, the plot-level mean values of the above-mentioned spectral features' plot level mean values were used as input variables.
To remove outliers from the LiDAR data, the statistical outlier removal (SOR) algorithm was used. Plot boundary polygons were used to attain plot-level point clouds, which resulted in plot-level 3D point-cloud visualization (Figure3 (Maimaitijiang et al. 2020a). LiDAR intensity-based metrics were extracted at plot level along with a series of canopy height metrics, often representing canopy structural characteristics. Preprocessed LiDAR point cloud two-dimensional and threedimensional visualization is illustrated in Figure 2 below.

Modeling Methods
To build maize yield prediction models using UAV multisensoryderived canopy spectral, texture and structure features, this research applied the H2O-AutoML framework. H2O-AutoML contains supervised algorithms for classification and regression using tabular datasets. Feature scaling and selection, hyperparameter tuning and optimization through random grid searches, and generating several models based on numerous model performance metrics are all conducted automatically in H2O-AutoML. Therefore, H2O-AutoML permits a time-efficient workflow to find the optimal model without manual trial and error H2O-AutoML also supports the efficient processing of complicated datasets.

Model Evaluation Metrics
Randomly selected 80% of the canopy spectral, texture and structure features extracted from UAV-based Hyperspectral imagery and LiDAR point clouds were used as a training set, whereas the rest 20% of those features were used for the model testing. The fully automated processing pipeline of H2O-AutoML conducted feature scaling, feature selection, along with hyperparameter tuning; Therefore, features extracted from UAV imagery were directly fed to the AutoML models. The workflow of UAV image processing, feature extraction, model building, and testing is displayed in Figure 3. The predicted yield values were referenced to the ground truth values to evaluate AutoML models performance; three commonly used matrices: the coefficients of determination (R 2 ), the root mean square error (RMSE), and relative RMSE (RRMSE), were calculated to be quantified. Respectively, n is the number of yield samples used during the model testing phase, ̂, And ̅ are corresponding to the estimated, measured, and mean of measured yield values

Maize yield prediction results analysis
To predict maize yield with hyperspectral and LiDAR data alone and their combination (denoted as Hyper + LiDAR), NN, DRF, XRT, GBM and GLM machine learning methods under the H2O-AutoML framework were applied. The models and model validation statistics for the yield are presented in Table 2. Regardless of regression methods, hyperspectral and LiDAR data fusion produced a superior performance to using single sensorbased data alone, with R 2 ranging from 0.75 to 0.79, and RRMSE from 1.73% to 1.59% (Table 2). Canopy spectral and texture features extracted from hyperspectral imagery generated prediction accuracies (R 2 ) ranging from 0.73 to 0.76 and RRMSE from 1.79% to 1.68% (Table 2). Compared to hyperspectral imagery-based prediction results, the structure features extracted from LiDAR point clouds resulted in lower prediction accuracies with R 2 ranging from 0.55 to 0.67 and RRMSE from 2.32% to 1.98% (Table 2). Regarding the performance of regression methods, NN and GBM outperformed other methods when applied to the hyperspectral data; in the case of data fusion and LiDAR data, GBM provided the best results, DRF resulted in the lowest accuracies when using hyperspectral data, and GLM produced the poorest outcome in the case of data fusion and LiDAR data ( Table 2).
In Figure 4, predicted maize yield values resulting from five different machine learning methods are plotted against the ground truth maize yield values. Maize yield (Figure 4) samples are slightly underestimated when using hyperspectral imagery, perhaps partially due to the optical saturation problem. With the highest R 2 , 0.79 and lowest RRMSE, 21.1, hyperspectral and LiDAR data fusion proved to be the best prediction results, which was also demonstrated by convergence patterns of the spread points around the bisector.

Comparisons of Hyperspectral and LiDAR Performance
Canopy spectral and texture information extracted from remote sensing-based imagery are the critical features that have been widely used in crop monitoring and agricultural applications. In all of the prediction models, UAV hyperspectral imagery-based canopy spectral & texture features constantly outperformed LiDAR-based canopy structure features in maize yield prediction. The higher performance of canopy spectral & texture features to structure features is established with previous studies on plant biochemical and biophysical traits assessment (Maimaitijiang et al. 2020b).

Table 2. Validation statistics of AutoML of maize yield prediction
Hyperspectral-based canopy texture features are correlated with spectral features but also provides additional information associated with spatial canopy architecture and subtle structure characteristics, suppressing the soil-background effect and saturation issues while experiencing high spatial heterogeneity (Maimaitijiang et al. 2020c;Pacifici et al. 2009). That is probably the reason for the decent performance of the hyperspectral imagerybased yield prediction study.
Nevertheless, as listed in Table 2, canopy structure features extracted from LiDAR point cloud data produced lesser yet considerably decent prediction values to Hyperspectral imagerybased features, demonstrating point cloud-based 3D canopy structure information is a promising substitute to frequently used VIs and texture features. Previous findings have presented the competency of point cloud-derived canopy structure features in crop biomass, LAI and nitrogen concentration estimation (Luo et al. 2019;Maimaitijiang et al. 2020a). Canopy features extracted from LiDAR point could data contain great 3D canopy structural information and display additional details on canopy internal and side profiles, canopy density, distribution, and architectural patterns (Maimaitijiang et al. 2020a). Those details often reflect the cultivar differences and, canopy health growth status across the field. The points mentioned above likely are the cause behind the decent performance of LiDAR point cloud produced information for yield prediction to some degree.
Operational and processing difficulties (Verma et al. 2016) along with the limited canopy penetration capacity (Maimaitijiang et al. 2020a) obstruct LiDAR's applications for low-stature vegetation like maize. In this study, it is worth noting that high-density LiDAR point clouds generated decent performance for corn. This can be partially attributed to LiDAR's ability to characterize 3D corn canopy structure, as displayed in Figure 2a since LiDAR captures the median and lower canopy characteristics of corn and the overall status of the canopy. This could be attributed to corn having a relatively higher crop stand and is possibly easier to penetrate.

Contribution of Multisensory Data Fusion for Maize Yield Prediction
Fusion of LiDAR point cloud-based canopy structure features with multispectral/hyperspectral imagery-based canopy spectral information proved to increase model performance in several previous studies about plant traits estimation such as LAI (Maimaitijiang et al. 2017), biomass, and N estimation. A similar trend was also observed in this study. As shown in Figure 4, in all the prediction methods, the fusion of hyperspectral and LiDAR data constantly yielded higher performance than using a single sensor and increased R 2 and decreased RMSE%. Optical remote sensing data such as multispectral/hyperspectral imagery often suffers asymptotic saturation issues along with limited canopy 3D structure information; thus, the capability of multispectral/hyperspectral for plant traits estimation and yield prediction, especially in the case of dense or heterogeneous crop canopies, is often limited (Maimaitijiang et al. 2017). The addition of point cloud-derived 3D canopy structure features can provide vertical profiles concerning canopy height and 3D distribution; furthermore, combing LiDAR features with hyperspectral data can help with minimizing the saturation effect of optical remote sensing and complements spectral information to some amount. Consequently, this is likely the reason for improved prediction results. Extensive experiments should be done to evaluate the contribution of multisensory data fusion in crop yield perdition by considering various crop species and development stages, as well as different environmental and field conditions.

Performance of Yield Prediction Models
Among the five prediction models, GBM generated superior results to other models with the highest R 2 and lower RRMSE in all data input categories; NN equaled GBM when using the hyperspectral dataset as the highest prediction model. GBM and NN models are followed by DRF and XRT methods with relatively lower R 2 and higher RRMSE. The deep learning method NN frequently displays strong capability in dealing with nonlinear, complex, and large datasets (LeCun et al. 2015). It often overcomes other popular machine learning algorithms such as random forest, support vector machine, and gradient boost machine in classification and prediction applications (Cota et al. 2021); however, GBM exhibited relatively high performance than NN in the present study, which is likely due to the simple data structure and somewhat smaller sample size used in this our case.
Tree-based models, GBM, DRF and XRT, have different construction and internal evaluation; they often have a higher tolerance for data issues such as outliers and noise and can solve collinearity and overfitting problems, consequently exhibiting superior performance in remote sensing-based plant traits estimation and yield prediction in previous studies (Maimaitijiang et al. 2020b;Srivastava et al. 2021). The GBM, the boosting strategy-based model, is more suitable for the GBM (Talebpour et al. 2015) algorithm due to the specific data structure. DRF and XRT models are tree-based algorithms and employ a bagging strategy, yielding slightly different yet comparable prediction results in most cases. GLM model, in almost all instances, provided the lowest R 2 and highest RRMSE (table2). GLM is the extension version of linear regression; it supports linear, non-normally distributed dependent variables and supposes the independent variables are not correlated (Robinson and Schumacker 2009); which potentially restrains its ability when working with nonlinear data. That is likely caused by the poorer performance of GLM in this research.

CONCLUSIONS
This work demonstrated a comprehensive comparative study on maize yield prediction from UAV-based multisensory data combination using automated machine learning adds concluded the following findings.
 UAV platform incorporated with multiple sensors can provide multi-domain, canopy spectral, texture, structure, 2D and 3D, etc., crop canopy information, thus proving to be a capable tool for maize yield prediction.  With slightly lower prediction accuracies than hyperspectral data, LiDAR point cloud-based canopy structure features are significant indicators for maize yield. Hyperspectral imagery-derived canopy spectral and texture features can also successfully predict maize yield.  UAV-based multisensory data fusion provided superior performance in many previous studies concerning plant traits estimation and grain yield prediction. This study demonstrated the potential of multisensory data fusion in maize yield prediction compared to using a single sensor alone. Specifically, the inclusion of LiDAR-based canopy structure information to hyperspectral-based features decreases the optical saturation issues, and complementary information improves prediction accuracy.  The automated machine learning approaches such as H2O-AutoML are a valuable and efficient framework that substantially facilitated model building and evaluation procedures. Regarding the specific algorithm under H2O-AutoML, GBM outperformed other methods in most cases and was followed by the NN method; GLM generated the poorest results.
The study validated the enormous potential of UAV-based hyperspectral imagery and LiDAR point clouds, particularly Hyperspectral and LiDAR data fusion, in predicting maize yield via machine learning. Nonetheless, yield estimation via UAV-based multisensory data fusion and machine learning should be investigated across various crop types and in different field environments.