A FULLY AUTOMATED AND FAST APPROACH FOR CANOPY COVER ESTIMATION USING SUPER HIGH- RESOLUTION REMOTE SENSING IMAGERY

Canopy cover is a key agronomic variable for understanding plant growth and crop development status. Estimation of canopy cover rapidly and accurately through a fully automated manner is significant with respect to high throughput plant phenotyping. In this work, we propose a simple, robust and fully automated approach, namely a rule-based method, that leverages the unique spectral pattern of green vegetation at visible (VIS) and near-infrared red (NIR) spectra regions to distinguish the green vegetation from background (i.e., soil, plant residue, non-photosynthetic vegetation leaves etc.), and then derive canopy cover. The proposed method was applied to high-resolution hyperspectral and multispectral imagery collected from gantry-based scanner and Unmanned Aerial Vehicle (UAV) platforms to estimate canopy cover. Additionally, machine learning methods, i.e., Support Vector Machine (SVM) and Random Forest (RF) were also employed as bench mark methods. The results show that: the rule-based method demonstrated promising classification accuracies that are comparable to SVM and RF for both hyperspectral and multispectral datasets. Although the rule-based method is more sensitive to mixed pixels and shaded canopy region, which potentially resulted in classification errors and underestimation of canopy cover in some cases; it showed better performance to detect smaller leaves than SVM and RF. Most importantly, the rule-based method substantially outperformed machine learning methods with respect to processing speed, indicating its greater potential for high-throughput plant phenotyping applications.


INTRODUCTION
Rapid advancement of genetic engineering in recent years highly accelerated the plant breeding process which has put forward a growing demand for high-throughput phenotyping systems capable of efficiently measuring and quantifying plant traits to select the genotypes of interest (Ashapure et al., 2019;Yu et al., 2017). Canopy cover, also known as fractional vegetation cover, is often expressed as the percentage of plant canopy area within a unit ground area. Canopy cover is one of the most important agronomic indicators and phenotypical traits that reflect crop growth and development status, as well as plant photosynthesis and water use. Additionally, canopy cover has also been used to study crop leaf nitrogen concentration, leaf area index, biomass and grain yield (Maimaitijiang et al., 2020c;Xu et al., 2020). Thus, rapid and accurate estimation of canopy cover, particularly in a fully automated manner, is imperative for improved high throughput plant phenotyping.
High-resolution remote sensing is regarded as a key tool to extract canopy cover. RGB digital imagery acquired from low altitude aerial or ground platforms such as Unmanned Aerial Vehicle (UAV) or manned/unmanned ground systems are often used for canopy cover estimation. The most common approaches for canopy cover estimation include thresholding and classification/segmentation methods. Single or multiple thresholds are set to RGB imagery * Corresponding author transformed color space, such as CIE (Commission Internationale d'Eclairage) L*a*b* and HSI (hue saturation intensity) color (Liu et al., 2012), as well as vegetation indices such as ExG (Excessive Green Index) and CIVE (Color Index of Vegetation Extraction) (Meyer and Neto, 2008) through manual (Grieder et al., 2015), semi-automated (Xu et al., 2020) or fully automated (Coy et al., 2016;Yu et al., 2017) methods to separate the plant pixels from background non-green vegetation pixels such as soil, plant residue, and non-photosynthetic plant leaves etc. The thresholding approach is simple and widely employed in many applications, but it is less effective with images captured at various illumination conditions, or from different plant growth stages, or plants with variable water, nutrient and health status (Banerjee et al., 2020;Sadeghi-Tehran et al., 2017). Machine learning-based unsupervised classification methods such as k-means clustering (Aho et al., 2008;Marcial-Pablo et al., 2019), as well as supervised methods such as Decision Tree (DT) (Guo et al., 2013), Support Vector Machine (SVM) (Maimaitijiang et al., 2020b) and Random Forest (RF) (Sadeghi-Tehran et al., 2017) were previously used to estimate canopy cover and achieved superior results over thresholding method in many cases. However, classification methods require a certain level of human intervention preventing automation. Additionally, sample selection is often time consuming and model training and application may also be computationally intensive (Coy et al., 2016;Xu et al., 2020). In addition to using RGB digital imagery for canopy cover estimation, multispectral (Yu et al., 2017) and hyperspectral imagery (Banerjee et al., 2020) were utilized for canopy cover estimation through thresholding, machine learningbased classification, as well as spectral angle mapping method. Multispectral and hyperspectral imagery often provides stable and temporally comparable reflectance values, and demonstrates more accurate and robust performance compared to RGB imagery under various illumination conditions and for different plant canopy densities and background complexities (Ashapure et al., 2019;Marcial-Pablo et al., 2019) through differentiation of green vegetation pixels from non-green vegetation pixels (i.e., soil, plant residue, and non-photosynthetic vegetation leaves etc.). The objective of this work is to develop a simple, fast, fully automated and scale-invariant method for canopy cover estimation. Table 1, high-resolution remote sensing images were acquired over multiple experimental fields located across Maricopa in Arizona (AZ), Columbia in Missouri (MO) and Urbana in Illinois (IL) from 2017 to 2020, using various multispectral and hyperspectral sensors mounted on Gantry and UAV platforms (all datasets are collection from 10:00 am to 2 pm local time) ( Figure  1). A series of image preprocessing procedures, including lens distortion correction, geometric correction, as well as radiometric calibration were applied. It is worth noting that radiometric calibration is an important step for preprocessing multispectral (MSI) and hyperspectral (HSI) images which converts the raw image digital values/numbers (DNs) to normalized reflectance values that have physical meanings. Additionally, HSI and MSI with radiometrically calibrated reflectance values are able to cope with changes in illumination and solar geometry and often provide a stable and temporally comparable product (Marcial-Pablo et al., 2019).

Rule-based method
Photosynthetically active green vegetation presents a strong reflection pattern in the near infrared (NIR) spectral region (~700 -2500 nm) due to the cellular structure of the leaves, specifically the spongy mesophyll. In the visible (VIS) spectral region (~400 -700 nm), the reflectance is relatively lower because most light in this region is absorbed by the leaf pigments. Chlorophyll strongly absorbs energy in the blue (B) and red (R) spectral region, and reflects at the green (G) spectral region (Gamon et al., 2019). Thus, as displayed in Figure 2, the reflectance values of green vegetation are higher in the NIR region than the VIS region, and they are greater at the G spectral region than the R and B regions. In crop fields, background soil is the major surface type in addition to crop canopy, and the reflectance pattern of soil shows a completely different trend from vegetation; it gradually increases from VIS to NIR region ( Figure 2). Additionally, the spectral patterns of crop residue and dead leaves are also different from green vegetation. Therefore, on the basis of the distinct spectral profile pattern of green vegetation at the VIS and NIR spectral regions, a rule-based method (denoted as ) that is able to differentiate green vegetation from non-green vegetation areas such as background soil, as well as crop residue and dead leaves was proposed as follows: are reflectance values at G, R, B and NIR spectral regions, respectively. If the spectral profile of a pixel meets the 1 requirement (if 1 is true), that pixel will be classified as photosynthetically active green vegetation, otherwise, the pixel will be regarded as non-green vegetation areas such as background soil, as well as crop residue and dead leaves. Canopy cover is obtained by dividing the number of green vegetation pixels by the total pixel numbers within the target area. As shown in Figure 2, the reflectance value of soil at the NIR region is also higher than that at the R, G and B regions, thus, the restricted 1 is relaxed by excluding the > condition, and a relaxed rule was defined as 2 , which relies on R, G and B bands at the VIS region and expressed as follows: : > > If 2 is true, the pixel will be classified as green vegetation. The most distinct spectral pattern between green vegetation and background soil is that: if is greater or smaller than ( Figure 2), thus, a more relaxed rule, namely 3 , which only depends on G and R bands at the VIS spectral region was proposed as follows: : > If is true, the pixel will be classified as green vegetation.

Machine learning methods
Machine learning is one of the most common methods that has been employed for canopy cover estimation. Evidenced in previous studies, machine learning-based supervised classification approaches demonstrated superior performance to thresholding methods in many cases (Ashapure et al., 2019;Sadeghi-Tehran et al., 2017;Yu et al., 2017). Thus, in this work, we implemented two machine learning methods, i.e., Support Vector Machine (SVM) and Random Forest (RF) as benchmark methods for canopy cover estimation. For each dataset listed in Table 1, about 10,000 pixels were carefully selected to train and test the SVM and RF classifiers. Specifically, 70% of the 10,000 pixels from each dataset were randomly selected for training, and the rest were used for classification model testing. The performance of the proposed rulebased methods, as well as the SVM and RF classifiers were assessed by computing the overall accuracy (OA) for each method using the 30% testing samples. For SVM and RF, optimal model parameters were obtained through a k-fold cross-validation and grid-search technique (Maimaitijiang et al., 2020b).

Canopy cover estimation using hyperspectral data
The performance of the proposed rule-based methods, and the machine learning methods SVM and RF were evaluated using the manually selected testing samples (about 3,000 samples for each dataset). As shown in Table 2, the rule-based methods, as well as SVM and RF produced very high classification accuracies close to 100% for both hyperspectral datasets. Notably, the rule-based methods, regardless of the most restricted 1 (uses R, G, B, NIR bands) or the relaxed 2 (uses R, G and B bands) and 3 (uses R and G bands), yielded 100% accuracy in the case of the gantry hyperspectral data. Additionally, for the UAV hyperspectral dataset, the rule-based methods also generated slightly lower, yet very close performance to SVM and RF, indicating the robustness of the rule-based methods for green vegetation extraction and canopy cover estimation when using hyperspectral data. In the case of using the UAV hyperspectral dataset, although the performance of the rule-based methods slightly attenuated from the most restricted 1 with 99.8%, to the relaxed 2 with 99.7% and 3 with 99.6%, they still maintained comparable high accuracies with the machine learning methods. Since background soil comprises the majority of non-vegetation classes in crop fields, and the most distinct spectral pattern of green vegetation from soil is at the G and R spectral regions (Figure 2), specifically, for green vegetation, > , and for soil, < ; thus, even with a relaxed rule ( 3 ) method which only utilizes G and R two bands, it still obtains a high and comparable accuracies with the restricted rule method ( 1 ) which uses R, G, B and NIR four bands. This also indicates that NIR and B bands play less important roles in green vegetation/background soil discrimination. The rule-based method ( 1 ), as well as the trained SVM and RF models were implemented to multiple gantry and UAV-based hyperspectral data cubes to map green vegetation area and nongreen vegetation area (i.e., background soil, crop residue and dead leaves etc.), and estimate canopy cover. As displayed in Figure 3, the first column presents the RGB true color view of each hyperspectral data cube, and the rest of the columns are the mapping results along with estimated canopy cover values (the percentage values at the right-bottom corner of each map) based on different methods. For super high-resolution gantry hyperspectral datasets, the 1 method demonstrated better performance in detecting smaller leaves (Figure 3(f)), which were "ignored" by SVM (Figure 3(g)) and RF (Figure 3(h)). Additionally, as shown in (Figure 3(i) and (j)), the 1 method yielded good performance for the gantry hyperspectral image with a complex background. SVM also yielded good performance for different gantry hyperspectral data cubes. RF was not able to detect some shaded (Figure 3(d)) and overexposed (Figure 3(i)) green vegetation pixels, which likely led to possible underestimation of canopy cover, compared to the 1 method and SVM. In the case of UAV hyperspectral datasets, as shown in Figure 3(m) to Figure 3(t), the 1 method exhibited comparable performance to SVM with respect to excluding complex non-vegetation areas (Figure 3(q) and (r)), while RF produced poorer results with "pepper and salt" pattern noise. However, the spatial resolution of the UAV hyperspectral data is much lower than that of gantry imagery (Table  1), which potentially resulted in a higher chance of vegetation/background mixed pixels in the UAV hyperspectral data, especially the pixels at the outer border of each canopy row (Figure  3(m) and (q)). Those mixed pixels would possibly be classified as non-vegetation area by the rule-based method, which explains the relatively lower canopy cover derived by the 1 method for the UAV-sorghum scenes (Figure 3(n) and (r)). It is worth noting that, the performance of SVM and RF is more dependent on the quality of the samples selected for model training, and the criteria that those samples are defined/selected is subjective to user experiences and knowledge to some extent, for instance, whether the mixed pixels are treated as green vegetation area, or background soil area (Coy et al., 2016;Xu et al., 2020). However, the rule-based methods provided more objective and consistent results.

Canopy cover estimation using multispectral data
The testing results of different methods across multispectral datasets are shown in Table 3. Overall, both rule-based and machine learning methods demonstrated high classification accuracies greater than 90% across different scenes. The rule-based methods yielded slightly lower, yet comparable accuracies to SVM and RF for both the Altum and Parrot multispectral datasets (Table 3). Additionally, the accuracy attenuation pattern of the rule-based methods due to constraint relaxing, observed in the hyperspectral datasets, was also the case in the Altum and Parrot multispectral imagery (Table 3). Altum imagery has one more spectral band (blue band) than Parrot imagery, however, Altum imagery-based classification accuracies are lower than that of the Parrot datasets; this is likely due to the complexity of Altum datasets, which include images from multiple crops (corn and soybean) with various canopy densities, as well as variable illumination conditions (Table  1), while Parrot multispectral data is more uniform, which is collected from one UAV flight over a single soybean field (Table  1). It is noteworthy that the relaxation with the inclusion of only R, G, and B bands are important to robustly generalize the success of rule-based methods to regular digital RGB images for segmenting canopy cover. However, these remote sensing RGB images may require a normalization/calibration scheme to alleviate various illumination conditions and other sensing dynamics (Xu et al., 2020).  and machine leaning methods across five Altum and two Parrot multispectral image scenes. In agreement with the performance for hyperspectral datasets, the 1 method presented more powerful capability in detecting smaller leaves (Figure 4(b)), while they were "ignored" by SVM (Figure 4(c)) and RF (Figure 4(d)), which likely led to possible underestimation of canopy cover by SVM and RF. It is noteworthy that, consistent with the observation for hyperspectral datasets, the rule-based method ( 1 ) is sensitive to mixed pixels that do not present a distinct green vegetation spectral pattern. Additionally, as shown in Figure 4(f), (j), (n) and (z), the rule-based method is not able to accurately segment some shaded leaves as green vegetation area, which might be attributed to a shadow effect. As shown in Figure 5(b), the plant shadows may "break" the distinct spectral pattern of green vegetation, particularly in the case of multispectral datasets, and result in the failure of the rule-based method. Thus, this might lead to the possible underestimation of canopy cover by the rule-based method for scenes with higher densities that often have higher shaded areas (Figure 4(e), (i), (m) and (y)). As observed in Figure 5(a), for hyperspectral datasets, the shaded green vegetation still displayed a distinct spectral pattern that meets the requirement of 1 ( > > > ) to some extent. Thus, the rule-based methods are less sensitive to shadow effects in the case of hyperspectral data, which was also evidenced in Figure 3(b) in which the shaded leaves were successfully segmented as green vegetation by the rule-based method. However, the impact of shadow to the rule-based methods need to be systematically evaluated in future work.

Comparison of processing speed
In addition to the automatable property and estimation accuracy, processing speed is also a key factor for canopy cover estimation, particularly in the context of high-throughput phenotyping (Yu et al., 2017). The rule-based methods, as well as the trained SVM and RF models were implemented to four image cubes, respectively, to compare the processing time of each method used to classify those four images (Table 4). Overall, the rule-based methods substantially outperformed machine learning methods in terms of processing speed, particularly the RF method. Additionally, with the relaxation of the rules for the rule-based methods (from 1 to 3 ), the processing time decreased correspondingly, which is likely caused by the decreased number of bands/features involved in classification.

Correlation analysis of canopy cover with NDVI and LAI
Correlation analysis between canopy cover estimates and NDVI (Normalized Difference Vegetation Index) or field measured LAI (Leaf Area Index) is often conducted to evaluate the robustness of canopy cover estimation methods (Jimenez-Berni et al., 2018). The rule-based method ( 1 ) and trained SVM and RF models were implemented to the ortho-mosaicked UAV hyperspectral imagery (Table 1) acquired from an experimental sorghum filed established for high-throughput phenotyping research at the Maricopa Agricultural Center in Arizona on June 26 th , 2019, and the canopy cover was estimated for about 640 plots ( Figure 6). Plot-level mean NDVI were extracted for those 640 plots as well. Then, correlation analysis was conducted between plot-level mean NDVI and canopy cover estimates from the rule-based, SVM and RF methods. As shown in Figure 6(a, b, c), strong correlation relationships between NDVI values and the estimated canopy cover from all three methods were observed, with R 2 ranging from 0.95 to 0.98. Additionally, synchronously measured plot-level LAI with UAV imagery collected from 107 plots over the sorghum filed, were plotted against the corresponding canopy cover values estimated using the rule-based, SVM and RF methods, respectively (Detailed descriptions of the sorghum field and LAI measurement can be found in Maimaitijiang et al. (2020a)). As shown in Figure 6(d, d, f), decent correlation relationships between LAI and canopy cover were observed, with R 2 ranging from 0.66 to 0.70. The strong correlation relationships of the rule-based canopy cover with NDVI and LAI, particularly, the comparable performance of the rulebased canopy cover estimates to machine learning SVM and RF with respect to R 2 , affirmed the robustness of the rule-based method to some extent; this further revealed the great potential of the rulebased method for high-throughput plant phenotyping. Figure 6. Correlation analysis of canopy cover derived from the rule-based method, SVM, and RF, against NDVI (Normalized Difference Vegetation Index) (a, b, c) and LAI (Leaf Area Index) (d, e, f). Left side image is the RGB true color image of the sorghum filed overplayed with 640 plots.

CONCLUSIONS
This work demonstrated a fully automated, accurate and fast method to differentiate green vegetation from non-green vegetation area for canopy cover estimation. The proposed method, namely the rule-based method, outperformed machine learning methods such as SVM and RF for the very high-resolution gantry hyperspectral dataset. For UAV-based hyperspectral and multispectral imagery, the rule-based method yielded slightly lower but comparable accuracies with respect to SVM and RF. Additionally, the rulebased method showed better performance for detecting smaller leaves, and it also illustrated rapid processing speed, compared to SVM and RF. It is worth to noting that, the rule-based method is less robust in identifying the vegetation/soil mixed pixels, which would potentially lead to an underestimation of canopy cover. The rule-based method showed a higher sensitivity to shadow effects and was not able to accurately segment shaded green vegetation in the case of multispectral data to some extent. For future work, it will be of interest to further evaluate and improve the proposed rule-based method, particularly with respect to its performance over mixed pixels, as well as its adaptability to shaded green vegetation.