POINTNET++ TRANSFER LEARNING FOR TREE EXTRACTION FROM MOBILE LIDAR POINT CLOUDS

: Trees are an essential part of the natural and urban environment due to providing crucial benefits such as increasing air quality and wildlife habitats. Therefore, various remote sensing and photogrammetry technologies, including Mobile Laser Scanner (MLS), have been recently introduced for precise 3D tree mapping and modeling. The MLS provides densely 3D LiDAR point clouds from the surrounding, which results in measuring applicable information of trees like stem diameter or elevation. In this paper, a transfer learning procedure on the PointNet++ has been proposed for tree extraction. Initially, two steps of converting the MLS point clouds into same-length smaller sections and eliminating ground points have been conducted to overcome the massive volume of MLS data. The algorithm was tested on four LiDAR datasets ranging from challengeable urban environments containing multiple objects like tall buildings to railway surroundings. F1-Score accuracy was gained at around 93% and 98%, which showed the feasibility and efficiency of the proposed algorithm. Noticeably, the algorithms also measured geometrical information of extracted trees such as 2D coordinate space, height, stem diameter, and 3D boundary tree locations.


INTRODUCTION
Improving human health outcomes, including 16% lower heart disease and stroke rates, reducing household cooling costs by more than 10%, providing wildlife habitats, and improving air quality are just examples of tree advantages (www.mycanopy.unley.sa.gov.au).Also, reducing erosion, flooding, and stormwater runoff is another considerable benefit of trees (Tomar et al., 2022).These have resulted in attracting scientists' attention toward automatically recognizing vegetations, whether forests or urban environments, for planning and studying the distribution of trees.
Recently Remote Sensing (RS) and Photogrammetry equipment have played a key role in object detection and evaluation, including trees.This is because various valuable tools like radiometric images, Synthetic Aperture Radar (SAR) data, and Light Detection and Ranging (LiDAR) point clouds can be efficiently mounted on diverse RS platforms such as airborne, Unmanned Aerial Vehicle (UAV), cars, or even backpacks (Li et al., 2022;Wang et al., 2022;Wang et al., 2019;Yadav, 2021).Through these platforms, Mobile Laser Scanner (MLS) provides beneficial information in terms of both recording time and acquired accuracy (Che et al., 2019).Valuable data can be mentioned as (1) collecting thousands of 3D point clouds per a square meter from a side view look, (2) acquiring accuracy around sub-centimeters, and (3) recording about 50 kilometers of roadside objects per an hour due to mounting on a car, (4) giving directly precise 3-dimensional coordinate points without needing any preprocessing step unlike 2-d images and also (5) being georeferenced is another positive side of the MLS because can be quickly registered and matched with another source of data like * Corresponding Author satellite imageries for getting more information (Wang et al., 2019).
The MLS system consists of laser scanners for collecting points in a polar coordinate system, Initial Measurement Unit (IMU)measuring angular rate and orientation, Global Navigation Positioning System (GNSS) -registering and georeferencing the laser scanner beams, Distance Measurement Instrument (DMI)recording the MLS vehicle position on the 2-dimensional coordinate system (literally this is called trajectory data), and cameras (Shokri et al., 2019).The trajectory data is a noticeable advantage of the MLS system over other popular collecting point clouds platforms such as Airborne Laser Scanner (ALS) (Behley et al., 2021).Because it is georeferenced with the MLS point clouds, has a low volume, and is located on the road surface, it is used vastly beside the recorded MLS point clouds.MLS outputs are irregular 3D point clouds, intensity data representing the strength of reflected point beam, trajectory data, and radiometric images from a 360-degree view.Although these data provide essential information about the surroundings, immense volume and noisy points are two main challenges that should be minimized or removed efficiently (Awrangjeb, 2019).
The GNSS highly depends on the satellite visibility for calculating the 3D positioning; otherwise, the overall accuracy would be decreased sharply with a few satellite visibilities (Behley et al., 2021).More importantly, higher objects like trees and buildings are available than the MLS vehicle, causing the little view of satellite visibility.Likewise, the error of multi-path in the GNSS may occur because of available buildings in the surrounding (Soloviev and Graas, 2009).Although the integration of IMU and GNSS, moreover, using Simultaneous Localization and Mapping (SLAM) techniques would considerably adjust this issue, in some cases, they may not work properly (Tan et al., 2020).This error would cause the elevation of some points to be an abnormal value called noisy points.The noisy points should be eliminated because of having an unpredictable pattern and may negatively influence the final outputs.
Researchers have used the MLS LiDAR point clouds in various challengeable and complex objects like lengthy power lines (Zhu and Hyyppä, 2014), pole-shaped objects (Shi et al., 2018), tall buildings (Li et al., 2016), road surface analyzing (Zhao et al., 2021), traffic signs (Wu et al., 2015), road markings (Rastiveis et al., 2020) and densely trees (Pérez-Martín et al., 2021).These proposed algorithms tried to detect objects in an acceptable computation time and high accuracy because one of the MLS challenges is the immense volume of the collected data.Still, tree recognition in the MLS data is a hot topic due to its complex geometric structure and unpredictable patterns in shape, size, and growth.
Although the trees would supply a vital role in society, ranging from pollution reduction to acid rain control, they may also face some challenges.Examples can be seen in wildfires which frequently occur all over the world.More than about 2.5 million acres of trees were burnt in 2021 and around 4.3 million acres in 2020 in California (www.universityofcalifornia.edu).Electricity power lines, lightning, unusual drought, and heat exacerbated by climate change are the main factors of these fires (Petrov et al., 2022).Also, trees may block the mobility of road surfaces by falling off in stormy weather conditions (Cai et al., 2022).Likewise, the tree roots may destroy the structure of pavements or road surfaces by getting out of the ground.More importantly, as the trees consume water, particularly in the urban environments, managing the amount of needed water is a considerable problem for the governments, especially in semiarid and arid countries like Iran.
Recently various methods ranging from deep learning neural network structures to rule-based descriptors have been proposed for tree extraction from MLS LiDAR point clouds (Dai et al., 2018).They can be categorized into three groups (1) mathematical-based methods, (2) rule-based descriptors, and (3) deep learning algorithms.Regarding the mathematical methods, Canopy Height Model (CHM) and Hough Transform (HT) are two popular ones in tree detection, which both convert the point clouds into 2-dimensional raster images.For example, Safaie et al. (2021) initially filter ground points in a preprocessing step to accelerate the computation time.Next, the tree trunks were extracted by applying the HT algorithm to the raster images.Converting the point clouds into an image space is the main drawback of these algorithms despite getting acceptable results.In terms of the rule-based methods, the handicraft descriptors like Linearity can be implied on the point clouds and then fed to the classifiers of machine learning procedure such as Support Vector Machine (SVM).These descriptors apply meaningful values to each point, meaning that a cable which has a linear structure would get a higher value than other non-linear objects such as building facades.Zaboli et al. (2019) classified the MLS point clouds like roads and trees by feeding ten descriptors to classifiers of SVM, Random Forest (RF), K Nearest Neighbor (KNN), and Multi-Layer Perceptron (MLP).They finally concluded that the RF performed better in object extraction like trees.The primary positive side of these methods is high computation time and describing objects' geometrical structure.Unlike the mathematical and rule-based methods, algorithms of deep learning-based one would not need any conversion from 3 dimensional to a lower space or measuring manual descriptors.
These methods directly consume the point clouds and extract objects.PointNet and PointNet++ are commonly used procedures in point cloud classification and segmentation.A PointNet++ algorithm was proposed by Ma et al. (2022) for urban area object extraction, including a tree which gained an accuracy of around 84.7% for tree detection.The considerable disadvantage of these methods is needing big training data and, more importantly, the high computational time.
In this paper, we have tried to propose an efficient and fast algorithm for extracting urban and sub-urban trees recorded by the MLS system.To overcome the immense volume of data, separating the recorded data into the same-length section initially is suggested that afterward, the ground points are eliminated inside each section.This would result in removing a considerable number of unneeded points.Notably, the proposed algorithm measures characteristic features of trees like elevation and stem diameter besides evaluating tree encroachments on power line cables and road surface structures.The tree should be trimmed if the encroachment was not following a standard distance.The contributions of our work are as follows: • Proposing a fast and robust deep learning neural network structure such as PointNet++ for tree extraction.This has been tested on four challengeable urban and suburban areas for tree analysis.

•
Calculating tree parameters such as 2-dimensional coordinate space, elevation, stem diameter, and maximum foliage diameter.

•
Evaluating the tree encroachment on power line cables and road surface infrastructures to evaluate the possibility of fires and reducing of blocking driver view by leaves.

METHODOLOGY
Figure1.Flowchart of the proposed algorithm.
The proposed algorithm consists of four steps (1) preprocessing to overcome the immense volume of MLS LiDAR point clouds, (2) candidate point selection to limit the search area to find tree points, (3) tree extraction by PointNet++, and lastly (4) tree inventory measurement to measure characteristic features and encroachment analyzing.Fig. 1 shows the flowchart of the proposed algorithm.

Pre-processing
This step is considered for enhancing the proposed algorithm in terms of computation time and robustness on noisy points.It includes three parts of (1) Sectioning, (2) Noise Removal (NR), and (3) Ground Point Elimination (GPE).Sectioning and GPE enhance the computation time process, while NR reduces the negative impact of noisy points on the final outputs, each of which is discussed as follows: • Sectioning Thanks to the capability of Robot Operating System (ROS) technology, when the MLS vehicle is recording data, saving the MLS LiDAR point clouds as separate LAS files would be simultaneously implemented, meaning that the whole data is recorded with various small size files, not a heavy file.The LAS file is a popular format designed for archiving LiDAR point clouds and saving data.This step needs a conditional parameter to save LAS files where the trajectory data here would play a positive role.The conditional parameter is when the vehicle passes a specific passway length (L).The recorded point clouds are saved as a separate LAS file archive.The parameter of L is calculated with the help of trajectory data and Euclidean distance measuring like the equation below (Eq.1).The sectioning stage would result in collecting same-length point clouds with approximately the same number of points in each section.2021a) study, a comprehensive parameter evaluation, including sectioning, was made about computation time and acquired output accuracies.They concluded that the optimized value for L is 90m long which here this value is also used.
• Noise removal (NR) Five steps are considered to detect noisy points inside each crested section.Initially, every point (seed) finds its adjacent points using the K Nearest Neighbor (KNN) algorithm.Then, the Euclidean distance between the seed point and their k neighborhood points would be computed, called local measuring.For each seed point, k distances are calculated.Afterward, two parameters of local mean and local standard deviation are measured for calculated distances in the local measuring, meaning that the mean and standard deviation of the calculated distances are just measured.Assuming that the MLS point clouds distribution is Gaussian, global mean and global standard deviation parameters are calculated from the whole calculated local means and local standard deviations.Each created section from the sectioning only has a unique global mean and global standard deviation.Since the number of noisy points would not be more than one percent of each section volume, the measured global mean and standard deviations are far away from the noisy points.Consequently, those neighborhood points which have a distance from the points more than the sum of the global mean and global standard deviation are considered the noisy points and eliminated (Fig. 2).Fig. 2 displays the result of the noise removal process on a sample LiDAR point cloud section where the color is height index, meaning that as long as the elevation of a point is increased, the color of it is brighter.The parameter of k here was selected equal to 20. •

Ground point elimination (GPE)
The difference between trees and other natural vegetations like bushes is the elevation parameter, meaning that those vegetations that have an elevation more than 4m from the ground surface are considered trees (www.frontiersin.org).This type of vegetation classification plays a vital role in removing unneeded points such as ground surface ones.The popular ground extraction method like pseudo nDSM generation or Cloth Simulation Filter (CSF) needs an elevation parameter as an inputted value (Yang et al., 2020).
Here, the CSF was considered for ground point extraction because it robustly simulates the ground surface structure ranging from flat to steep ones.Moreover, if a place of collected LiDAR point clouds did not have points, it would also simulate the ground surface properly.CSF needs two parameters of grid size and elevation where selected, 0.5m and 0.3, respectively, as suggested by the author.

Candidate Points Selection
After eliminating the ground points, various challengeable objects are still available, like buildings, pole-shaped objects, and traffic signs beside the trees.Therefore, this step aims to identify candidate points for limiting the search area to tree recognition.Thanks to the descriptors of rule-based descriptors, which the Principal Component Analysis calculates (PCA) algorithm, they can assign values between [0,1] to the remained points.For instance, by applying the descriptor of Linearity, those objects like power line cables following a linear structure would get a value near one and other non-linear objects much lower than one.Likewise, the descriptor of planarity also detects the building facades and flat structures like traffic signs.This trend also goes through the pole-shaped objects, meaning that the Verticality descriptor detects them.The Linearity, Planarity, and Verticality are measured based on the eigenvalues of PCA parameters as follows: where ( 1  2  3 ) = the PCA eigenvectors ( 1  2  3 ) = the PCA eigenvalues

Tree Extraction
Pointnet++, as a deep neural network, consumes the 3D point cloud directly and provides a unified approach using Pointnet as the local feature learner for classification and segmentation applications.
Pointnet++ is a robust network architecture for processing a set of sampled points in a metric space.Pointnet++ functions recursively on a nested partitioning of the point set and is effective in learning hierarchical features for the distance metrics.The hierarchical structure is composed of several set abstraction levels.The set abstraction layers consist of the sampling layer, the Grouping layer, and the PointNet layer.
The first approach to implementing the proposed algorithm to the dataset is applying transfer learning to take advantage of the learned features in a more complex dataset.The transfer learning process starts by training the Pointnet++ in the Dublin city dataset, and the resulting model is used for feature extraction (www.v-sense.scss.tcd.ie).This means that the transfer learning model relies more on the geometry learned within its pre-trained weights, thus avoiding overfitting.The introduced dataset was tested with Pointnet++ classification pretraining weights to evaluate the proposed architecture.
• Tree inventory measurement After detecting tree points, individual trees from the extracted trees are segmented based on the methodology proposed by Safaie et al. (2021).Next, this step aims to measure the characteristic feature of each tree as follow:

•
Characteristic measuring Characteristic measuring means a set of tree parameters is going to be estimated.Elevation, stem diameter, stem elevation, and, more importantly, the 3D boundary of leaves are measured from this stage.Tree elevation parameter.assume   = {  ,   ,   }, 0 ≤  ≤  ( is the number of individual tree points) is the points of an individual tree.Initially, the two points which have the maximum (  ) and minimum elevation (  ) in T are found.The elevation of an individual tree is equal to the subtraction of   and   .
Stem Diameter (SD) parameter.Thanks to the methods of circle fitting procedures with the help of Random Sample Consensus (RANSAC), the diameter of the tree stem will be estimated.As stems are located bottom part of trees, firstly, those points placed in the lowest part of  with elevation ranging from   and   + 1 are detected.Afterward, a circle equation (Eq.6) is fitted to the extracted points.
3D Boundary extraction parameter.Recently, the procedure of convex hull has shown positive performance in object analysis from radiometric imageries or LiDAR point clouds (Yan et al., 2019).Regarding tree points, the convex hull remains the 3D boundary points and eliminates interior ones that are useless in parameter measuring.Indeed, the convex hull is the smallest shape that contains the tree, reducing each tree's volume.
Foliage height (FH), maximum foliage diameter (MFD), and trunk height (TH) are three other parameters that will be measured from each tree (Fig. 3).• Encroaching on power line (EPL) and roadway corridor (ERC) Here, the proximity of trees to power transmission lines and roadway is estimated.Regarding power lines, cables are extracted by a machine learning procedure proposed by Shokri et al. (2021b), where fast implementation is one of its powerful features.Then, the minimum distance between extracted 3D boundary points and cables is measured.If the minimum value was about 2m, the tree should be trimmed and considered an encroachment.This trend also goes through the roadway between boundary trees and roadway points.
Nevertheless, the trajectory data is considered instead of road surface points.Road boundary extraction would not run rapidly due to millions of points processing located on the road structure.Also, the trajectory points are placed on the road surface and have a much lower volume of implementation.The distance condition here is 3.6m.

Accuracy Assessment
Generally, object extraction methods, including the proposed algorithm, are evaluated by precision and recall accuracies.Three variables of True Positive (TP), False Positive (FP), and False Negative (FN) would be needed to calculate the accuracies.whrere, TP = the extrcated tree points FP = not extracted tree points FN = extracted non-tree points as the tree

Study Area and Results
The efficiency and performance of the proposed algorithm are going to be assessed in four urban and sub-urban environments as follows: • USA urban environment This dataset includes about 18 million points along 250m roadside.The selected region was located in South Carolina, the USA, surrounded by many traffic signs, tall buildings, and vehicles on the whether roads or parking lots.The algorithm successfully extracted about one million points as the tree (TP + FN), where around 953 thousand points correctly belonged to the trees (TP).Lidarusa is an organization that has introduced various solutions for civil engineering and heritage mapping projects (www.lidarusa.com).Noticeably, they made diverse LiDAR point cloud datasets available to the public like Fig. 6, which has been considered to evaluate the performance of the proposed algorithm.This dataset includes about 14.5 million points along 330m roadside with various challengeable tree structures.Our algorithm respectively gained 97.7% and 99.8% at precision and recall.

•
Lidarusa railway environment Another open-access dataset at lidarusa organization was a railway environment covered with massive trees (Fig. 7).This area includes 22.5 million points at about 500m railway.As if our algorithm gained more than 98.3% accuracy in whether precision and recall, but it has shown high sensitivity to the corner of the existed building in the dataset.Tables 1 show the acquired accuracies and measured characteristic information of three sample tree points.As can be seen, the algorithm has shown acceptable performance in tree extraction.The algorithm gained an F1 score in the range of 93.8 to 98.8, which represents the efficiency and feasibility of the algorithm.Also, ten parameters ranging from planarity positioning to stem diameter have been adequately measured.

Discussion
The proposed algorithm was implemented on an ordinary computer system with the specification of Intel (R) Core (TM) i5-3210M CPU @2.50GHz, 12GB RAM, DDR 3, NVidia GeForce 2.630 GB, unlike Yadav et al. (2016) work which used a cloud computer system.
The selected regions were full of diverse challengeable, whether massive trees or other non-tree objects like tall buildings, traffic signs, or even vehicles.Likewise, one of the most positive contributions of the proposed algorithm is getting more than 97% accuracy on two different MLS platforms.Previous works rarely evaluated their algorithm's efficiency on various datasets or even on two different MLS systems (Safaie et al., 2021).
A deep learning neural network structure like PointNet++ has been introduced to tree extraction, showing acceptable performance.Unlike the mathematical-based approaches, the algorithm does not need any conversion to lower-dimensional spaces.This means that the algorithm directly consumes LiDAR point clouds to classify.Planarity location, tree height, stem diameter, maximum foliage height, trunk height, and 3D boundary are measured in tree information samples.By analyzing these calculated parameters, scientists can analyze if trees are in a standard condition or not.Also, tree encroachment in the vicinity of road structures and transmission lines has been assessed.

CONCLUSION AND FUTURE WORKS
In the last decade, Mobile Laser Scanner (MLS) system has attracted scientists' attention towards tree assessment because of recording hundreds of millions of points from a side-view look.This platform records leaves and tree trunks with sub-centimeter accuracy, resulting in measuring tree parameters.Our algorithm includes preprocessing, candidate selection, tree extraction, and tree inventory measurement.The algorithm overcomes the immense volume of MLS data by eliminating 80% of unneeded data.Also, the most challengeable objects, such as buildings, traffic signs, and pole-shaped ones, have been removed by three descriptors of Linearity, Planarity, and Verticality.The overall positive contributions of our work are (i) proposing a fast and accurate deep learning neural network structure (PointNet++) for tree recognition, (ii) Measuring tree inventory parameters such as planarity coordinate space, and lastly, (iii) analyzing the dangerous of trees from power line cables and road infrastructures.
Regarding future works, it is highly recommended to specify types of trees.As MLS LiDAR point clouds are georeferenced, researchers can use satellite datasets like Landsat to determine tree types.Also, PoinNet++ is somewhat sensitive in tree extraction, particularly building and traffic sign points.It is suggested to use post-processing methodologies to enhance the performance of the algorithm.
1) Where X, Y = trajectory data positions, L = the sum of calculated distances n = the final number of created sections Based on Shokri et al. (

Figure 2 .
Eliminating the noisy points inside each created section; (a) a noisy sample section, (b) a free noisy point section.

Figure 4 .
USA urban environment; (a) Collected LiDAR point clouds and (b) Outputs acquired by the proposed algorithm.

Figure 5 .Figure 6 .
Fig. 4 shows the collected LiDAR point clouds and acquired results in green color.The results indicate the high performance of the algorithm in tree extraction despite extracting some traffic signs falsely as the tree.The specification of the MLS system used for collecting point clouds can be found in Shokri et al. (2021a) study.USA suburban environment: This environment, like the urban one, was collected in South Carolina, USA (Fig. 5.).Here, the MLS system records around 25 million points which 4.85 million of them (TP + FP) belonged to the tree points.The proposed algorithm correctly detected 4.7 million tree points (TP) while around 10 thousand points were falsely extracted as the tree (FN).USA suburban environment; (a) collected LiDAR point clouds, (b) the output of the proposed algorithm.Lidarusa urban dataset; (a) Recorded LiDAR point clouds, (b) Output of the proposed algorithm in tree extraction.

Figure 7 .
Lidarusa railway dataset; (a) collected LiDAR point clouds; (b) Output of the proposed algorithm.

Table 1 .
The acquired accuracies of proposed algorithm.