SMART FUSION OF MOBILE LASER SCANNER DATA WITH LARGE SCALE TOPOGRAPHIC MAPS

The classification of Mobile Laser Scanner (MLS) data is challenging due to the combination of high variation in point density with a high variation of object appearances. The way how objects appear in the MLS data highly depends on the speed and orientation of the mobile mapping platform and the occlusion by other vehicles. There have been many approaches dealing with the geometric and contextual appearance of MLS points, voxels and segments to classify the MLS data. We present a completely different strategy by fusing the MLS data with a large scale topographic map. Underlying assumption is that the map delivers a clear hint on what to expect in the MLS data, at its approximate location. The approach presented here first fuses polygon objects, such as road, water, terrain and buildings, with ground and non-ground MLS points. Non-ground MLS points above roads and terrain are further classified by segmenting and matching the laser points to corresponding map point objects. The segmentation parameters depend on the class of the map points. We show that the fusion process is capable of classifying MLS data and detecting changes between the map and MLS data. The segmentation algorithm is not perfect, at some occasions not all the MLS points are correctly assigned to the corresponding map object. However, it is without doubt that the proposed map fusion delivers a very rich labelled point cloud automatically, which in future work can be used as training data in deep learning approaches.


INTRODUCTION
Mobile laser scanner (MLS) data consists of a huge collection of points captured from a streetwise perspective. Points are reflected on various types of objects such as roads, building facades, trees, street furniture but also cars, cyclists and pedestrians. Filtering, data reduction and classification are important tasks when dealing with MLS data. The main challenge in both science and practice is to rapidly get useful information out of the point cloud. Che et al. (2019) presented a comprehensive overview on existing object recognition and classification algorithms. Object recognition mainly deals with road surfaces and boundaries, plus the wide range of approaches dealing with pole like objects (Guan et al., 2016;Li et al., 2018;Yan et al., 2017;Wang et al., 2017;Fukado and Masuda, 2015). Che et al. (2019) divide the classification literature into three categories: point based, segment based and object based classification, where the object based classification is considered as an extended segmentation approach. Point based methods calculate features and relations from point neighborhoods (Weinmann et al., 2015), and use descriptors in supervised classification algorithms, like in Luo et al. 2018. Benchmark datasets such as Paris-Lille-3D (Roynard et al., 2017) are available to test the performances of classification algorithms. One of the common problems in processing MLS data is the high variation in point density and the high variability of object appearances due to the dynamic environment during the data acquisition. Hackel et al. (2016) dealt with varying point densities by presenting a down sampling and pyramid building strategy, which eased the neighborhood selection and the time efficiency. Wang et al. (2018) voxelised the MLS points before feeding into a deep learning multiscale network. Typical accuracies for object recognition and classification are around In this research MLS data is fused with a 2D large scale topographic map to support the labelling process. Main research challenge described here is the correct fusion of 2D map data with 3D point clouds, i.e. to correctly assign laser points to map objects. Next to potential registration problems, challenges may occur at situations where objects in the point cloud are not represented by the map, e.g. cars and pedestrians. The main aim is to automatically label the MLS points following the map classes for corresponding situations and label them as unknown if there is not a logical correspondence between map class and the nearby MLS points. Next to a labelled MLS point cloud, this procedure results in discrepancies between map and MLS data, which can directly be used for map updating. Additional benefit of such a map based labelling is the automatic generation of detailed training samples which can be used as input in a deep learning classification network. After all, in cases where the map data is not available or out of date, a supervised classification algorithm is needed to classify the MLS points.
In this paper the focus is on the design of a method that smartly fuses laser points with their corresponding map objects, by looking at semantic characteristics of the objects.

What is meant by smart fusion
The term smart can be interpreted in various ways, so in the section it is explained what is meant by smart fusion. The basic fusion principle is based on spatial fusion two datasets in the same coordinate system. Based on the x-and y-location of the laser points, a point is assigned to a corresponding map object. For most of the points and polygon features, this can be done by a point-in-polygon operation. This works well if the registration between two datasets is perfect, and if there is no difference in the type, amount and appearance of objects in both datasets.
However, in reality there are several causes why a basic fusion will not work properly: -The registration between both datasets is not perfect; errors in the position of MLS data can easily reach 1.0 m caused by inaccuracies of the platform. Next to that, the map accuracy is in the order of 0.1-0.2 m. -Objects in the map have another appearance in the point cloud, e.g. a traffic light is a point feature in the map but a cluster of points in the point cloud. -Objects may have been recorded in the point cloud which do not exist in the map, like cars and pedestrians. -Objects may have been changed between the acquisition of the map and the point cloud.
We consider our algorithm smart if it deals with registration errors up to 1 meter, differences in type, appearance and amount of objects in both datasets. Underlying assumption is that the map delivers a clear hint on what to expect in the MLS data, at its approximate location. In order to achieve this, we make use of logical relations between map objects and the corresponding appearance in the point clouds. The relations depend on the class and type of the map objects. Basically there are three object types present in the map: polygonal, linear and point features. Polygonal features include land cover classes such as buildings, roads, terrain. Examples of linear features are power lines and rail way tracks. Point features include pole like objects, single trees and utility service assets such as mail boxes. In this paper the focus is on polygon and point features, as line features as power lines were not present in the area of data acquisition.
For objects which are represented by a point in the map, e.g. a lamp post, single trees, the fusion cannot be achieved by a pointin-polygon procedure but is mainly based on vicinity criteria. The research problems there are the segmentation of corresponding MLS points to a single map points and if multiple point objects are close to each other, one has to carefully assign laser points to the correct corresponding map object. Figure 2 shows the workflow of our fusion algorithm for map polygons and map points at one side with MLS points at the other side.

Filtering MLS data
Before starting the fusion process MLS data is processed to remove the largest outliers and to distinguish between ground and non-ground points.

4.2.1
Outlier removal: for various reasons several laser points are recorded below or above the surface without actually representing a point at an object or surface. Typical property of these points is that the local point density is very low compared to other locations. Determining the number of neighbouring points within a small sphere, e.g. with radius 10 cm, gives enough evidence to decide whether a point can be considered as outlier or not. Some points at the outer crown of trees may be incorrectly considered as outliers in this step; this can easily be repaired later when it is known which components do belong to trees. Any nearby filtered point are added to the tree component.

4.2.2
Ground and non-ground filtering: filtering points into ground and non-ground has been an active research topic for Airborne Laser Scanner (ALS) data. For filtering MLS data many researchers use existing algorithms, originally built for ALS data. Our filtering algorithm is based on a constrained region growing algorithm. The constraints come from the point attribute "height above local minimum". Segments that are larger than 1 m 2 with points within 5 cm of its local minimum are considered to be ground. Typically the segments with points within 5 or 10 cm above local minimum represent the curb stones, and edges of traffic islands. If there are no nearby other points higher than 10 cm above the local minimum, these points are added to the ground dataset. Otherwise these points likely belong to vegetation or other non-ground objects.

Fusion of polygonal map objects with MLS data
First, the polygon objects are fused with the MLS data. Map polygon features represent a surface area, such as part of a road, sidewalks and buildings. For the majority of these polygons it is clear whether the surface area belongs the ground or corresponds to above ground objects, and therefore it is clear to fuse the polygon with either ground or non-ground MLS data. Also, basic assumptions on the appearance of the MLS data can be applied per class to guide the fusion process. For example, for road polygons the algorithm is searching for ground points on smooth surfaces within or close to the polygon. For building polygons, the corresponding mobile laser points are likely to be on the façade, just in or outside the polygon. For classes such as playgrounds, forests and agricultural fields it is expected that corresponding MLS points come from both ground and nonground data.
Due to the unknown registration errors between the two datasets, points near the polygon boundary may belong to objects at either side of the boundary. In cases where the boundary represents a difference between ground and non-ground objects, it is straightforward to assign the corresponding points to the polygons by adding a search buffer to the polygon, based on the knowledge that nearby points should be selected from either the ground or non-ground dataset. For building polygons a buffer of 1m was generated which result in the addition of any non-ground MLS point within 1 m of the building. Additionally, these points are further refined by keeping only points on large or very planar surfaces.

Object segmentation of MLS data
Non-ground MLS points which are not assigned to polygon objects are considered to be potentially belonging to map point objects.
The goal of this object segmentation step is to group MLS points which belong to a single map object. Segmentation is crucial for a proper fusion between MLS data and map point objects, e.g. single trees and traffic signs. The reason is that in the map it is represented by a single 2D point, where the corresponding MLS points are a collection of 3D points. Initially, both ground and non-ground datasets were segmented by a constrained connected component algorithm as described in section 2.2, resulting in segments that either belong to ground surfaces, or objects on top of the surface. For the objects on top of the surface as refinement step is needed to aim for a segmentation into individual objects.
A proper object segmentation depends on how well the object is represented by the data in relation to the segmentation strategy. Often, the problem is that the best segmentation strategy depends on the type of the object, in other words, segmenting a tree may need another algorithm than segmenting a building façade or road surface. The advantage of working with a large scale topographic map is the ability to incorporate knowledge on what objects to expect in the laser data and therefore to adapt the segmentation strategy to the situation Though we do not expect that the map data is perfect, it is taken as approximate and initial knowledge on the type of objects. This is achieved by the following procedures: 1. Map analysis phase for map point objects. Each map point object is analysed on the proximity of other map points. What is the distance to the nearest point object and what is the class of that nearest object? This step is necessary to gain information on how many other objects to expect in the direct neighbourhood. This information is used locally when setting the growing distance and maximum size of a segment. 2. If map point objects are close together, we can expect that the objects are also captured close together in the MLS data. Map points within 1.5 meter from each other are taken as a group for which the task is to correctly assign the map point to the corresponding MLS object. That means that attention should be given to a proper separation of the segments, and a careful matching is necessary as the closest correspondence may not be the correct one. 3. Near the location of point objects, seeds are sought that fulfil the basic assumptions of that object. If a seed is found, it is grown into segments using the parameter settings set in the map analysis step. This step is further explained in section 2.5. 4. If no seed is found, the map object may not exist at that place, it may not be captured in the MLS data or the basic assumptions were not valid. In all cases the map point is flagged as not found in MLS data. 5. MLS points which are still not yet assigned to map objects, are analysed further. Many of the points are non-ground points above road polygons. Typically, these points are reflected on cars, pedestrians and cyclists and other components which are located on or next to the roads. The latter ones are potentially new objects which did not yet exist in the map.

Fusion of map point objects with MLS data
In this section the focus is particularly on the assignment of MLS points to nearby map points. From the map point analysis phase it is known what is the distance to the closest map point. This distance is stored per map point and is used to select the correct segment procedure and to select nearby MLS points. In the example shown in figure 3, a situation is given which shows that several map point objects are within 1.5 meter from each other, and other map points are further apart. The group of points closer than 1.5 meter to each other are treated as group, meaning that the corresponding MLS data should be carefully segmented and assigned to the correct map point.
For the pole like objects such as traffic signs, street lights and trees, the seed is sought on the lower part of the object which mostly contains most of the points on the pole structure. First, MLS points are selected a bit above the ground near the map point. This is accomplished by first selecting points between 0.5 and 1.5 meter above the lowest point in the neighbourhood. The reason to ignore the lowest 50 cm at this stage is that it often contains noisy objects and low vegetation, leading to false seed selections. For the selected nearby points the distances in X,Y between map points and each of these MLS points are calculated and stored as point attributes. These point attributes are used as input in a constrained connected component algorithm where the point attributes ensure that only points with similar dX and dY values are grouped together, thus grouping points on vertical poles. These corresponds to the seeds on pole like objects. The largest component is selected as seed segment. In case of analysing n multiple objects as a group, the largest n components are selected. The relative orientation between the map points is used to assign the corresponding MLS segments, as the nearest segment may not be the correct one.
For segmenting MLS points nearby map points representing volumetric objects such as electricity boxes, fixed garbage bins, the seeds are sought just above the ground without the constraint of a pole like shape.
After the seed selection stage, it is known whether a seed is found for a map point or not. Map points without a seed, e.g. the most right pole object in figure 3, are labelled as 'not matched' with MLS data. These points are either outside the MLS dataset or these objects have changed. After the seed selection and assignment the task is to grow the seeds by adding points that belong to the same object. The growing parameters also depend on the map class to ensure a class dependent segmentation. In figure 4 the growing phases are explained for poles and trees: first the segment is grown in vertical direction using only points within 1 meter radius of the map point. Secondly, points which are further away are considered to be added in a iterative growing stage. Points are added if they are within growing radius distance. This distance is iteratively increased to first add points which are tightly 2 www.pdok.nl connected to the segment. This enables the growth of segments into the space between two objects, ensuring the assignment of points to the segment which has the lower point spacing between the points. To avoid segments grow too large, e.g. when points on fences join the segment, the maximum range of the segment is limited to the distance to the nearest map point object. Trees are grown following the individual tree segmentation approach as mentioned in Gorte et al. (2015).

5.1.1
Mobile laser data: The mobile laser scanned data was captured by a TopCon IP-S3 scanner, collecting data with a rate of 700k pulses per second. Data was recorded in Rotterdam, The Netherlands. Point density can be considered rather coarse for MLS data with 300 points per square meter below the sensor to 10 points per square meter on the ground surface at 30 meter distance from the sensor. From this dataset two subsets are cropped into datasets of 4 and 10 Million points. In these datasets the planimetric accuracy of the MLS has been checked on GPS reference points. RMSE results between GPS measurements and corresponding MLS points were between 5 and 120 cm. For the selected datasets, the planimetric differences between MLS and map data is estimated to be around 50 cm.

Large scale topographic map:
The large scale topographic map with a scale of 1:1000, was retrieved from a national Spatial Data Infrastructure, called PDOK 2 . According to the national standards, the geometric accuracy of the map is required to be better than 20 cm standard deviation in planarity. Various types of road classes, such as bicycle lanes, highways and secondary roads are grouped into a single road class. Several infrastructural objects like lamp posts, traffic signs and traffic lights are grouped into infrastructural poles. The following classes have been selected: buildings, terrain, road, water, bridges for the polygon classes and trees, poles and boxes such as utility boxes for the object classes.

Polygon based classification results
Polygons are fused with the MLS data in a workbench implemented in FME from Safe Software. The workbench is setup once in such a way that the polygons representing the terrain are fused with the ground MLS points, and the building polygons are fused with the non-ground MLS points. The building polygons have been buffered by setting a single parameter on 1 m. The output of this procedure are two point clouds: one with points assigned to polygons, and one with the remaining points.
The first contains the polygon based results where both ground and non-ground MLS have been fused with the corresponding map objects; the file contains 3D point coordinates plus object_id and class_id. The fusion process takes about 2 minutes for the fusion of a stretch of about 300 meter MLS data. This step can be enhanced further as the map data is read in tiles of 1k x 1k meter, meaning that many of the polygons do not contain any MLS point. The second output file contains non-ground MLS points which do not correspond to polygon objects, representing points on single trees, street furniture and points reflected on cars, pedestrians etcetera. This file will be input for the object based classification process.

Object classification by map point induced segmentation
The object classification consist of the seed selection and growing of matches between map points and MLS data. Results are shown for the seed selection, and the following growing stages of segmentation and classification in figures 7, 8 and 9.

Detected map changes
One of the major advantages of the smart fusions between map and MLS data is the automatic detection of map points that do not correspond to a MLS seed. Our algorithm produces a side product indicating whether a map point is matched to a MLS segment or not. The changes between map data and the MLS data are analysed to highlight the potential impact of the approach for map updating. First the map point objects which did not produce a corresponding segment in the MLS data are shown as purple dots (emphasized by the red arrows) in figure 10. Green dots are the map point objects which successfully generated a segment. Table 2 shows that for many of the map point points without MLS segment, there was no overlap in the data due to occlusions and distance to the mobile mapping sensor. For other missing objects in the overlapping area, it seems that the actual object has been moved by a few decimetre or even several meters, but is still stored twice in the map data with a slightly different location. It is of high interest for map owners to directly see which objects may have moved and/or stored twice.   . Several detected changes between map and MLS data in dataset 2, of which 1 (cyan box) missed the match because of having too few points in the seed. Figure 11 shows a case in the cyan box where there is no match found between an box type of object and MLS points. Though it is not clearly a box in the MLS points, it looks like a pole like object which was not matched to the map point. Feedback is needed from the map owner to check whether the map is incorrect or whether the MLS data was just too sparse to find a seed.
Next, MLS points are analysed which are not assigned to map points. Note that these points also were not assigned to polygon objects in the earlier stage. Typically these points belong to cars, pedestrians but also trees which were not represented in the map. In addition, points on extrusions of buildings are present in this remaining points, as the building polygons represented the footprint of the building at the location where the wall touches the ground. Our algorithm now uses the information from the polygon based classification to further classify the remaining points. So, in this step the remaining points near MLS points classified as buildings are assigned to buildings, remaining points directly (less than 2.5 meter) above the road are assigned to potential cars and pedestrian class. In figure 12 it can be seen that points on extruded buildings are assigned to buildings, cars and other objects above the road are grouped into a separate class (blue in figure 12). Note that new objects which should be updated in the map are also in this class. Figure 12. Remaining points near buildings are automatically assigned to buildings (red), objects above the road including cars, pedestrians (blue), points above terrain polygons (yellow).
The MLS points in the remaining class just above the road belong to either objects which were not present in the map, or these points did not comply to the assumptions that were made on seed selection and growing. These points are further analysed in section 3.5 and 4.

Analysis of segment growing results
For groups of nearby map points the results are not satisfactory in situations as shown on the left in figure 13. In our current implementation, we restricted the growing range if two pole like objects are within 1.5 meter distance. Part of the horizontal pole with traffic lights are not grown into the segment. This could easily be solved afterwards by assigning the remaining unsegmented points to the nearest segment. On the other hand, one should be careful in just assigning points to the nearest segment, see the example on the right. A situation is shown where two tree map points resulted in two segments (blue and green). The two trees on the right were not in the map, so it was good that the green segment did not grow too far. Still, we see that some points which should belong to the blue segment were not assigned correctly. Figure 13. Points not segmented due to limiting growing constraints which is in some occasions not correct (red circles), and in some correct (green circle).
Towards the outer side of the MLS datasets, the point density gets lower and the occlusions per object are getting more severe. In those areas, our region growing algorithm assigns the MLS points within the maximum range assigned to each map point, but it lacks the interpretation skills to understand which point belong to which object. Therefore it is advised to add a post processing step to check whether the shape of the segment complies with the expected shape, which until now we did not want to enforce. It is in our opinion that the expected shape should not be built from heuristic rules, but should be learned from a huge collection of training samples. So our next aim is to classify the MLS data using a supervised classification approach and check whether that class corresponds to the map based classification. This is part of future work of the authors.

OUTLOOK
The main contribution of this research is not so much the classification of the MLS data but to perform a rapid change detection to an existing topographic map at the areas where the MLS data has been captured. After all, it directly becomes clear where there is not a logical match between MLS data and the map data. Another application is the possibility to use geometric differences between both datasets to enhance the quality of the MLS dataset: differences at pole locations can potentially be used to adjust locally the trajectory of the MMS platform similar to using differences between aerial images and MLS data as shown in (Hussnain et al, 2018).
However, to the authors opinion the largest potential of this approach is to generate a huge amount of training data for deep learning algorithms, especially when a selection step is added which filters out the erroneous training samples. Even more details can be transferred from map to MLS data, such as functional use and age of objects. Also relations between objects can directly learned from the map instead of contextually derived from MLS data. The automatic transfer of knowledge from a semantically rich data source as the map to a MLS dataset is crucial in an era where algorithms are supposed to learn to understand the scene.

CONCLUSIONS AND FUTURE WORK
In this paper a method is presented that automatically transfers detailed map information to mobile laser scanner data. We have shown that it is possible to segment and classify the MLS data based on what to expect from the map. Fusing MLS data with topographic map information makes it possible to locally process the laser data depending on the class type. The segmentation is necessary to group all MLS points belonging to a certain object. We considered this to be a smart fusion as the segmentation parameters depend on the corresponding map class. Changes between objects in the map and MLS data can be found if there is no evidence for the existence of the map object in the MLS data. In over 90% of the missing matches, the reason is that the object has changed, in about 10% it did not became clear whether the object was still there or not. The potential of the map fusion is enormous as it does not only transfer the map class, but also at an object instance level.
In future research training samples are generated by the fusion with a large scale topographic map. A classifier is trained using these samples. As our current approach is automatically generating these samples for every location with coverage of both map and MLS data, it potentially collects a large amount of training data. This enables the training of a deep learning network for classification of MLS data. The classification is then performed on data without topographic map. The challenge is find a procedure to select on optimal set of training samples to correctly classify a MLS dataset.