A LINE-BASED APPROACH FOR PRECISE EXTRACTION OF ROAD AND CURB REGION FROM MOBILE MAPPING DATA

Planar structure detection from point clouds is important process in many applications such as maintenance of infrastructure facility including roads and curbs because most artificial structures consists of planar surfaces. The Mobile Mapping System can obtain a large amount of points with traveling at a standard speed. However, in the case that the high-end laser scanning system is equipped, the distribution density of points is uneven. In the point-based method, this situation causes the problem to the method of calculating geometric information using neighborhood points. In this paper, we propose a line-based region growing method in order to detect planar structures with precise boundary from point clouds with uneven distribution density of points. The precise boundary of a planar structure is maintained by appropriately creating line segments from the input clouds. We adapt the definition of neighborhood and the estimation of the normal vector to the line-based region growing. The evaluation by comparing our result with manually extracted points shows that more than 98% of curb points are detected. And, about 90% of the boundary points between a road and a curb are detected with less than 0.005 meters of the distance error.


INTRODUCTION
A recent development of three dimensional laser scanning system make it possible to obtain high density point cloud easily. The Mobile Mapping System (MMS) which is equipped with such a high density laser scanning system can measure the environment around a road precisely by simply traveling on the road at a standard speed. Therefore, the traffic regulation is not needed to measure the environments around road by MMS. This is very helpful to the survey of road environment for the purpose of maintenance and management.
An application of a large scale three dimensional point cloud for the management of the infrastructure facilities like road environments is one of the important topics. Numerous techniques are offered to utilize a point cloud for practical purpose (Mertz, 2011), (Aoki et al., 2012). In particular, the planar structure detection from a point cloud is an important technique because artificial facilities are often constituted by planar surfaces. Demantké et al. focus on facade in order to utilize point cloud data for several purposes including localization of autonomous vehicles, registration of point cloud and fine building modeling (Demantké et al., 2012), (Demantké et al., 2013). Belton et al. proposed a method for automating some of the processof road scenes including defining the road and the curbs from unorganized point data (Belton and Bae, 2010). Their method divides point cloud into a set of grid. Then, the ground points are extracted by using the linear regression for ordered vertical values of points in each grid. Then, the region growing is applied with respect to the grids in order to merge the ground points. The curb points are also defined by using information of the points in the grid. The curb is considered to exist in the the grid which includes the edge of the road region or is adjacent to the grid including the * Corresponding author. edge of the road. In such a grid, the 2D cross section of points is created and the edge of the curb is detected. The difficulty of this method is in setting of the grid size. The grid size should be large enough to contain significant number of points, but also small enough so that a very small number of salient features be present.
Eigenvalues and eigenvectors of the covariance matrix are often calculated within the local neighborhood area of the query point in order to characterize geometric property at that point. The normal vector at the point is defined as the eigenvector which corresponds to the largest eigenvalue. El-Halawany et al. proposed the method to detect road curbs from a point cloud by using such an information (El-Halawany et al., 2011). They segment ground point from a point cloud by thresholding eigenvalues and the angle of the normal vector to the horizontal plane. Other objects including walls or poles are segmented by changing thresholds.
Bremer et al. also used eigenvalues for object extraction from a point cloud (Bremer et al., 2013). They proposed multi scale feature computation method that uses two sets of eigenvalues computed from different size of neighborhood regions.
In such a method, precision of the boundary between segmented regions depends on the size of the neighborhood area. The size of the neighborhood area should be large enough so that the effect of noise is as small as possible. However, the wide neighborhood area causes the disappearance of relatively small changes in the estimation of geometric property. Furthermore, it is difficult to estimate the normal vector around the edge where planar structures are adjacent each other.
Other information defined by the neighborhood is also used. Ibrahim et al. proposed a method to extract roads from point cloud data (Ibrahim and Lichti, 2012). They use the density of points, the normal vector direction, the planarity, the elevation gradient within the neighborhood and the 3D edge detection.
On the other hand, other methods which use straight line segments as the basic element are proposed. A line segment is created from a sequence of points which is arranged in a straight line. These method are based on the idea that the points in each scan line belonging to a surface of a planar structure form a straight line. He et al. proposed the method of detecting planar structures by a line-based spectral clustering from airborne LiDAR data (He et al., 2013). In the case of airborne LiDAR data, the interest planar structure is a roof of a building. It is unclear that there method can apply to the nearly planar surface like a road which is gently curved.
For the purpose of road maintenance, there is the case that requires very high precision in some situations. For example, according to Japan Road Association, the depth of a rut is measured in the precision of millimeters. And, it is considered that a rut should be repaired if the depth exceeds a few centimeters. Recent high density laser scanning system can obtain the point cloud which satisfies the demand of such a high precision. Furthermore, structures of road environment including road and curbstone must be extracted in similar accuracy in order to automate the assessment for road maintenance. However, it is often difficult to apply existing algorithms to the measured data by the MMS equipped with the high density laser scanning system, due to the characteristics of such a data. This paper is aimed to precisely detecting the planar structure from a point cloud measured by the MMS with the high-end laser scanning system. In particular, we focus on precise detection of the boundary of planar structures and detecting relatively small planar structure like a curb.
Our approach is the line-based region growing method. In practice, roads often have a roll and there are ruts on a road. Moreover, a curb might be bent. So, the surfaces of roads and curbs are not strictly planar although they are locally considered as a planar. The region growing approach is appropriate to detect such a structure. However, the point-based region growing includes a disadvantage to the point cloud which is measured by the MMS equipped with the high-end laser scanner.
We tackle the problem to precise processing of such a data by the line-based approach. Our proposal creates line segments by creating a polyline from a point sequence of which a point cloud is segmented into scan lines. If a point sequence is approximated by a polyline appropriately, vertices of a polyline are assigned to the boundary of structures precisely. Therefore, we use line segments as elements for the region growing in order to precisely detect the boundary of structures.
The planar structure detection by the region growing needs the definition of neighborhood and the estimation of a normal vector. In the point-based approach, the neighborhood of a target point is simply defined using the distance between points. The normal vector at a point is estimated by least squares fitting within neighborhood points. However, these cannot be applied to the linebased approach directly. We adapt the definition of the neighborhood and the estimation of the normal vector in order to achieve the line-based planar structure detection by region growing in this paper.
The advantage of our proposal over ordinary point-based methods is that the boundary of planar structures is precisely kept. This is because the element for planar structure detection is the line segment which is created so that an end point of a line segment is placed at the boundary. On the other hand, it is difficult to detect boundary points by ordinary point-based methods because the normal vector which defines the direction of a plane surface is estimated from a certain range of neighborhood points.

Utilized data
The point cloud which is obtained by MMS with the high density laser scanning system share the characteristics in distribution of points. Recent laser scanning system can measure up to a million points per second. In the direction traversing a road, points are measured very dense. Actually, the interval of measrued points is often a few millimeters.
However, the interval of measured points along the direction of which the MMS travels depends on the rotating speed of the laser irradiation part and the speed of which the MMS travels. The rotation period of the laser irradiation part is much longer compared to the laser irradiation period. The measurement interval along the direction the MMS travels is often about a few hundreds of millimeters. So, the density of point distribution is greatly unbalanced with the direction (Figure 1).  This situation causes the problem to the method of calculating geometric information using neighborhood points because it is necessary to define the neighborhood range too wide in the sparse direction (Figure 2(a)). If the neighborhood range is not wide enough, the neighborhood includes points in one direction only (Figure 2(b)). For example, if the range of neighborhood is defined too wide in estimating the normal vector, we cannot estimate it precisely at the point on the planar region around the boundary of structures. Figure 3 shows an example of such a case. The estimated normal vector is tilting because the neighborhood region includes points of vertical plane even though the target point is on the horizontal plane.  shows an example of the road detection by a little larger threshold (15 degrees). The region exceeds the boundary between the road and the curb.

Line segment creation
First, we separate the input point cloud into sequences of points that represent scan lines. Each point of input point cloud contains angle information of which the laser has been irradiated. The angle increases monotonically from 0 (degree) to 360 in the same scan line. Then, the angle becomes 0 at the beginning of the next scan line. In practice, there is the case that the 0 degree point is not measured. So, let the beginning of the scan line be the point at which the value of angle decreases from the previous point.
Next, we create a polyline which approximates a sequence of points of each scan line. We use the line segment merging approach for the polyline creation. The process of the polyline creation is as follows. The points in the sequence are ordered by the value of the angle. Let the sequence of n points {p 0 , p 1 , · · · , p n−1 } be P S (0,n−1) . The initial polyline is created by connecting the point p i and next point p i+1 , (0 ≤ i ≤ n − 2). Then, we define the approximation error E (i,j) that the sequence of points P S (i,j) is approximated by the line segment pipj as the equation (1).
where, dist(p k , p i p j ) is the perpendicular distance from the point p k to the line segment p i p j .
The initial polyline is simplified by merging adjacent two line segments. In the simplification process, we select the pair of line segments of which the approximation error after merging is the minimum. The polyline simplification is performed by repeating the segment merging process until the approximation error exceeds the threshold.
Finally, we obtain polylines of the number of scan lines by applying polyline creation to each scan line. Then we use line segments which constitute polylines as a proxy of a subsequence of points that is approximated by the line segment ( Figure 5).

Line-based region growing
We apply the region growing approach to a set of line segments in order to detect a nearly planar region. In general, the region growing approach starts from selecting a seed as the first query. The query is added to the region and the neighborhoods of the query are examined whether should be added to the region. The neighborhood which is added to the region becomes the next query in order to grow the region.
In the case of the planar structure detection by the point-based region growing, a data structure like a k-d tree is used for defining neighborhood points. However, another technique is necessary to define the neighborhoods in the line-based case. Zhang et al proposed the method using the cylindrical region whose axis is the query line segment (Zhang and Faugeras, 1992). He et al. also adopted this method to define neighboring line segments (He et al., 2013). Zhang's method seems to be reasonable for neighborhoods detection. But, we have to check all line segments in the previous and the next scan lines.
We use the 3D segment tree (de Berg et al., 1997) for defining neighboring line segments. The 3D segment tree can determine the relation of inclusion and the relation of partially overlap between the intervals in three dimensional space. So, we create appropriately three dimensional intervals from line segments, and define neighborhoods of a line segment by the intervals. We use the axis aligned bounding box (AABB) of a line segment for representing the interval of the 3D segment tree. However, if a line segment is aligned parallel to an axis, its AABB is degenerated. To avoid this situation, The redundancy along each axis is added to a line segment. For the end points p i = (x i , y i , z i ) and p j = (x j , y j , z j ) of a line segment p i p j , let x min , y min , z min , xmax, ymax and zmax be min(xi, xj), min(yi, yj), min(zi, zj), max(x i , x j ), max(y i , y j ) and max(z i , z j ), respectively. And, we define the AABB whose diagonal is the line segment (x min − c, ymin−c, zmin−c) (xmax+c, ymax+czmax+c). This AABB is used for the interval of the 3D segment tree ( Figure 6). The line segment is defined as the neighborhood of the query segment if the line segment satisfies the following two conditions: 1. The AABB of the line segment partially overlaps the AABB of the query segment or is included in the AABB of the query segment. An example of defining neighborhood line segment is shown in figure 7. The red line segment is the query line segment and its AABB is drawn in red. The AABB of green line segment partially overlaps the AABB of the query line segment. Then, it is defined as the neighborhood line segment of the query line segment. But, the blue line segment whose AABB does not intersect to the AABB of the query line segment is not included in the neighborhoods. In the planar structure detection by the region growing, the difference of the angles of the normal vectors is used for examining whether the neighboring primitive is added to the region. The least squares fitting within neighborhoods is often used for estimating the normal vector. However, the area of the neighborhoods will be too wide to precisely estimate the normal vector in the situation mentioned in the section 2.1. So, we adopt the local best-fit plane of the neighborhood line segments for estimating the normal vector at the line segment. The local best-fit plane is defined as the plane which passes the query line segment and includes the neighborhood line segments most. We use the modified version of the local best-fit plane used in (He et al., 2013) in this paper.
Let the query line segment and its neighborhoods be li and N (li), respectively. The candidate plane P i,j is generated from two end points of l i and the mid-point of l j ∈ N (l i ). The line segment l k is defined as an inlier of Pi,j if the distance between the Pi,j and at least one end point of l k is smaller than a threshold and n i,j · d k is closed to zero, where n i,j is the normal vector of P i,j and d k is the direction of l k . We calculate the degree of fit as the summation of the lengths of the inlier line segments. Then, the candidate plane with the largest degree of fit is defined as the best-fit plane P i of l i . And, the normal vector n i of P i is used for the normal vector of l i in the region growing procedure. Figure  8(a) shows an example of the neighborhood line segments. And, Figure 8 shows inliers of the best-fit plane. The line segments colored in cyan are selected as the inliers of the best-fit plane from the neighborhood line segments.  The seed segment is selected by using the degree of fit mentioned above. A large degree of fit means that the target line segment is placed on the large planar region. So, we select the seed segment of the largest degree of fit from line segments that are not assigned to any planar structure yet.

EXPERIMENTAL RESULTS
We evaluate our method on the point cloud data measured by the MMS equipped with the laser scanning system Z+F IMAGER 5010. This laser scanning system can obtain about a million points per second. The laser irradiation part rotates at 50 rounds per second. So, in this case, the measurement interval along the  direction traversing a road is a few millimeters. On the other hand, measurement interval along the direction the MMS travels is about 280 millimeters when the MMS runs at 50 kilometers per hour. Figure 9 shows the input point cloud used by this experiment. This point cloud consists of one and a half million points. Figure 10 shows the result of creating line segments from the input cloud. These line segments are created by approximating a point sequence of each scan line. The approximation error is within 0.015 meters. In practical application of planar structure detection, short line segments can be eliminated from target line segments. Intuitively, short line segments tend to be placed on vegetation or small object which belong to irrelevant features. In this experiment, we filter out short line segments whose length is less than 0.05 meters.
Then, the planar structures are detected by our proposal. In this experiment, we use 15 degrees and 0.05 meters as the threshold of the difference of angle of normal vectors and the distance error, respectively. Figure 11 shows the result of extracting line segments that constitue planar structures. In the input point cloud of this experiment, there are four planar structures that represent curb regions.
We can see that all four curbs are extracted by visual evaluation. Figure 12 shows the illustration which is focused on the end of a curb. The end of a curb may become lower gradually. It can be seen that such a small planar structure is also detected.

Evaluation of precision of roads and curbs segmentation
For evaluating the precision of planar structure detection, we focus on roads and curbs. An expert extracts points of curbs manually in order to compare with points of curbs extracted by our proposal. Figure 13 shows extracted points for comparison.
For comparison with manually extracted points, we discriminate the curb and the ground region from the result shown in Figure  11. Discrimination of the ground region is based on the observation that the ground region is nearly horizontal and its height is similar to the point where MMS travels. The position where  the MMS travels is defined as the point whose angle information shows just under the MMS. Then, we define the ground region according to following two conditions: • The difference of the angle between the vertical vector and the normal vector of the least squares fitting plane is within 15 degrees.
• All differences of the height between the point under the MMS and line segments in the detected region are within 0.5 meters.
The curb region is discriminated by the following two observations. The curb region is nearly vertical. And, there are ground regions above and under the curb regions. So, the curb region is defined as follows: • The angle between the vertical vector and the normal vector of the least squares fitting plane is nearly perpendicular. The threshold is 15 degrees in this case.
• More than half of line segments include the ground segment within the neighborhood of both end points.   Figure 14 shows the result of discrimination. The ground region and the curb region are colored in blue and red, respectively.
Discrimination of curbs and roads in this experiment depends on the observation of the input cloud. The environment of curbs and roads will be different in some cases. Robust discrimination such a structure is the future work.
In our proposal, each line segment approximates a subsequence of points in a scan line. So, we can restore a subsequence of points from a line segment. As the first evaluation, we count correctly extracted points (correct), non-curb points which is extracted by our proposal (false positive) and curb points which is  not extracted (false negative). An example of the evaluation result is illustrated in Figure 15.
The statistical evaluation of extracted points is tabulated in Table  1. More than 98 % of curb points are extracted by our proposal. And, the missing rate is 1.71 %. Most points of relatively small planar structures like a curb can be extracted by our proposal. Secondly, we evaluate the precision of detection of the boundary between a curb and a road by the distance error. The boundary points between a curb and a road are also extracted by a expert. And, we regard the point with lower height of a line segment representing a curb as the boundary between a curb and a road.
An example of manually extracted boundary points and boundary points extracted by our proposal is shown in Figure 16. Usually, there are two boundary points in a scan line. One is the left side of a point of which MMS travels. And, the other is on the opposite side. We associate the manually extracted boundary point with the boundary point extracted by our proposal.
Then, the distance error between corresponding points is evaluated. In this evaluation, 234 points are manually extracted for comparison. The statistical evaluation of the boundary points extraction is tabulated in Tables 2 and 3. Average of the distance error is 0.004 meters and median of the distance error is 0.0022 meters. From Table 3, about 90 percent of boundary points were detected with the precision of 0.005 meters by our proposal.
The maximum distance error is 0.128 meters. Figure 17 shows the part where the distance error is large. In this figure, the surface of the road near the curb is dented. The planar structure detection tend to fail in such regions. However, In the case that the surface is not deteriorated, our proposal can detect planar structures with precise boundary.

CONCLUSIONS AND FUTURE WORK
We presented the method for planar structure detection with precise boundary from a point cloud which has uneven distribution density of points. In the case that points measured by the MMS which is traveling at standard speed, the distribution density of points changes with the direction. The unevenness of point density can be avoided by slowing down the travel of the MMS. However, the utilization of the MMS at a standard speed is the advantage of the MMS. In such a condition of point clouds, a point-based approach has the problem that the neighborhood area has to be set too wide. A line-based approach can keep the precise boundary of structures by creating line segments appropriately approximating input points. We created line segments by creating a polyline approximating a sequence of points of each scan line. Moreover, we adopted the 3D segment tree and the best-fit plane in order to adapt the definition of the neighborhood and the estimation of the normal vector for the line-based region growing.
The proposed method was evaluated by comparing with manually extracted curb points. More than 98 % of curb points are correctly extracted by our proposal. Furthermore, about 90% of the boundary points between a curb and a road are detected with less than 0.005 meters of the distance error. The distance error at the location where the surface is deteriorated tends to be large. But, the boundary points of structures without breakage have been detected precisely. In this paper, we evaluated the curb points only. But, other planar structures will be also detected as well.
Automatic reconstruction of three-dimensional models including buildings, roads and curbs is one of the future works. Furthermore, we aim to utilize reconstructed models for the management of the infrastructure facilities.