AUTOMATIC EXTRACTION AND RECOGNITION OF ROAD MARKINGS BASED ON VEHICLE LASER POINT CLOUD

At present, automatic driving technology has become one of the development direction of the future intelligent transportation system. The high-precision map, which is an important supplement of the on-board sensors under the condition of shielding or the restriction of observation distance, provides a priori information for high-precision positioning and path planning of the automatic driving with the level of L3 and above. The position and semantic information of the road markings, such as the absolute coordinates of the solid line and the broken line, are the basic components of the high-precision map. At present, point cloud data are still one of the most important data source of the high-precision map. So, how to get road markings information from original point clouds automatically deserve study. In this paper, point cloud is sliced by the mileage of the road, then each slice is projected onto respective vertical section. Random Sample Consensus (RANSAC) algorithm is applied to establish road surface buffer area. Finally, moving window filtering is used to extract road surface point cloud from road surface buffer area. On this basis, the road surface point cloud image is transformed into raster image with a certain resolution by using the method of inverse distance weighted interpolation, and the grid image is converted into binary image by using the method of adaptive threshold segmentation based on the integral graph. Then the method of the Euclidean clustering is used to extract the road markings point cloud from the binary image. Characteristic attribute detection is applied to recognize solid line marking from all clusters. Deep learning network framework pointnet++ is applied to recognize remain road markings including guideline, broken line, straight arrow, and right turn arrow.


INTRODUCTION
Image processing is applied to get road markings in early studies Chen et al., 2015;Rebut J et al., 2004;Noda et al., 2011;Foucher et al., 2011). However, the changes of imaging environment such as tree shadow, car shelter, will cause difficulties to the image processing inevitably. Road marking extraction based on laser point cloud depends upon the intensity differences of the intensity of point clouds. Road marking has higher reflection rate compared to the asphalt. Therefor, road marking can be extracted based on intensity differences between road marking and asphalt road surface (Yao et al., 2018;Yang et al., 2012). A. Hervieu, B, et, al. (2015) use ortho-image which is computed by vertical projection of georeferenced point clouds to extract road markings. Guan et al. (2014) extracted road marking based on the point cloud density differences. Kumar et al. (2014) used road surface position to figure out segmentation threshold between road marking and its surrounding road surface.  adopted self-adaption binary segmentation to extract road marking form raster image, and use deep learning to recognize road marking. But this literature only studies pedestrian crossing and arrow marking. Yu et al. (2015) divided point cloud into several parts along the vertical direction of the car driving trace, then Otsu threshold segmentation was applied to extract road marking in each part. Guan et al. (2015) firstly utilized inverse distance weight interpolation to transform point cloud into raster image, then Weighted neighborhood difference histogram was applied to determine intensity segmentation threshold so as to transform the raster image into binary image, finally two dimensional multiscale tensor voting was applied to extracted road marking from binary image.

Corresponding author
In this paper, point cloud is sliced by the mileage of the road, then each slice is projected onto its own vertical section. Random Sample Consensus (RANSAC) algorithm is applied to establish road surface buffer area. Finally, moving window filtering is utilized for the sake of extracting road surface point cloud from road surface buffer area. On this basis, the road surface point cloud image is transformed into raster image with a certain resolution by using the method of inverse distance weighted interpolation, and the grid image is converted into binary image though using the method of adaptive threshold segmentation based on the integral graph. Then the method of the Euclidean clustering is used to generate road marking clusters from original road marking point cloud which are restored from binary image. However, these road markings clusters still does not include any semantic information. For the purpose of getting semantic information of these road marking clusters, characteristic attribute detection is applied to recognize solid line marking from all clusters. Deep learning network framework pointnet++ (Charles R, et al., 2017) is applied to recognize remain road markings including guideline, broken line, straight arrow, and right turn arrow.

METHODOLOGY
In this paper, the process of extraction and recognition of road markings are divided into three steps: road surface point clouds extraction, road marking point clouds extraction and road marking point clouds recognition. The flowchart of the proposed method is shown in figure(1). (1) Suppose that the corresponding coordinate azimuth of the point equals to , and the length of each slice along the road direction equals to D, then the mileage of the slicing point would be equal to ( − 1) + 2 . Suppose that ( ′ , ′ ) represents the position of the slicing point, then the coordinate of the point can be determined by the equation (2).
where Δ = min {| − ( − 1) − 2 | | = 1,2,3 …} The parameter Δ represents the distance between slicing position and the nearest point adjacent with the slicing position on the trace(see figure(2)). The subscript 0 is determined by the parameter Δ . This paper utilizes the weighted sum of the two points which are adjacent to slicing point on the trace to calculate the azimuth of the slice point, which is determined by the equation (3).
where Δ = 0 − ( − 1) − 2 ( ′ , ′ ), ′ , and determines the position of the slice, the direction of the slice and the length of the slice respectively. For the purpose of slicing the point cloud into several slices, the distance between point cloud and slicing section has to be calculated. When the distance is less than half of length of a slice, the point clouds are bound to lie on the same slice. As is shown in figure(3), The symbol represents the projection of the slicing section on the XOY plane. The symbol represents the normal of the slicing section. The symbol represents the coordinates of some point cloud. The parameter represents the distance between the slicing section and the point cloud. Then the equation(4) can be determined by relationship between the vector ⃗⃗⃗⃗⃗⃗⃗ and the normal vector . The slicing result of the original point clouds is shown in figure(4) when the length of a slice is assigned to 2m.

Slicing projection
Considering the trace direction is parallel to the road direction at most time, this paper used trace section to replace the road section. Suppose that the point ( , , ) represents point cloud which lie on one slice, ( , ) represents the coordinate of the slicing point, and the parameter represents the azimuth of the trace point(see figure (5)), then the coordinate of the point ( , , ) after projection can be determined by the equation (5).
For the purpose of convenience of processing point clouds, the three-dimensional point ( , , ) need to be transformed into two-dimensional point ( , ). Suppose that (min ( ), min ( )) represents the origin of the twodimensional coordinate system, Then ( , ) can be determined by the equation (6).

The Establishing of Road Surface Buffer Area
This paper establishes section buffer area for the purpose of filtering the noise points such as trees, building and road lamp. Firstly, pseudo scan points are generated by dividing points on the projection plane into several small intervals which are [ 0, W), [W, 2W), [2W, 3W) … , [(k − 1)W, kW) according to designative interval W. Suppose that ( , ) represents the points which lie in the k-th interval, then ((k − 1)W, min ( )) is the coordinates of pseudo scan points generated from the k-th interval. The RANSAC algorithm is applied to detect straight line . Finally, the points which lie in buffer area can be extracted by establishing buffer area of the straight line. The straight line is assigned to be the center of the buffer area. Assign the parameter W to be 0.05m, then the buffer area of the straight line is shown in figure (7).

Moving Window Filter
This paper uses moving window filter to crop point clouds which lie inside the buffer area. As is shown in figure(8-a). Firstly, the point clouds are divided into several intervals according to a certain interval . The minimum bounding rectangle of the point clouds which lie inside each interval need to be calculated. Let Rect (ℎ , ℎ ) represent the minimum bounding rectangle, and ℎ 、 ℎ represent the elevation of the higher edge and the elevation of the Rect respectively. Let ℎ represent a certain threshold value, when the equation (7) is satisfied, Rect is marked as the key window(the window which is marked as red color in figure (8)).
Secondly, the key window need to be cropped for the purpose of extracting road surface point clouds. There are two situations when the point clouds are cropped. One is that the key window such as Rect 1 、 Rect 9 , lies on the edge of one section, as is shown in figure(8-a). Take Rect 1 as an example. Suppose that point set Set 1 ( , ) are included in Rect 1 , and 1 ′ ( , ) represents a subset where the points satisfied the equation > ℎ 2 , then let the crop boundary to be ( = ( )), when the points in Set 1 satisfy the equation (8), the points are recognized to be road surface point clouds, as is shown in figure(8-b).
The other situation is the key window lie in the interior of the road. Take Rect 5 as an example. Suppose that the point clouds of Set 5 ( , ) are included in Rect 5 , then the point clouds which satisfied the equation (9) are recognized as road surface point clouds, as is shown in figure(8-c).

Road Surface Point Cloud Rasterization
This paper uses Inverse Distance Weighted( IDW) to transform the three-dimensional point clouds into the two-dimensional raster image. Let ( , , , ) represent the point cloud, where is the intensity of the laser reflection. Let represent the resolution of the raster image. Let represent interpolation radius. When the point is projected onto the two-dimensional plane, the corresponding point after projection is ( , , ) . When the following four equations = ( )， = ( ) ， = ( ) ， and = ( ) are satisfied, the size of the raster image is to be determined by the equation (10).
The point clouds( ) which lie interior of interpolation radius are determined by the position of the interpolation point ( , ) and the interpolation radius(see equation (11)  2 )) 2 Figure 10. Inverse distance weighted interpolation ( , , ) represents the distance between the point and the pixel center ( , ) . Then the intensity value of the interpolation point is determined by the equation (12), where 1/ ( , , ) is assigned to be the weight.

The Adaptive Threshold Segmentation
The adaptive threshold segmentation based on integral image is proposed by Bradley and Roth (2007). The integral image refers to the image that each pixel value is the sum of top left rectangular area of original image and it is convenient to calculate pixel value sum of arbitrary rectangular area of original image by integral image. The adaptive threshold of each pixel is determined by the average of its surrounding pixels' grey value. Gaussian filter is applied to process intensity image, and let the original pixel be ( , ), corresponding grey value be ( , ) and integral image be ( , ) . We define two parameters being background radius and being segmentation coefficient. Then the sum of the grey value of pixels which lie in a rectangular area, the centre of which is and the length of which is 2 + 1 (see Figure 12), is calculated as follows: The image boundary shall prevail when the rectangular area is beyond it. The binary image is as follows: We selected four segments of 60 m data samples and generated intensity image with the parameter being 0.15 m and the parameter being 0.05 m (see Figure 13). Suppose that TP(true positive), FN(false negative), and FP(false positive) represent the counts of the pixels which are correctly, defectively, redundantly extracted respectively in the process of automatic extraction of road markings. The manual annotation result is assessment reference (Yao et al., 2018) and (Cheng et al., 2017) is calculated by using the method proposed by Cheng. The intensity image is processed with adaptive threshold method by different parameters of s and t according the equation(16~18). Figure 14 is the extracted marking of samples point cloud data with their own max . Table 1 is the evaluation of extracted marking of samples.  Table 1. Evaluation of extracted marking of samples

Point Cloud Euclidean Clustering
Road surface point clouds can be restored from the binary image according to the foreground pixels. Point cloud clustering means that the points which belong to the same category(though the category may be unknown after clustering) are clustered into one cluster according to a certain principle. This paper use Euclidean clustering to divide the point clouds into different clusters based on the different Euclidean distances among point clouds. Let = { ( , )| =1,2,3…} represent point set which are to be clustered, parameter D represent the cluster radius. The concrete steps of Euclidean clustering is as follows: (1). Take any point from the point set , and put the point into the point set cluster.
(2). Let C represent the circle, the center of which is the point , and the radius of which is D. Put the points which lie inside the circle C into the cluster defined by the step one(see figure 15-a). (4). Start a new cluster by repeating from step one to step three using remain points which come from the point set (see figure 15-d).
When the parameter D is assigned to be 0.5m, the clustering result of the point clouds of figure (14) is shown in figure(16).

Road Marking Semantic Recognition
Separate cluster is generated after the discrete point clouds are clustered into point cloud cluster. The minimum bounding rectangle(MBR) of each cluster need to be calculated in order to get some attribute characteristics of each cluster. Solid line can be detected from all the clusters according to one or more attribute characteristics of the MBR of each cluster. Other road markings such as broken line, guideline, straight arrow, and right turn arrow can be recognized by using the deep learning network framework pointnet++.

Solid line Recognition:
Suppose that represents the longer side of the MBR, and represents the shorter side of the MBR. In order to reveal geometric characteristics of the marking cluster, some attribute characteristics of the MBR of each separate cluster are defined as follows: Length-width ratio( ). The length-width ratio reflects how similar the MBR is to the square. The parameter is determined by the equation(19). = When the parameter and the parameter is assigned a certain threshold value ( > 7m, > 20), the solidline can be detected from all the extracted clusters(see figure (17)).

Recognition of Other Road Markings:
The solid line can be discarded after it was recognized. The remain clusters here included guideline, straight arrow and right turn arrow. In early studies, Most of deep learning network framework in the realm of point clouds like pointnet++, pointcnn (Li et al., 2018), KPconv (Thomas et al., 2019), et, al., were applied to indoor point clouds. There were also some network framework which were used in outdoor point clouds for the purpose of semantic recognition of the outdoor objects such as trees, buildings, and road lamp. But there were seldom network framework which were applied to semantic recognition of the road markings. In this paper, The deep learning network pointnet++ was used to recognize remain clusters. The result of recognition was shown in figure (17). Different kinds of road markings were marked as different color. For example, all the broken lines which were correctly recognized were marked as blue color. There was one broken line which was erroneously recognized to be straight arrow. We can see that all the straight arrow and one right turn arrow was correctly recognized. One guideline was correctly recognized. From all separate broken lines, only one broken line was erroneously recognized. Some broken lines which lie very close to the solid line were erroneously recognized to be solid line. The reason was that when the method of Euclidean clustering was applied to discrete point clouds, these broken lines which lay very close to the solid line were clustered together with solid line. For the purpose of evaluating the proposed method in this paper quantitatively, 5km of the road were selected. The detected length of the solid line was 3.6km, So the detection rate was 0.72. The evaluation of other kinds of road markings was listed in table2.

CONCLUSIONS AND DISCUSSIONS
This paper extracted road surface point clouds through slicing original point clouds into separate slice along the road direction.
IDW was used to transform road surface point clouds into road raster image with three-dimensional coordinate and the intensity of the point clouds. Then Adaptive threshold segmentation was applied into the raster image for the purpose of transforming raster image into binary image. After we get the binary image, Euclidean clustering was utilized to cluster discrete point cloud into separate point cloud cluster. In order to get geometric feature of the separate point cloud cluster, the MBR of each point cloud cluster was calculated. Finally, the solid line was detected by restricting the length of the longer side and the length-width ratio of the MBR of each point cloud cluster. The remain road markings including the broken line, the straight arrow, the right turn arrow and the guideline were recognized through the deep learning network framework pointnet++. The result indicates that all the solid line were detected correctly. All the broken line except one and those which lay very close to the solid line were recognized correctly, All the arrows and the guideline was recognized correctly. But not all kinds of road markings have been studied in this paper. Future work will be focused on other road markings such as pedestrian crossing and crosswalk notice sign.