SEGMENTATION OF TRAFFIC SIGNS FROM POLES WITH MATHEMATICAL MORPHOLOGY APPLIED TO POINT CLOUDS

Traffic signs are one of the most relevant road assets for driving, as the safety of drivers depends to a great extent on their correct location. In this paper two methods are compared for the segmentation of the sign and the pole supporting it. Both methods are based on the morphological opening to identify the sign points, the first one directly employs the mathematical morphology directly applied to point clouds and the second one through point cloud rasterization into images. The comparison was conducted on twenty real traffic signs acquired with Mobile Laser Scanning obtaining point clouds from environments with signposts, traffic lights and lampposts. The results showed a correct segmentation of the signs, obtaining a F-score of 0.81 by the point-based method and a 0.75 by 2D image method. In particular, the point-based mathematical morphology proved to be more accurate in the segmentation of traffic sings installed on traffic lights and lampposts, avoiding over detection shown by the 2D image method.


INTRODUCTION
Traffic sign detection in 3D point clouds from road environments has been an active research topic during the past decade, motivated by the relevance of the asset within the road infrastructure and its typically standardized geometries. The work of Pu et al. (2011) proposed a shape analysis algorithm for traffic sign recognition, using shape-based features such as Minimum Bounding Rectangle, Minimum Bounding Circle and Hu moments to extract rectangular, circular and triangular shapes. However, radiometric properties have been found more useful than geometry for the detection of traffic signs. As the intensity attribute of 3D point clouds is related with the retroreflectivity of the materials, it has been commonly used as a key feature in the literature. Riveiro et al. (2015) generate an intensity-based raster image to isolate the location of the traffic signs on the raster plane, and then filter high-intensity points using a Gaussian Mixture Model (GMM). This approach is generalized in 3D, with no rasterization step, by several authors (Guan et al., 2018;Soilán et al., 2016;Wen et al., 2015) that use the intensity attribute as well as previous knowledge about the traffic sign geometry. More recently, strategies based on Deep Learning have been used for traffic sign detection (Liu et al., 2020), although recognition processes that are able to read the sign have still to rely on digital images (Arcos-García et al., 2017;Gudigar et al., 2019). While these methods can detect traffic sign panels, they need further processing steps to detect any structure the traffic sign is attached to. These structures are generally polelike objects, which have been an object of study in the literature * Corresponding author as well. Yu et al. (2015) detect pole-like objects using a similarity measurement with respect to an affine-invariant 3D shape context, focusing on the detection of street light poles. Previous knowledge of the geometry of a pole is used as well by Cabo et al. (2014), studying the occupancy of a local voxel grid, or by Li et al. (2016), who define an adaptive radius cylinder model. Other noteworthy approaches include the usage of supervised classification algorithms such as Support Vector Machines (Wu et al., 2017) or Random Forests (Yan et al., 2017).
As it can be seen from this state-of-the-art analysis, the literature is extensive on this topic, but traffic sign panels and poles have been typically considered separated entities. Differently, this work considers both as part of the same object in an initial stage.
The aim of this work is the separation of the traffic signs from poles, applying mathematical morphology in two ways: directly in a point cloud (Balado et al., 2020b) and in a 2D image through point cloud rasterization on the plane of the traffic sign.
The rest of this paper is organized as follows. In section 2, both proposed methods are explained, the point based one as well as the image based one. Section 3 presents and analyses the results of the application. Section 4 concludes the work.

METHODS
In this work, two types of morphological openings are compared. One is applied directly to the point cloud and another is applied to the 2D image after a rasterization. Morphological opening is composed of an erosion process followed by a dilation process. In the erosion process, the points/pixels coinciding with the shape of the structuring element are selected, while the remaining points/pixels are removed. In the dilation process, the shape of the object is recovered from the shape of the structuring element and the points/pixels preserved in the erosion. Figure 1 shows the traffic sign segmentation workflow for each method.

Input data
The input data are individual point clouds of only traffic sign and pole sets. In the proposed methods only geometric information is used, therefore each traffic sign point cloud is defined as = ( ) ∈ ℝ with n points. These input data (traffic sign with pole) can be obtained from a street point cloud acquired with Mobile Laser Scanning using multiple ways. The ground of the street is segmented to break de connectivity between objects (Novo et al., 2020;Soilán et al., 2018). Then, objects are clustered with connected components. The each object is classified through point cloud to image conversion and a Convolutional Neural Network trained with images from online sources (Balado et al., 2020a).

Rotation
Rotation is a fundamental process for orienting all traffic signs along the same axis and obtaining a front view in the case of image processing. The aim of this process is to adjust the sign plane to the ZX-plane, applying a rotation on the Z-axis, since it is assumed that the traffic sign with pole are vertical elements.
First, Principal Component Analysis (Jolliffe and Cadima, 2016) is applied to the and components to obtain the covariance matrix (Equation 1). The component is omitted, as the correct verticality of the traffic sign is assumed ( Figure 2). The rotation angle α is calculated from the components of the covariance matrix (Equation 2) and a Z-axis rotation matrix R is generated (Equation 3), which is applied to the P cloud to obtain the traffic sign adjusted to the ZX-plane.

Point-based morphological opening
The application of point based morphological opening was presented in (Balado et al., 2020b). Morphological opening is an iterative contrast of each point of the input cloud against another point cloud that functions as a structuring element.

Structuring element definition.
In order to correctly segment the signal from the pole, a structuring element must be defined based on the distinctive geometry of the signal in front of the pole. The traffic signal is a flat element with a minimum size of 60 cm (Richard C. Moeur, 2019;UK Government, 2016). The pole has a smaller size (approximately 10 cm horizontally). Even if the sign is installed on a traffic light or lamp post, the pole is still smaller than the sign. In view of these considerations of shape and size, a structuring element is a point cloud of five points, distributed in a planar cross oriented on the XZ plane, coinciding with the sign orientation. The horizontal points of the cross are placed at 10 cm from each other, to ensure that the horizontal segmentation removes the too narrow elements (poles) while retaining the sign. The vertical points are placed 5 cm from each other, generating an XZ plane with the other points of the structuring element, to ensure that detected points are in a correct oriented plane. The cross shape does not produce a good segmentation of the corners of square signals but allows for better fitting of triangular and circular signals. The point cloud of the structuring element is shown in Figure 3. 2.3.2 Morphological opening application. In the point morphological erosion (Figure 4), the structuring element is translated iteratively to each point Pi of the input cloud P. In each position, a match is sought between input data and all structuring element points. If there is a completely match, the point Pi is preserved, else, the point Pi is removed. The match between points is based on a proximity threshold d, which is calculated as the average distance of each point from its 5 nearest neighbours in the input point cloud. The remaining points are then subject to morphological dilatation ( Figure 5), which consists of moving the structuring element to each point Pi in the eroded point cloud and copying the points of the structuring element in that position if there are non-matching points.

2D image morphological opening
Morphological opening in images is a widespread process for detection (Mat Said and Jambek, 2016;Salazar-Colores et al., 2018). The input of the mathematical morphology in images is usually a binary image and a binary structuring element in the form of a 2D matrix.
2.4.1 Rasterization. Before applying the 2D morphological opening, the point cloud must be converted into an image. Rasterization is the process by which the dimensionality of the cloud is reduced, projecting the points to pixels (Balado et al., 2017). The projection is on the XZ plane, according to rotated sign. The points are projected onto the cells (pixels) of a grid (image). Rasterization produces a distortion effect when fitting the point cloud to a grid. The size of the pixels should be greater than the distance between nearest points to ensure a compact distribution of the objects on the grid and to prevent holes inside.

Structuring element definition.
The structuring element applied to the raster has the purpose of filtering the cells that do not belong to the traffic sign, therefore the structuring element has two dimensions like the sign, and the size (in number of pixels) must be enough to remove the pixels belonging to poles. A cross-shaped 3 x 3 pixels (Equation 4) is selected as structuring element, equivalent to the structuring element used in the point-based method and meeting the two requirements abovementioned, bidimensional and pixel size is fixed by the raster process.

Morphological opening application.
Given a binary input image (1-existence and 0-absence of points in the cell) and the image of the structuring element , the erosion process is applied followed by the dilation. During the erosion process ( Figure 6), the structuring element is translated to each pixel of the image with value 1, acting as an image mask. If the structuring element values coincide with the pixels of neighbouring pixel , the pixel value is preserved and if not, eroded (changed to 0). In the dilation process (Figure 7), the structuring element is translated to each pixel of the eroded image with value 1. The values of the neighbouring points are changed to 1 coinciding with the overlap of the structuring element.

Data
The test data were twenty traffic signs, acquired with the LYNX Mobile Mapper of Optech (Puente et al., 2013), in the city of Vigo (Spain). Eleven traffic signs were located on independent poles, six on streetlights and three on traffic lights. The average distance between nearest points were d = 1.9 cm, and for the raster, a pixel size of 10 cm was established to ensure no holes within traffic sign.

Results
The precision, recall and F-score metrics (Sánchez-Rodríguez et al., 2019) were used to analyse the quality of the results. Since in point-based mathematical morphology, input points can be replaced by new points (in the dilation of the point cloud according to the structuring element) that may not exactly coincide with the position of the input points, the consideration of correct point detected was made based on the proximity threshold d. A traffic sign point is correctly detected when it is within d of a reference traffic sign point (ground truth). This assumption was used for both tests, point-based and 2D mathematical morphology. The traffic signs were manually segmented for generate the ground truth. The results of the metrics for both methods for each traffic sing are given in Table  1 and point clouds are shown in Figures 8 to 11.

Analysis and discussion
Both methods showed a correct segmentation in most of the signs, although the difficulty in those signs installed with traffic lights and lampposts was greater than when sings were alone on a sign pole, as there were more possible confusion points and common geometries. The point-based mathematical morphology showed 7% better F-score than the 2D image method. Although the segmentation between two methods was similar in sings alone in poles, the point-based method showed best results on the study cases where the traffic sings were installed in lampposts and traffic lights.     The use of morphological opening applied to images showed a clear tendency towards over detection. The fact of projecting the points on an image, produced that wide forms (like parts of traffic lights and lampposts) were not correctly distinguished from the signs (e. g. traffic sign #3 in Figure 8). On the other hand, when the signs were alone on the pole, the detection was very precise. In addition, the 2D image method ensured the preservation of the original points of the segmented signs.
The use of point-based morphological opening showed a slight tendency to under detection in all traffic sings. This behaviour was positive for the sign not being confused with other large elements such as traffic lights and lamppost lights (e. g. traffic sign #12 in Figure 10). The 3D structuring element filters out by size and shape the parts do not correspond to a traffic sign. However, although the signs were detected, the segmentation at the edges did not show good delineation (e. g. traffic sign #16 in Figure 11).
Both methods presented a common limitation in the rotation process. If the traffic sign was not properly adjusted on the XZ plane, the segmentation was not conducted correctly. By the number of points, the sign defines the plane that orients the object. If the object has major points, for example in traffic lights and lampposts, the large amount of these points influences the orientation, although sign and object orientations normally coincide. If no-signs points are oriented in a different direction from the sign, (e. g. traffic sign #5 in Figure 8), the sign is not segmented as it does not coincide with the orientation of the structuring element. This problem would also appear with several unaligned traffic signs on the same pole.
Another limitation of the mathematical morphology, common to both methods, is the selection of the structuring element. The parameters for correctly choosing a suitable structuring element are based on prior knowledge of the problem and are not without a trial-and-error component. A common problem in 2D and point clouds is defining the size and shape of the structuring element. Both parameters are obtained from the part of the pole-sign to be kept (sign) and the part to be discarded (pole). In addition, in point-based mathematical morphology, a point density of the structuring element must be defined. The point-to-point distance of the structuring element is related to the point density of the input cloud, hence to the threshold d. The point distance of the structuring element must ensure two conditions: (1) maintain the desired structuring element shape and, (2) be equal to or larger than the average point distance of the input point cloud. The fewer points in structuring elements involves fewer iterations in morphological erosion and dilation, saving processing time.
The methods were programmed in Matlab, and the tests were performed on an Intel® Core computer ™ i7-7700HQ 2.8GHz 16GM RAM CPU. The reorientation of all signs was performed in 0.11 seconds, the morphological opening by image processing (and the rasterization) took 0.4 seconds, approximately 0.02 seconds per sign; while using point-based method took 83 seconds, approximately 4.16 seconds per signal. Therefore, despite having an integrated rasterization phase, the 2D image method took 200 times less time than the point-based method.

CONCLUSION
This paper showed the potentiality of the point-based mathematical morphology (Balado et al., 2020b) versus 2D mathematical morphology for the segmentation of traffic sings from poles in point clouds. Two methods were designed to segment the traffic sign from the pole, but also from lampposts and traffic lights. Both methods were tested on twenty point cloud signs. With the point-based mathematical morphology obtained the best results with an F-score of 0.82, while the 2D mathematical morphology only reached 0.75. In addition, the point-based method showed a more equilibrated behaviour in the segmentation in lampposts and traffic lights, while the 2D image method tended to over detection.
The results obtained by the point-based morphological opening indicate that the mathematical morphology applied directly to point clouds allows a better analysis of the 3D geometry of complex elements because it allows the definition of structuring elements in 3D and the confusion due to the dimensional losses associated with the transformation of point clouds into images is reduced. Furthermore, although the work shows results for a simple structuring element, it is also possible to generate complex 3D structuring elements that greatly extend the range of possibilities with respect to 2D.
Future work will focus on improving the point-based mathematical morphology, mainly by opting for an automatic orientation of the structuring element to the input point cloud and looking for preserving the original points in the morphological opening. Furthermore, this method will be extended to the segmentation of parts of other objects and improving processing times with discrete or continuous Level of Detail techniques.