POINT-BASED MORPHOLOGICAL OPENING WITH INPUT DATA RETRIEVAL

: Mathematical morphology is a technique recently applied directly for point cloud data. Its working principle is based on the removal and addition of points from an auxiliary point cloud that acts as a structuring element. However, in certain applications within a more complex process, these changes to the original data represent an unacceptable loss of information. The aim of this work is to provide a modification of the morphological opening to retain original points and attributes. The proposed amendment involved in the morphological opening: erosion followed by dilatation. In morphological erosion, the new eroded points are retained. In morphological dilation, the structuring element does not add its points directly, but uses the point positions to search through the previously eroded points and retrieve them for the dilated point cloud. The modification was tested on synthetic and real data, showing a correct performance at the morphological level, and preserving the precision of the original points and their attributes. Furthermore, the conservation is shown to be very relevant in two possible applications such as traffic sign segmentation and occluded edge detection.


INTRODUCTION
Mathematical morphology has proven to be a very useful tool for detection and segmentation in image processing (Dufour et al., 2013;Loddo et al., 2018;Morales et al., 2013;Rodrigues and Marengoni, 2017;Serna and Marcotegui, 2014;Zhao et al., 2018). In addition, mathematical morphology has recently been adapted for direct use on point clouds (Balado et al., 2020). The same problems mentioned above can be solved without the need for cloud-to-image transformation and taking advantage of the 3D distribution of points in space. In point clouds, morphological erosion is used for the detection of 3D shapes and morphological opening (erosion followed by dilation) for the segmentation of 3D shapes.
Point based mathematical morphology defines morphological erosion as a process of removing points and morphological dilation as the addition of points, both following a specific structuring element. Although these two processes are independent, when executed consecutively, for example in morphological opening, we would expect that the points removed in erosion would be recovered in dilation. However, the eroded input points are definitively lost and then, in the dilation, new points are generated based on structuring element points. This removal and replacement by new points implies a loss of the exact position and a loss of the attributes of the eroded points, information very relevant for point cloud processing (Morell et al., 2014).
The aim of this work is to modify the point based morphological opening to preserve the eroded points while maintaining the input 3D geometry and attributes. The modification is done by saving the eroded cloud and the discarded point cloud separately. In the dilation process, the eroded point cloud is used to position the structuring element which, instead of adding its own points, searches for nearby points in the discarded point cloud. The rest of this paper is organized as follows. In Section 2 the best known point cloud segmentation methods are compiled. Section 3 contains the explanation of morphological opening with the modification for input data retrieval memory. In Section 4, results on synthetic data and real point cloud applications are presented. Section 5 concludes the work.

RELATED WORK
Point cloud segmentation is one of the fundamental operations of point cloud processing. Segmentation can be based directly on the geometry and distribution of the points, but also on attributes calculated from geometry (surface normals, eigenvalues, etc.), attributes related to the LiDAR nature (reflectivity, number of returns, etc.) or attributes added from other sources (the clearest example is the RGB colour).
The most direct segmentation is through manually defined thresholds. In many cases, human knowledge of the problem or regulations allow the points to be separated. The limitation of manual selection of fixed thresholds is the sensitivity to noise and unforeseen situations. In this respect, Artificial Intelligencebased methods (both supervised and unsupervised) rely on the acquired data to set the thresholds they consider optimal. The kmeans algorithm is the most widespread unsupervised method for segmentation (Shi et al., 2011). The algorithm cluster the point cloud into k regions, such that the distance (either geometric or in attributes) from each cluster centre to the points assigned to the cluster is minimal.
An alternative to the direct selection of thresholds, there are methods based on region growing (Khaloo and Lattanzi, 2017). These methods start from a seed point to which other points with similar attributes are added for region generation (Vo et al., 2015). The difficulty of seed-based methods lies in the selection of the first seed.
A very common algorithm for the detection of geometric shapes (mainly planes, but extendable to lines, cylinders and spheres) is RANSAC (RANdom SAmple Consensus) (Li et al., 2017). This algorithm is of the model-based method type, whereby the point cloud is segmented following geometric models (Nguyen and Le, 2013;Sánchez-Rodríguez et al., 2019). These algorithms are very robust in the definition of outliers and do not require the calculation of attributes for their operation.
The unstructured nature of the point clouds is a limitation in data processing, so many authors prefer to work by converting the point cloud to voxels. Voxels can be grouped into supervoxels (Ramiya et al., 2016;Wang et al., 2015) when they have similar attributes and thus segment the point cloud according to the points belonging to each supervoxel.
To structure the point cloud without loss of precision, many authors prefer the use of graphs. By applying knn search, points can be connected with the nearest neighbor (Balado et al., 2019). Then, point can be considered as nodes and the connections as edges to create de graph. By converting the cloud into a graph, graph theory algorithms can be applied (Landrieu and Simonovsky, 2018). A common algorithm for this is the connected components algorithm (Aissou and Aissa, 2020), which allows individualizing unconnected elements in the cloud.
In relation to the methods mentioned above, point-based morphological opening is based exclusively on geometric and spatial relations between the points and the coincidence with a model defined by the structuring element. This allows great freedom for the segmentation of complex shapes, but at the same time, it makes the selection of a correct structuring element difficult due to the infinite degrees of freedom.

METHOD
The morphological opening consists of a morphological erosion followed by a morphological dilation, with the same structuring element. The input data is a point P = (PX, PY, PZ, PA1, PA2…PAm) with m attributes and n number of points; and a point cloud used as structuring element S = (SX, SY, SZ), with a reference point R used to iteratively move the structuring element to each point Pi of the input data.
After erosion, an eroded point cloud E is obtained, with E = (EX, EY, EZ, EA1, EA2…EAm), same m attributes and ne number of points, where ne < n. After dilation, (and as output data of the morphological opening process), a dilated point cloud D is obtained, with D = (DX, DY, DZ, DA1, DA2…DAm), same m attributes and nd number of points, where ne < nd < n. The morphological opening proposed in (Balado et al., 2020) discards the eroded points and replaces them with points from structuring elements, losing the attributes and precision of the input point cloud. The proposed modification presented in this paper consists in the conservation of the eroded points and their recovery if necessary, in the dilatation process. Figure 1 shows original workflow of the point-based mathematical opening and Figure 2 shows the morphological opening with the proposed modification.

Morphological erosion
The modification in the morphological erosion consists of saving and retaining the eroded points in a separate cloud. Thus, at the end of the process, an eroded point cloud E (containing the remaining points) and a discarded point cloud T (containing the eroded points) are obtained.
Given a point cloud P and a structuring element S with a reference point R, defined above, the erosion process starts with the calculation of the average distance from each point Pi to the 5 nearest points in the point cloud P to define a search threshold d between the S and P. Then, the structuring element S is iteratively translated to each point Pi of the input cloud using R as the reference point for the translation. No rotations are performed, therefore, the orientations of the shapes of the structuring element S searched in P must coincide.
When S is translated to each Pi (geometrically R = Pi), for each point in S, a search for near points at a distance d in the input point cloud P is performed. In each iteration, if for all S translated points there is at least one point close (within d), the point is stored in the eroded point cloud E, if not, the point is discarded and stored in the discard point cloud T.  ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume VIII-4/W2-2021 16th 3D GeoInfo Conference 2021, 11-14 October 2021, New York City, USA

Morphological dilation
The modification in the morphological dilation consists of using the eroded point cloud E to position the structuring element S, but not to add the points of the structuring element. Instead, the structuring element S is used to search for nearby points in the discard point cloud T and add them to the eroded point cloud and generate the dilated point cloud D.
The process starts with copying the eroded point cloud E to a new point cloud Dtemp (which will be the dilated point cloud in the output). Then, the structuring element S is iteratively translated to each point Ei, taking as reference point R. For each translation of the structuring element, points in S are used to search for near points in the discard point cloud T at a distance smaller than d.
The d value is retained from its calculation in the morphological erosion, as no relevant changes in point density are assumed. The points found in the discarded point cloud T are added to the dilated point cloud Dtemp and removed from T. After all iterations, all eroded points close to the structuring element are retrieved, obtaining the dilated point cloud D as output.

Application on synthetic data
The test on synthetic data aims to show the behaviour of the designed algorithm on 3D data, with regular point distribution and clear attributes, such as RGB colour. A semi-open and empty cube was chosen, with 2243 points, a distance between points of 0.1 m and a colour gradient in the direction of the X axis. The dimensions and orientation of the input synthetic point cloud and structuring elements are shown in Figure 3. Two tests were conducted on the synthetic point cloud, varying the structural element.
The first experiment on synthetic data consisted of testing the method for segmenting vertically oriented points. A structuring element consisting of three points oriented on the Z-axis separated by 0.1 m was used (Figure 3.b). In the erosion process, the horizontal areas as well as the upper and lower edges of the vertical ones were removed (Figure 4.b). In the dilation, the point cloud grew from the previously undeleted vertical areas and regained its original points and colour (Figure 4.c).
The second experiment consisted of using the point-based morphological opening to eliminate small areas with an orientation coincident with the structuring element. In this case, the size of the structuring element acted as a filter to eliminate areas whose size was smaller than the structuring element. Therefore, the structuring element was defined as three points, separated 0.3 m, forming an X-axis oriented straight line ( Figure  3.c). The total width of the structuring element was 0.6 m, while the width of the area to be removed was 0.5 m. In the erosion process, the narrow area was completely eliminated, and the wide area was eroded with a size close to the SE (Figure 5.b). Since the structuring element has an orientation adjusted to the X-axis,  the process affected both horizontal and vertical surfaces X-axis oriented. Upon dilation process, the point cloud recovered the original size and attributes from the preserved points ( Figure 5.c).
In both cases, the maintenance of colour (as an attribute to be preserved) was satisfactory. The colour gradient of the input no loss or replacement of colour was observed. Furthermore, a correct application of the mathematical morphology was observed in terms of removing points according to the size and shape of the structuring element. However, given the grid adjustment of the synthetic point cloud, produced a perfect pointto-point matching of input point cloud and structuring element so, to analyse the conservation of points in an irregularly distributed grid, real case studies were used.

Application on acquired data
The testing on data acquired from real environments aims to show how the methodology works on case studies with a distribution of the most irregular points and possible applications in the real world.
The first experiment on a real case study consisted of segmenting a traffic sign from the sign pole. The signal was acquired with the LYNX Mobile Mapper of Optech (Puente et al., 2013), the point cloud had 17 thousand points, oriented on YZ plane, with an average distance between points of 1 cm and coloured according to reflectivity. To segment the sign from the pole, a structuring element oriented on the Y-axis (same plane as the traffic sign) was used. The structuring element was defined by three points spaced at 10 cm (20 cm in total), greater than the pole width (8 cm approx.). In this way, the erosion eliminated the pole but preserves enough signal points to be recovered in the dilation ( Figure 5).
The morphological opening showed satisfactory results, however, some interior points and borders were lost. The sign points (ground truth) were manually counted at 11540 points, of which 9680 points were correctly segmented (83.8%). The loss of points was due to a small gap in the interior of the sign, causing undesired internal erosion, and a poor fit at the sign border due to small irregularities. Even so, the correctly detected points, with their original intensity, represent most points in the traffic sign and would allow future identification. Furthermore, the proposed method does not introduce larger points or errors in the distance to the points like the original method, which doubles the number of points and introduces an error of 5.4 cm (Table 1).

Points
Average distance to ground truth Ground truth 11540 0 cm Output without retrieval 29134 5.4 cm Output with retrieval 9680 0 cm Table 1. Comparison between ground truth, output of the point based mathematical opening and the proposed method in terms of number of points and average distances to the nearest point of the ground truth data.
The second experiment on a real case study consisted of segmenting the horizontal surfaces on an indoor environment, acquired with a Faro Focus X330 (Frías et al., 2019). The room is a corner area enclosed by a plain wall, a curtain wall, a radiator, and a floor. The scene contains a table with various equipment on it and chairs around. The point cloud (Figure 7.a) contained 138 thousand points, with an average distance between points of 1 cm. The point cloud was coloured according to the normal-  curvature (Nurunnabi et al., 2015). The applied structure element consists of five points distributed in the form of a cross along the X and Y axes. The distance between points is 10 cm. In this case study, the point cloud was not adjusted to the X and Y axes, but a correct verticality of the environment is assumed.
As a result (Figure 7.b), the horizontal surfaces were correctly segmented with an accuracy of 96.1%, although corners whose size was smaller than the structuring element were lost. In addition, the seat of the chair was partially segmented due to its semi-curved geometry. In this example, the importance of maintaining the attribute is clear. The calculation of the curvature in the input data and maintenance in the segmented cloud makes it possible to differentiate edges in contact with other surfaces and edges produced by occlusions. In addition, the original method without input retrieval doubles the number of points and introduces an error of 4.2 cm with respect to the ground truth (  Table 2. Comparison between ground truth, output of the point based mathematical opening and the proposed method in terms of number of points and average distances to the nearest point of the ground truth data.

CONCLUSION
In this work, a modification of a point-based mathematical opening was presented to preserve the eroded points as well as the attributes of all points. Experiments have shown satisfactory results, both on synthetic and real data. The points were retrieved from the input point cloud and their attributes were preserved, using the structuring points to search for points in the input data and check for matches and not to add new geometric data directly.
This improvement allows to increase the possibilities of integration of the mathematical morphology in more complex processes, preserving the input data for a longer time, in terms of number of points, precision, and attributes. Segmentation continues to be the main use of the morphological opening, removing shapes not coincident with the structuring element.
Future work will focus on including attributes in the structuring element to be used as a new condition for the erosion/dilation of the input point clouds.
The code for point-based mathematical opening with input data retrieval is available in: https://github.com/jbalado/point-basedmathematical-opening-input-retrieval

ACKNOWLEDGMENTS
Authors would like to thank to the Xunta de Galicia given through human resources grant (ED481B-2019-061, ED481D 2019/020) and competitive reference groups (ED431C 2020/01), the Ministerio de Ciencia, Innovación y Universidades -Gobierno de España-(PID2019-105221RB-C43/AEI/10.13039/501100011033). This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 769255. This document reflects only the views of the author(s). Neither the Innovation and Networks Executive Agency (INEA) or the European Commission is in any way responsible for any use that may be made of the information it contains. The statements made herein are solely the responsibility of the authors.