PLANAR SURFACE SEGMENTATION USING A COLOR-ENHANCED HYBRID MODEL FOR RGB-D CAMERA-BASED INDOOR MOBILE MAPPING POINT CLOUDS

Point clouds acquired by RGB-D camera-based indoor mobile mapping system suffer the problems of being noisy, exhibiting an uneven distribution, and incompleteness, which are the problems that introduce difficulties for point cloud planar surface segmentation. This paper presents a novel color-enhanced hybrid planar surface segmentation model for RGB-D camera-based indoor mobile mapping point clouds based on region growing method, and the model specially addresses the planar surface extraction task over point cloud according to the noisy and incomplete indoor mobile mapping point clouds. The proposed model combines the color moments features with the curvature feature to select the seed points better. Additionally, a more robust growing criteria based on the hybrid features is developed to avoid the generation of excessive over-segmentation debris. A segmentation evaluation process with a small set of labeled segmented data is used to determine the optimal hybrid weight. Several comparative experiments were conducted to evaluate the segmentation model, and the experimental results demonstrate the effectiveness and efficiency of the proposed hybrid segmentation method for indoor mobile mapping three-dimensional (3D) point cloud data.


INTRODUCTION
With the huge demands of emergency response simulation and training, cultural heritage protection, digital city, and other related applications, the indoor mobile mapping system (Pathak et al., 2009, Bouvrie et al., 2011), which integrated the mobile platform with positioning sensors, laser scanners, optical cameras and other sensors, provides a highly efficient way to obtain threedimensional (3D) point cloud data for an indoor environment.In particular, with the use of a Kinect sensor, a commercial RGB-D camera providing the depth image along with the RGB image, many indoor mobile mapping systems have been developed recently (Andreasson et al., 2012, Henry et al., 2012, Endres et al., 2012, Wen et al., 2014).
For RGB-D camera-based indoor mobile mapping applications, there are two issues of concern: (1) unique structure characteristics of the indoor environment, such as close-range sensing, multi-occlusion, lack of absolute positioning, and uneven lighting, and (2) the limited performance of the commercial RGB-D camera, such as low resolution, limited working distance, and noisy data acquired (Khoshelham and Elberink, 2012).In general, the point cloud data acquired by the RGB-D camera-based indoor mobile mapping system suffers the problems of being noisy, exhibiting an uneven distribution, low resolution and incompleteness (Han et al., 2013).These data quality problems are much more obvious than the point cloud data acquired by a laser scanner.
Planar surface extraction for point cloud (point cloud segmentation) separates the point cloud into surfaces and merges the surfaces with similar properties (such as color, geometry) into a single surface.Effective point cloud segmentation is a * Corresponding author.Chenglu Wen (clwen@xmu.edu.cn).
prerequisite of feature extraction, object recognition and scene labeling, and the segments achieved from segmentation will be the smallest unit for feature extraction and object recognition.
Several works have been performed related to laser scannerbased point cloud segmentation (Liu and Xiong, 2008, Burunnabi et al., 2012, Golovinskiy and Funkhouser, 2009).Additionally, Euclidean clustering method segments the point cloud by the assumption of continuity of the segmented surface controlled by a setting threshold (Rusu, 2010).However, the scattered oversegmentation surfaces are obtained because only the continuity of the point is considered and none of the features are considered along with inflexible control over the setting threshold of continuity of the points.Ioannou et al. (2012) proposed a difference-of-normals method to segment the large scale unorganized point cloud; the method has high computation efficiency and is also multi-scale adaptive.However, the difference-of-normals method can only segment a limited planar surface, and there are many planar surfaces that exist in indoor scenes.T. Rabbani et al. (2006) proposed a region growth method to segment the unorganized point cloud based on the similarity of the curvature and the continuity of the local points.Incorrect selection of the curvature and angle thresholds will lead to excessive under-segmentation or over-segmentation, and also the method is sensitive to the seed point selection.In particular, the principal curvatures or continuity-based segmentation methods are not robust enough when the point cloud is unevenly distributed and noisy.
Considering the low-quality problems of the RGB-D camerabased indoor mobile mapping point cloud, the above point cloud segmentation methods cannot solve the segmentation task well.As a calibrated sensor, the RGB-D camera provides the RGB ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-1, 2014ISPRS Technical Commission I Symposium, 17 -20 November 2014, Denver, Colorado, USA This contribution has been peer-reviewed.The double-blind peer-review was conducted on the basis of the full paper.doi:10.5194/isprsannals-II-1-61-2014information along with depth information for each point.It is possible to use color information to improve the robustness of the segmentation performance.
In this paper, a color-enhanced hybrid segmentation model based on the region growing method is proposed for RGB-D camerabased indoor mobile mapping point cloud planar surface segmentation, and the model is more robust to the clustered, noisy and incomplete point cloud data compared to the traditional point cloud segmentation method.The model combines the color-based information with the curvature-based information for the seed point selection in the region growing method.A hybrid growing criteria is also developed with consideration of the color similarity, curvature similarity and point continuity.The hybrid weight is determined by a segmentation evaluation processing based on a small set of labeled segmented data.The segmentation results are given based on the hybrid weight effect on the segmentation performance comparison between the segmentation methods.

DATA ACQUISITION AND PRE-PROCESSING
The indoor mobile mapping system is with a weight of 22.5 kg and with a size of 46 cm × 50 cm × 82 cm (length × width × height).The basic mobile platform is four-wheeled Pioneer3-AT robot.The system is equipped with a RGB-D camera (Kinect sensor, 640 × 480 pixels, and 57 o × 43 o field-of-view for 3D range data), and the camera acquires 3D range data under various illumination situations because it illuminates the object based on infrared radiation.A 2D laser scanner (SICK LMS100), which covers a scanning area of 270 o , is mounted on the platform to achieve 2D scan profile for 2D map building.The mapping system is capable of operating 4 hours with three full charged batteries (12 V lead acid, 7.2 AH), and its core system is an Intel-i5-2.53GHz processor and 2 GB RAM with a Linux operating system (As shown in figure 1).

Figure 1. Mobile mapping system design
Since the point cloud data acquired by the RGB-D camera-based system are limited (Han et al., 2013), a pre-processing process is needed (shown in figure 2).The point cloud pre-processing method used in this paper include: (1) down-sampling for acquiring point clouds with consistent resolution, (2) de-noising using Gaussian filtering (Liu et al., 2012), (3) point cloud data interpolation using the moving least squares (MLS) smoothing.We determined that with the down-sampling, the point cloud density decreased dramatically but are accompanied by low resolution and an uneven distribution.Finally, the points are evenly distributed after MLS smoothing.

COLOR-ENHANCED HYBRID SEGMENTATION MODEL
For the original region growing segmentation algorithm, only the curvature information is used for seed selection.With the quality problem of the RGB-D camera-based indoor mobile mapping point clouds data, a more robust seed selection method and growing criteria are required.We combine the color moment features with the curvature feature for the seed point selection and growing criteria, and use a segmentation evaluation process to optimize the hybrid weight.

Color-enhanced Seed Point Selection
For each point in the point cloud, select k neighbor points of radius and calculate the covariance matrix as: where = centroid position of the k neighbor points of = the feature value in the covariance matrix = the feature vector in the covariance matrix The smallest component of refers to the normal vector of .The curvature of is expressed as: (2) The first-, second-and third-order moments of the color feature of in the radius of r is calculated as: where = the first-order moment of the hue, saturation and illumination components of the HSV color space of = the second-order moment of the hue, saturation and illumination components of the HSV color space of = the third-order moment of the hue, saturation and illumination components of the HSV color space of = one of the color channels of the nth neighbor point of When the features of the normal vector, curvature and color moments for each point are computed, the seed point for region growing is selected based on two principles: (1) the color-stable region and (2) the geometry-stable region.As shown in figure 3, the steadiness of color feature is represented by the second-order moment ( ∑ ).The steadiness of the geometric shape is represented by the curvature ( ).The unstable value of the candidate seed point considering the color and curvature information, , is represented as:

Growing Criteria
In the process of region growing, the local color features (color moments) and local structural features (normal vectors) are used as a basis for growth.For RGB-D camera acquired point cloud data, the shape and color information of an object is non-uniform, noisy and incomplete; as a result, adding multiple similarity measurements of features improves the segmentation algorithm by decreasing the computation complexity and adding robustness.

Suppose
, , and , , are the two points in the point cloud, the proposed method merges the two points into a segment by determining the similarity of the two points, , , as: , 1，when ， or 1，when ， and 0，otherwise • (10) where = Euclidean distance of the coordination between the two points = Euclidean distance of the color moment between the two points = normal vector angel between the two points = normal vector of point = normal vector of point = a setting angel value = a setting angel value The similarity value between the two points is of value 1 when they are merged into a segment, while the similarity value is 0 when the two points go to two segments.The two points in a point cloud can be determined to be connected by: , where is a setting value for the minimum distance between the two points.The two points are considered to be in one segment for three cases: (1) and are two connected points, and the Euclidean distance of the color moment between the two points is smaller than a setting value ; (2) and are the two connected points, and the normal vector angle between the two points is smaller than a setting value ; (3) and are the two connected points, the angle of the normal vector between the two points is smaller than a setting value , and the Euclidean distance of the color moment between the two points is smaller than a setting value ( ≫ ， ≫ ).

Hybrid Weight Optimization
The hybrid weight in (6) presents the importance weight that the color or curvature features are assigned during the seed selection.
In the paper, a segmentation evaluation process based on a set of labeled segmented data is used to obtain the optimal hybrid weight.
For RGB-D camera-based indoor point cloud data, five segmentation evaluation states, which are correct segmentation, over-segmentation, under-segmentation, mis-segmentation, noisy segmentation, are used to determine the segmentation performance (Hoover and Jean-Baptiste, 1996).Based on the above definition, the first four segmentation states for each segment are represented as: where ∈ (0,1) = evaluation threshold value = segments obtained by the proposed method = corresponding manually segmented results of A higher value denoting a more strict evaluation.The segmentation result is exactly the same as the manually segmented result when 1.The four states (Correct, False, Over, and Under) represent the correct segmentation, missegmentation, over-segmentation, and under-segmentation results, respectively.Except the above four states of segmentation, the unlabeled point cloud segments after segmentation are classified as noisy segmentation.
The over-segmentation and correct segmentation states of a point cloud are helpful for semantic labeling.Obtaining segments that are too dense usually results in segments with meaningless content, thereby ultimately leading to low labeling accuracy.The point cloud segmentation errors due to under-segmentation and mis-segmentation indicate that it will be unable to distinguish between two different objects in the underlying semantics.
Suppose there are segments after segmentation; an evaluation score S is defined as: The optimal hybrid weight is determined by the maximum value as:

Effects of Hybrid Weight on Segmentation
The hybrid weight in the segmentation model determines the proportion of color or curvature information used for the seed point selection.To test the hybrid weight effects on segmentation, we set 0, 0.1, 0.2, … 1.0, the color moment threshold value 50.0, and the normal vector threshold value 0.06.Then, we compared the segmentation performance under these conditions.The subsequent segmentation results of the scene using the hybrid model are shown in figure 4.
As shown in figure 4, a scene of a relatively complex indoor environment is used to test the segmentation performance.The manual segmentation result is used as the ground-truth.Blocks with different colors represent different planar surfaces extracted.When 1, the seed selection for region growing is totally dependent on the curvature feature, which mainly results in an over-segmented situation.The seed selection for region growing is totally dependent on color moment feature when 0, which primarily results in an under-segmented situation.The segmentation result is best when 0.7.To better evaluate the proposed method, we also use the Cornell-RGBD-Database (Cornell database) (Anand et al., 2013) as the data input.This database contains two types of scenarios (Office or Home) of 52 labels for over 150 frames of point cloud data.Three scenes from the Cornell database are selected to test the effects that hybrid weight value has on the segmentation results (figure 5).
The Original RGBD point cloud data are shown in the first row, and the manually segmented results (as ground-truth) are presented in the second row in figure 5.With different hybrid weight ( ) used, the segmentation results purely based on the curvature feature ( 1) and the color moment feature ( 0) are not as good as the segmentation result when the hybrid weight is 0.7.For scene 3, the number of segments for different segmentation states is shown in figure 6.An excessively small hybrid weight is found to result in an excessive amount of under-segments.

RGB point cloud
Since the segmentation results are used for further scene understanding and recognition, over-segments are needed.Therefore, a larger hybrid weight is recommended for the proposed segmentation model.

Segmentation Performance Comparison
To compare the planar surface segmentation performance with other popular point clouds segmentation methods, we compared our method based on the Cornell database with three other methods: (1) the Euclidean clustering segmentation method, (2) the difference-of-normals method and, (3) the classical curvature-based region growing method.
The same scenes (scene 1, scene 2 and scene 3) in figure 5 are used to illustrate the segmentation results based on the four above-mentioned methods.The comparison of the segmentation results for different segmentation methods is given in figure 7.
Blocks with different colors represent different segments.The results from the difference-of-normals segmentation method achieved over-segmented elements, and most of the planar elements are also filtered (figure 7(a)).The Euclidean clustering method is not robust enough for incomplete and noisy data; for example, there are too many over-segments on the wall and screen area (figure 7

Evaluation of the Robustness to Noise
Because the RGB-D camera acquired indoor point cloud data are noisy and incomplete, the excessively small segments obtained after segmentation will be assigned as unlabeled noisy segmentation.We evaluated the robustness to the noisy data for the different segmentation methods by a noisy segmentation ratio.
The noisy segmentation ratio is defined as: where represents the point cloud number, and represents the unlabeled point.A smaller noisy segmentation ratio value means the method is more robust to the noisy and incomplete data.10 different scenes are tested to calculate the noisy segmentation ratio using the above four segmentation methods (figure 8).As we can see from Figure 8, the proposed hybrid method and the difference-of-normals method have the smallest noisy segmentation ratio considering all ten scenes in general, followed by Euclidean clustering and region growing method.The results indicate that the proposed hybrid method and the difference-ofnormals method perform better in robustness to incomplete data and noisy data and that these two methods are more suitable to address the RGB-D camera acquired indoor point cloud data.In particular, for scenes 4 and 5, all four of the methods exhibit a relatively high noisy segmentation ratio because of the more incomplete and noisy data due to occlusion and varied lighting.

Efficiency Evaluation
For the above four segmentation methods, we also tested the computational efficiency based on 25 indoor scenes.The comparison results are shown in figure 9.
Figure 9.Time consumption comparison using four segmentation methods to segment 25 scenes.
The difference-of-normals method required more time than the other methods.For the same data, the region growing method has the smallest computation complexity of the methods considered, the Euclidean clustering method required moderate time, followed by the proposed hybrid method.The computational time costing among the proposed hybrid method, Euclidean clustering method and region growing is comparable.It needs to be noticed that the proposed hybrid method has an additional step of calculating color moments related to seed point selection when compares to traditional region growing method.
The difference-of-normals method can be concluded to be suitable for large scale scene segmentation, and it is robust to noisy data, with a low mis-segmentation rate but with high computational complexity.However, indoor environment contains many structured artifacts (such as screen, books, chassis, etc.), which will be ignored by the difference-of-normals method.
The Euclidean clustering method has moderate computation complexity, and it is less error prone when dealing with clustering and messy indoor scene segmentation.The region growing method is of low computational complexity, but its output is primarily over-segmented planar surfaces and also it exhibits low robustness to noisy and incomplete data.The hybrid model works well for clustering and incomplete data and has acceptable computational complexity.

CONCLUSIONS
In this paper, a color-enhanced hybrid model of planar surface segmentation for RGB-D camera-based indoor mobile mapping point cloud data was introduced.The hybrid segmentation model is especially effective in dealing with the point cloud data with noise, exhibiting an uneven distribution, of low resolution and of incomplete nature, which caused are the quality problems that are raised by the special structure and lighting situation of the indoor environment, as well as the limitations of the RGB-D camera.
The proposed hybrid model makes full use of the color moment features and the curvature feature of the point cloud to select the seed points by adjusting the hybrid weight.An evaluation process using a small set of labeled segmented data is conducted to obtain the optimal hybrid weight.A growing criteria with hybrid features of the color moment and the normal vector helps to avoid excessive over-segmentation according to clustering and a messy indoor scene.The experimental results demonstrated the ability of the proposed hybrid model to segment the planar surface for clustering, noisy and incomplete indoor point cloud data with a computational complexity that is accredited compared with three popular segmentation methods, and the model was demonstrated to be robust and efficient in RGB-D based indoor point cloud data segmentation.

Figure 2 .
Figure 2. Point cloud data pre-processing.(a) Original Point cloud.(b) Points number.(c) Close look of the point cloud

Figure 3 .
Figure 3. Example of the point cloud curvature and the secondorder moments of the color map color.(a) Original point cloud.(b) Corresponding curvature distribution.(c)The second-order moment of the color feature (Hue element).(d) Hybrid result.Example of the point cloud curvature and the second-order moments of the color map color is given in figure 3. The red portion represents the more stable points in the region, while the blue portion represents the more unstable points in the region.The seed point selected in the original curvature-based region growing method (figure 3(b)) is improved by the color momentbased feature (figure 3(c), and the hybrid of the color and curvature features helps in the selection of a more stable seed point (figure 3(d)).

Figure 4 .
Figure 4. Comparison of the segmentation results with different weight values

Figure 5 .
Figure 5.Comparison of the segmentation results along with RGB point cloud and ground-truth regarding to different weight values.(a) Scene 1.(b) Scene 2. (c) Scene 3.

Figure 6 .
Figure 6.Segmentation results with different combination weighting values for scene 3.
(b)).The region growing method only extracts the planar elements and omits some of the feature details (figure7(c)).The proposed hybrid segmentation method ( 0.7) provides the best segmentation results relatively regarding to the completeness and quantity of the planar elements extracted (figure 7(d)).

Figure 7 .
Figure 7.Comparison of the segmentation results for different segmentation methods.(a) Segmentation results by the difference-of-normals method.(b) Segmentation results by the Euclidean clustering method.(c) Segmentation results by the region growing method.(d) Segmentation results by the proposed hybrid model with 0.7.

Figure 8 .
Figure 8. Unlabeled noisy segmentation ratio for ten indoor scenes processed by four segmentation methods.