Comparison of 3d Interest Point Detectors and Descriptors for Point Cloud Fusion

The extraction and description of keypoints as salient image parts has a long tradition within processing and analysis of 2D images. Nowadays, 3D data gains more and more importance. This paper discusses the benefits and limitations of keypoints for the task of fusing multiple 3D point clouds. For this goal, several combinations of 3D keypoint detectors and descriptors are tested. The experiments are based on 3D scenes with varying properties, including 3D scanner data as well as Kinect point clouds. The obtained results indicate that the specific method to extract and describe keypoints in 3D data has to be carefully chosen. In many cases the accuracy suffers from a too strong reduction of the available points to keypoints.


INTRODUCTION
The detection and description of keypoints is a well studied subject in the field of 2D image analysis.Keypoints (also called interest points or feature points) are a subset of all points, that exhibit certain properties which distinguish them from the remaining points.Depending on the used operator keypoints have a high information content, either radiometrically (e.g.contrast) or geometrically (e.g.cornerness), they only form a small fraction of the whole data set, they can be precisely located, and their appearance as well as location is robust to spatial and/or radiometric transformations.
Two dimensional keypoints have been used in many different applications from image registration and image stitching, to object recognition, to 3D reconstruction by structure from motion.Consequently, keypoint detectors are a well studied field within the 2D computer vision, with representative algorithms like SIFT (Lowe, 2004), SURF (Bay et al., 2006), MSER (Matas et al., 2004), or SUSAN (Smith and Brady, 1997)), to name only few of the available methods.
Nowadays, the processing and analysis of three-dimensional data gains more and more importance.There are many powerful algorithms available, that produce 3D point clouds from 2D images (e.g.VisualSFM (Wu, 2011)), or hardware devices that directly provide three-dimensional data (e.g.MS-Kinect).Keypoints in 3D provide similar advantages as they do in two dimensions.Although there exist considerably less 3D than 2D keypoint detectors, the number of publications proposing such approaches increased especially over the last years.Often 2D keypoint detectors are adapted to work with 3D data.For example, Thrift (Flint et al., 2007) extends the ideas of SIFT and SUSAN to the 3D case and also Harris 3D (Sipiran and Bustos, 2011) is an adapted 2D corner detector.
Previous publications on the evaluation of different 3D keypoint detectors focus on shape retrieval (Bronstein et al., 2010) or 3D object recognition (Salti et al., 2011).These works show that keypoint detectors behave very differently in terms of execution time and repeatability of keypoint detection under noise and transformations.This paper investigates the advantages and limits of 3D keypoint detectors and descriptors within the specific context of ).The resulting fused point cloud can then be used in subsequent tasks like surface reconstruction (see Figure 1(c)).In order to registrate two point clouds a rigid transformation consisting of a translation and rotation is computed by a state-of-theart combination of a keypoint-based coarse alignment (Rusu et al., 2008) and a point-based fine alignment (Chen and Medioni, 1992, Besl and McKay, 1992, Rusinkiewicz and Levoy, 2001).
The experiments are based on ten different data sets acquired from three different 3D scenes.Different combinations of keypoint detectors and descriptors are compared with respect to the gain in accuracy of the final fusion results.

POINT CLOUD FUSION
The focus of this paper is the comparison of 3D keypoint detectors and descriptors.The application scenario in which this comparison is carried out is the task of point cloud fusion.This section briefly explains the process applied to fuse point clouds with pairwise overlapping views into a single point cloud.This process is also known as point cloud registration or alignment.
In a first step a fully connected graph is built.Each node corresponds to one of the given point clouds.Each edge is assigned with a weight, that is inverse proportional to the number of keypoint-based correspondences between the two corresponding point clouds.A minimal spanning tree defines the order, in which multiple pairwise fusion steps are carried out.The subsequent, successive application of the estimated transformations as well as a final global fine alignment leads to a final, single point cloud.The alignment is based on a first coarse feature-based alignment and a subsequent fine point-based alignment using ICP.The feature-based alignment uses the computed feature descriptors of the points in both point clouds to establish point correspondences between similar points.From these set of point correspondences between both point clouds, a transformation is computed that aligns the corresponding points in a least-squares sense.This coarse pre-alignment is necessary, since ICP in the second step performs only a local optimization and can only correctly registrate two point clouds with small differences in rotation and translation.

3D KEYPOINT DETECTION
An explicit keypoint estimation reduces the set of points for which point descriptors have to be calculated and decreases the number of possible correspondences.In this paper Normal Aligned Radial Feature (NARF) and a 3D adaption of SIFT are used as they represent two interesting as well as complementary approaches to detect 3D keypoints in point clouds.

NARF
The Normal Aligned Radial Feature (NARF) keypoint detector (Steder et al., 2011) has two major characteristics: Firstly, NARF extracts keypoints in areas where the direct underlying surface is stable and the neighborhood contains major surface changes.This causes NARF keypoints to be located in the local environment of significant geometric structures and not directly on them.
According to the authors this characteristic leads to a more robust point descriptor computation.Secondly, NARF takes object borders into account, which arise from view dependent noncontinuous transitions from the foreground to the background.Thus, the silhouette of an object has a profound influence on the resulting keypoints.
where di is the average distance from p to its next three direct point neighbors in direction i ∈ {right, bottom, lef t, top} of the range image.The normal vector of border points and the principal curvature of non-border points are used to detect interest points.They are determined by the main direction α of surface change and a weight w.The normal vector of a border point p is projected onto a plane perpendicular to the vector between the viewpoint and p, where it is used to compute the main direction α.The weight is set to w = 1.In the case of a nonborder point p, the direction of maximal curvature is projected onto a plane perpendicular to the vector between the viewpoint and p.The resulting angle defines the main direction α and the corresponding curvature is set as the weight w.
An interest score I(p) is defined for every point p, which is based on all neighboring points {n0, ..., n k } of p within a radius of σ, which do not have a border in between: All points with an interest value higher than a user specified threshold are marked as the keypoints of the point cloud.Figure 4 shows the NARF keypoints in an example scene.The Scale Invariant Feature Transform (SIFT, (Lowe, 2004)) originally developed for 2D images was adapted by the community of the Point Cloud Library (PCL) to 3D point clouds (Rusu and Cousins, 2011) by replacing the role of the intensity of an pixel in the original algorithm by the principal curvature of a point within the 3D cloud.A good feature descriptor should be robust against noise, fast to compute, fast to compare, and invariant against rotation and translation of the point cloud (Lowe, 2004).
In (Arbeiter et al., 2012) the object recognition capabilities of different feature descriptors are evaluated.The work of (Salti et al., 2011) shows that not all feature descriptor and keypoint detector combinations deliver good results.In this paper Point Feature Histograms (PFH) and Signature of Histograms of Orientations (SHOT) with their variants are used.They are chosen because they represent common feature descriptors and behave differently in computation time and accuracy.The Point Feature Histograms (PFH) descriptor was developed in 2008 (Rusu et al., 2008).Besides the usage for point matching, the PFH descriptor is used to classify points in a point cloud, such as points on an edge, corner, plane, or similar primitives.Figure 7 shows an overview of the PFH computation steps for each point p in the point cloud.angle between its normal and the connecting line between the point pair ps and pt.If n s/t is the corresponding point normal, the Darboux frame u, v, w is constructed as follows:

Point Feature Histograms
Three angular values β, φ, and θ are computed based on the Darboux frame: where d is the distance between ps and pt.The PCL community also developed the PFHRGB variant (PCL, 2013), which additionally uses the RGB values of the neighboring points to define the feature descriptor.The number of bins of the histograms is doubled.The first half is filled based on the point normals.The second half is computed similar to the description above, but uses RGB values of the points instead of their 3D information.

Signature of Histograms of Orientations
In 2010 an evaluation of existing feature descriptors led to the conclusion, that one of the hardest problems of the evaluated descriptors is the definition of a single, unambiguous and stable local coordinate system at each point (Tombari et al., 2010).Based on this evaluation the authors proposed a new local coordinate system and the Signature of Histograms of Orientations (SHOT) as a new feature descriptor (Tombari et al., 2010).An overview of the computation steps for each point p in the point cloud is visualized in Figure 9.The first three steps consist of the computation of a local coordinate system at p.The n neighbors pi of a point p are used to compute a weighted covariance matrix C: where r is the radius of the neighborhood volume.
The direction of the Z-axis is similarly defined.The direction for the Y -axis is determined via the cross product between X and Z.This local coordinate system is used to divide the spatial environment of p with an isotropic spherical grid.Color-SHOT (Tombari et al., 2011) includes also color information.Each cell in the spherical grid contains two local histograms.
One for the angle between the normals and one new histogram, which consists of the sum of absolute differences of the RGB values between the points.

PERSISTENT FEATURES
Persistent features are another approach to reduce the number of points in a cloud.It is published together with the PFH descriptor and is strongly connected to its computation (Rusu et al., 2008).
In a first step the PFH descriptors are computed at multiple increasing radii ri for all points of the point cloud.A mean histogram µi is computed from all PFH point histograms at radius ri.
For each point histogram of radius ri the distance to the corresponding mean histogram µi is computed.In (Rusu et al., 2008) the authors propose to use the Kullback-Leibler distance as a distance metric.However, for this paper the Manhattan distance is used, since it led to better experimental results within the here discussed application scenario.Every point whose distance value in one radius is outside the interval of µi ± γ • σi is marked as a persistent point candidate.The γ value is a user specified value and σi is the standard deviation of the histogram distances for radius ri.The final persistent point set contains all points, which are marked at two radii ri and ri+1 as persistent point candidates.which depend on the silhouette of an object.In contrast, 3D-SIFT uses scale space extrema to find points, which are likely to be recognized even after viewpoint changes.The user specified parameters of both methods are empirically set to values, which led to reasonable results in all test cases.The 3D-SIFT detector has more parameters and is slightly more complex to configure.
The resulting set of keypoints differ significantly.Examples of NARF and 3D-SIFT keypoints are shown in Figure 4 and Figure 6, respectively.In both cases the same input point cloud with 111, 896 points is used.Table 1 summarizes, that the 3D-SIFT detector marks more keypoints than the NARF detector.Both keypoint detectors mostly ignore the center of the table plane and mark many points at the table border.In addition, 3D-SIFT strongly responds to object borders and major surface changes, while NARF mostly reacts to keypoints near object borders.The 3D-SIFT detector has a considerably higher runtime than the NARF detector.The median runtime of NARF for this point cloud is about 0.05 sec, while 3D-SIFT needs about 2 sec.

Quantitative Comparison of Keypoint Descriptors
The PFH and SHOT descriptor mainly differ in their way to define local coordinate systems.Both feature descriptors have in common, that they use the differences of point normals in the neighborhood to create a feature histogram as descriptor.The free parameters are set to the proposed values of the original publications of the algorithms.
The radius of the neighboring point search is especially crucial.
If the point cloud contains a lot of noise, then the radius should be increased to obtain stable results.But the neighbor search radius has a significant impact on the runtime of the feature descriptor.The graph in Figure 12   For a fast matching procedure, less bins are preferable.The FPFH descriptor is best suited for time-crucial applications, since it is not only fast to compute, but also consists only of 33 histogram bins per point.

Comparison of the Detector-Descriptor-Combinations
All possible keypoint detector and descriptor combinations are used to fuse the point clouds of all described test sets by the above explained fusion pipeline.All tests are performed with and without the persistent feature computation as well as with 10 and 100 iterations of the ICP algorithm.The persistent features are computed using the FPFH descriptor at three different radii of 3.25cm, 3.5cm, and 3.75cm and an γ = 0.8.The variation of the number of ICP iterations allows to observe the impact of ICP and to deduce the quality of the feature-based alignment.
All remaining parameters of the point cloud fusion algorithm are set to reasonable values for each test scene.Due to the reason that the Happy Buddha scene contains only a single object with little noise, the distance, plane, and outlier filter are skipped for the corresponding test sets.As the PFHRGB and Color-SHOT descriptor use color information, which are not available for the Happy Buddha scene, the total number of tests in the corresponding cases is six.In all other cases, all ten tests are performed.
For the first part of the following discussion, a rather subjective performance measure is used: A result of a test is classified as correct, if it contains no visible alignment error.The following tables contain the aggregated results of the performed tests.Tables 2-3 show the percentage of correct results with 10 and 100 ICP iterations, respectively, as well as the overall time of the fusion process.The top part of each table contains the results without and the lower part with the computation of persistent features.Each row contains the results of a different feature descriptor and each column represents the results with the specified feature detector.Column "None" represents the case, when no keypoint detector is applied.
The most important conclusion of these results is, that the proposed fusion pipeline using the FPFH descriptor with no feature detection and without the computation of persistent features outperforms all other tested combinations.This combination is able to produce correct results for all test sets with 100 iterations of ICP and has a comparatively low computation time.At almost all cases, the combinations using no feature detector have a better or at least equal success rate than combinations using NARF or 3D-SIFT.There are different reasons possible why feature detectors lead to inferior results.One important aspect is that the In general, the NARF detector does not seem to be a good keypoint detector for point clouds of a Kinect sensor or of point clouds that contain only a single object.NARF uses the borders of objects to detect feature points, but a Kinect sensor generates blurred object boundaries due to its working principle and a rotation of a single object can result into major silhouette changes.This leads to very different keypoint positions between the point clouds and to a bad initial alignment of the point clouds.Increasing the number of ICP iterations sometimes leads to the correct result.Using the optional persistent feature step leads in nearly all combinations with the NARF detector to better or at least equal success rates.The reason is that the persistent feature computation generates more homogenous object borders, which lead to less different NARF keypoint positions between the point clouds.
Using a descriptor of the PFH family delivers for most combinations a higher success rate than combinations using SHOT or Color-SHOT.Based on the results of the Happy Buddha scene it can be concluded that at least the FPFH descriptor and PFH descriptor tolerate larger viewpoint changes than the SHOT descriptor.In contrast to the PFH and FPFH descriptor, the SHOT descriptor is not able to produce a correct result at the Buddha-0-72 test set for any combination.From the used test sets, the Buddha-15pc is the most challenging.Tables 2-3 also contain the mean runtimes in seconds of all tests with 10 and 100 ICP iterations, respectively.As test platform a Windows 8 64bit system with a Xeon E3-1230V 2 processor was used.The values in these tables are intended to show the relative runtime changes between the different test parameters.Not apparent in these tables is the increase in computation time due to a larger number of point clouds or more points per point cloud.
The persistent feature computation is able to notably decrease the computation speed, especially if there are many points and a feature descriptor with many dimensions is used.This is the case at pipeline combinations, which are using no keypoint detector or the 3D-SIFT feature detector together with the PFHRGB, SHOT, or Color-SHOT descriptor.Under these circumstances the point descriptor computation and the point descriptor matching between the point clouds need more computation time than the persistent feature computation.But except for several of the NARF detector combinations and for the combination of the Color-SHOT descriptor without a feature detector, the computation of persistent features mostly decreases the success rate.The choice of an additional persistent feature computation step is a trade-off between a faster computation speed and a higher success rate.A similar trade-off is made by the selection of the number of ICP iterations.There is no test case, where 100 ICP iterations led to a wrong result, if the result was already correct at 10 ICP iterations.But there are many cases with an increased success rate with more ICP iterations.Especially the combinations using the NARF keypoint detector and without the persistent feature computation benefit from more ICP iterations.The average runtimes in nearly all cases show, that more dimensions of the used point descriptor lead to a higher computation time.One reason is, that a high dimensional k-d tree probably degenerates and loses its speed advantage.For example, Color-SHOT uses 1280 dimensions to describe a point and most of the computation time is spent at the correspondence point computation.Therefore, a keypoint detector or the persistent feature computation can reduce the computation time for point descriptors with many dimensions.

Scene max. MSE max. MCE Table-2ks
< 0.01% 0.08% In order to obtain a more objective impression about the accuracy of the obtained results, all cases defined above as "correct" are compared to reference results.This comparison allows qualitative statements about the fusion accuracy.

CONCLUSION
This paper discussed the benefits and limits of different keypoint detector and descriptor combinations for the task of point cloud fusion.An automatic multi point cloud fusion processing chain computes a transformation for each point cloud, which is based on a coarse feature-based alignment and a subsequent fine alignment by ICP.
Prior to the fusion step, all point clouds are pre-processed by different filters.The NARF or 3D-SIFT keypoint detectors optionally select a subset of points, which are the best representatives of an underlying geometric feature.The optional computation of persistent features reduces the set of points further by keeping only points that lie on exceptional geometry.Point descriptors are computed for the selected subset of the point cloud.This paper uses point descriptors from the PFH and SHOT families.
Evaluation results show that the proposed pipeline is able to produce correct fusion results at complex scenes and 360 • object scans.The best performing pipeline configuration with respect to the tradeoff between accuracy and speed uses the FPFH descriptor and no additional persistent feature or keypoint computation.
Both pipeline steps mainly lead to worse fusion results, but reduce the computation time if a complex feature descriptor is used.
A reason for the decreased accuracy is that the heuristic of the global point cloud graph generates better point cloud pairs, if more points are available.If a point cloud contains less 3D information (ie.points), it is harder to match it with another point cloud based on similarities between the respective 3D structures.
The NARF keypoint detector converts the point cloud into a range image and uses object borders to find keypoints.NARF keypoints are fast to compute, but the resulting keypoint subset is too small in order to use them in the global point cloud graph.Additionally, the detected NARF keypoints are unstable because the Kinect sensor produces blurred object borders due to its working principle.As result, the NARF keypoint detector is only usable to align one point cloud pair with enough ICP iterations.Using more than two point clouds as input for the point cloud fusion process with the NARF keypoint detector leads only in rare cases to a correct result.In comparison to the NARF keypoint detector, the 3D-SIFT keypoint detector needs more computation time, but also leads to better fusion results.It results in a larger set of keypoint than NARF, which helps the heuristic of the global point cloud graph to find the best point cloud pairs.Nevertheless, the point cloud fusion process is more often successful, if the keypoint detection is skipped during the point cloud preparation.
Point descriptors of the PFH family are faster to compute and more robust against viewpoint differences than descriptors of the SHOT family.The PFH family also delivers more reliable information about the most overlapping point cloud pairs during the global point cloud graph computation.As a consequence, the SHOT descriptors lead to inferior fusion results.The PFHRGB and Color-SHOT point descriptor are variants, which use the geometry as well as the point colors to describe a point.If the input point clouds are captured by different Kinect sensors and a colorbased descriptor is used, the RGB cameras should be calibrated beforehand.Otherwise, too different point cloud colors lead to wrong fusion results.

Figure 1 :
Figure 1: Point cloud fusion point cloud fusion: Two or more point clouds are acquired from the same scene but provided within their own, local coordinate system as illustrated in Figure 1(a).The system automatically performs a chain of successive pairwise registrations and thus aligns all point clouds into a global coordinate system (see Figure 1(b)).The resulting fused point cloud can then be used in subsequent tasks like surface reconstruction (see Figure1(c)).In order to registrate two point clouds a rigid transformation consisting of a translation and rotation is computed by a state-of-theart combination of a keypoint-based coarse alignment(Rusu et al., 2008) and a point-based fine alignment(Chen and Medioni, 1992, Besl and McKay, 1992, Rusinkiewicz and Levoy, 2001).

Figure 7 :
Figure 7: PFH descriptor computation for one point

Figure 8 :
Figure 8: Darboux frame between a point pair [Rus09] The three angular and the one distance value describe the relationship between the two points and the two normal vectors.These four values are added to the histogram of the point p, which shows the percentage of point pairs in the neighborhood of p, which have a similar relationship.Since the PFH descriptor uses all possible point pairs of the k neighbors of p, it has a complexity of O(n • k 2 ) for a point cloud with n points.The FPFH variant is used to reduce the computation time at the cost of accuracy(Rusu, 2009).It discards the distance d and decorrelates the remaining histogram dimensions.Thus, FPFH uses a histogram with only 3 • b bins instead of b 4 , where b is the number of bins per dimension.The time complexity is reduced by the computation of a preliminary descriptor value for each point p by using only the point pairs between p and its neighbors.In a second step it adds the weighted preliminary values of the neighbors to the preliminary value of each point.The weight is defined by the Euclidean distance from p to the neighboring point.This leads to a reduced complexity of O(nk) for the FPFH descriptor.

Figure 9 :
Figure 9: SHOT descriptor computation for one point For each point pi in a cell the angle ξi = pi • p is computed between the points normal and the normal of p.The local distribution of angles is subsequently described by one local histogram for each cell.If the spherical grid contains k different cells with local histograms and each histogram contains b bins, the resulting final histogram contains k•b values.These values are normalized to sum to one in order to handle different point densities in different point clouds.

Figure 10
Figure 10 shows an example of the resulting point cloud after a persistent feature computation.No additional filtering steps are used in this example and all 111, 896 points of the original point cloud are used.The FPFH descriptor was computed at three different radii of 3.25cm, 3.5cm, and 3.75cm and γ = 0.8.

Figure 10 :
Figure 10: Point cloud before (left) and after (right) the persistent features computation Figure 11: Example datasets

Figure 12 :
Figure 12: Runtime of point descriptors

Table scenes
These three test sets contain the point cloud scan at the 0 • position of the rotating table and the point cloud at the 24 • , 48 • , and 72 • position, respectively.The test set Buddha-15pc contains 15 point clouds from a 360 • rotation.Each point cloud of the Happy Buddha laser scan contains about 240, 000 points.
6.1 Quantitative Comparison of Keypoint DetectorsBoth keypoint detectors presented in Section 3. use different approaches to define interesting points in a point cloud.The NARF detector converts the captured point cloud into a 2D range image and tries to find stable points, which have borders or major surface changes within the environment.This results in keypoints, compares the median runtime values of 20 iterations on a system with a Xeon E3 − 1230V 2 processor.The input point cloud is a downsampled version of the table scene point cloud in Figure 4 containing 11, 276 points.As the diagram shows, the runtime of the PFH and PFHRGB algorithm increases quadratically with the search radius while it only grows linearly for FPFH, SHOT, and Color-SHOT.Another important characteristic of the presented feature descriptors is the number of bins of the histogram for each point.If the histogram of a point contains

Table 3 :
Runtime and subjective results (100 ICP iterations)global point cloud graph used in the fusion process produces better point cloud pairs for the pairwise registration, if more points are used.Especially the 3D-SIFT feature detector performs well at test sets with only two point clouds, but failed at nearly all test sets with more than two point clouds.The NARF detector shows a similar effect.
The algorithm needs to create 14 point cloud pairs from the 360 • scan in order to fuse all point clouds.If one point cloud pair does not have a sufficiently large overlap, the fusion result will be incorrect.The FPFH and PFH descriptor in combination with no persistent features computation and without a keypoint detection are able to find all 14 directly adjacent point cloud pairs and produce a correct fusion result.The SHOT descriptor leads to one not directly adjacent pair of point clouds, but still produced the correct result with 100 ICP iterations.All other combinations do not lead to adequate point cloud pairs and produced wrong fusion results.The results of the Table-2ks and Table-4ks test sets reveal one problem of the PFHRGB and the Color-SHOT feature descriptors.These sets contain significant color differences between the point clouds, which were acquired by different Kinect devices.Except for one combination, the PFHRGB and the Color-SHOT feature descriptors produced incorrect results.The feature descriptor variants, which only use the geometry but no color information, delivered correct results for more combinations.As a consequence, the RGB cameras of different Kinect devices should be calibrated, if the feature descriptor uses color information.

Table 4 :
Maximum error of correct results Table4shows the largest mean squared error (MSE) and the largest maximum correspondence error (MCE) of the correct results of each test set.They represent the mean and the maximum nearest point neighbor distance between all points of the fusion result and the reference result and are expressed as percentage of the maximum point distance of the reference result.This enables an easier comparison of the individual results.At the Buddha scene, the public available reference transformations from the Stanford 3D Scanning Repository are used to create the reference result.The reference results of the Station and Table scene are created by manual selection of corresponding points and additional ICP and global registration steps using MeshLab.Compared to the error values of the other test sets, the correct Table scene test results contain a large error with a MSE up to 0.06% and a MCE up to 3.54%.This is caused by the large amount of noise within this test scene, which allows a wider difference for correct results.The highest error of the remaining test results has a MSE value less than 0.01% and a MCE of 0.05%.