REASSEMBLING 3D THIN FRAGMENTS OF UNKNOWN GEOMETRY IN CULTURAL HERITAGE

: Many fragile antiques had already been broken upon being discovered at archaeology sites. The fragments of these objects cannot be effectively interpreted and studied unless they are successfully reassembled. However, there still exists many problems in the reassembly procedure in existing methods, such as the numerical instabilities of curvature and torsion based methods, the limitation of geometric assumption, and the error accumulation of the pairwise matching approach, etc. Regarding these problems, this paper proposed an approach to match the fragments to each other for their original 3D reconstruction. Instead of the curvatures and torsions, the approach is based on establishing a local Cartesian coordinate at every point of the 3D contour curves. First of all, the 3D meshes of the fragments are acquired by a structure-light based method, with the corresponding 3D contour curves extracted from the outer boundaries. Then, the contour curves are matched and aligned to each other by estimating all the possible 3D rigid transformations of the curve pairs with our defined local Cartesian coordinates, and then the maximum likelihood rigid transformations are selected. Finally, a global refinement is introduced to adjust the alignment errors and improve the final reassembling accuracy. In addition, experiments with two groups of fragments suggest that this approach cannot only match and align fragments effectively, but also improve the accuracy significantly. Comparing with the original 3D model acquired before being broken, the final reassembling accuracy reaches 0.47 mm.


INTRODUCTION
Large quantity of fragile antiques may be found at archaeological sites.And these cultural heritages are often discovered in broken form due to long time burying and their fragility.They must be reassembled before any other academic procedures.Therefore the reassembling of 3D fragments can join its force in cultural heritage study.Unfortunately, it is very challenging to reassemble such fragments due to their poor thickness and lacking of matching surface.Thus only the complementary among the contour curves of the fragments can be utilized for the matching and the aligning of the adjacent shards.Although there are many methods dealing with such challenge, they have some disadvantages nonetheless.For example, in order to make simplification and improve robustness, some authors (Cooper et al. 2001;Willis and Cooper 2004;Kampel et al. 2002;Razdan et al. 2001) imposed the assumption of a global model with known geometry to the original object, i.e. the object is axially symmetric, and the assumption may not be fulfilled in many cases.Methods (Kong and Kimia 2001;Üç oluk and HakkI 1999;Gruen and Akca 2005;Oxholm and Nishino 2013) rely on representing the curve with shape feature strings, such as curvatures and torsions are fast, the main problem is that the computation of curvature and torsion involves up to third-order derivatives, which is not robust.In addition, the alignment after the matching necessarily suffers from error accumulation, because it usually relies on pairwise approach.Even the improvement (Cooper et al. 2001;Kong and Kimia 2001) of pairwise approach, which delays the alignment phase until clusters of three matching fragments have been found, cannot improve the alignment accuracy due to local optimization.Willis and Cooper (2008) has already provided a detail survey of the state-of-the art in both automatic and semiautomatic ancient artifact reconstruction systems, including almost the current 2D and 3D artifact reconstruction methods and their extensions.However, as is pointed out by Willis and Cooper (2008), there are still some aspects of these methods needing to be improved in the future, such as the efficiency, the incorporating of more available information, the development of an information-theoretic basis for searching compatible matches etc.
Besides, some other related works are as follows.Papaioannou and Karabassi (2003) combined curve matching techniques with a surface matching algorithm to estimate the positioning and respective matching error for the joining of three-dimensional fragmented objects.And Filippas and Georgopoulos (2013) presented a Fragmatch Algorithm that accepted as input data point clouds, i.e.X, Y, and Z for each point of the broken surfaces and not of the whole fragment.Huang et al. (2006) also presented a system for automatic reassembly of broken 3D solids, in which they developed several new techniques in the area of geometry processing, including the novel integral invariants for computing multi-scale surface characteristics, registration based on forward search techniques and surface consistency, and a non-penetrating iterated closest point algorithm.What's more, Brown et al. (2008) focused on the specific problem of documenting and reconstructing fragments of wall paintings from the site of Akrotiri on the volcanic island of Thera.They take advantage of the fragments' flat front surfaces to limit their search space to planar transformations.
In general, the fragments found at archaeological sites can be divided into two categories: thick and thin fragments.And the research on fragment reassembling can also be divided into aiming at thick and aiming at thin fragments accordingly.This paper focuses on the assembling of 3D thin shards in archaeology sites, which cannot be converted to 2D problem.And an approach is put forward to match and align the shards for recovering their original 3D shape, which is based on establishing a local Cartesian coordinates at every point of the contour curves.The approach is free from any geometry assumption of original shape and computation of curvatures and torsions.The initial matching and alignments of the proposed approach are both based on estimating the transformations of the fragments pairwise.Besides, similar to Oxholm and Nishino (2013), the alignments are globally refined by utilizing the core functionality of Besl and McKay's iterative closest point (ICP) algorithm (Besl and McKay 1992).And instead of Oxholm and Nishino's original approach that iterates between optimizing the transformations and updating the correspondences of a subset (Oxholm and Nishino 2013), this paper takes advantage of the principle of Bundle Adjustment (Triggs et al. 2000) in Photogrammetry, which optimizes the transformations of all the fragments in each iteration.
In the following sections, this paper will introduce the proposed approach according to procedures in the flow chart of Figure 1, including data acquisition, initial matching and alignment, global refinement, followed by experiments and accuracy assessment.

DATA ACQUISITION
Three-dimensional measurement and reconstruction of fragments, which is the premise and foundation to automatically reassemble fragments in cultural heritage research, is base on accurate extraction of the fragments' geometry information.With its abilities to acquire reliable, precise, and dense point clouds at a low cost even when objects are texture lack, the structure-light based method is chosen in this paper to extract the fragments' geometry information.Since there are many existing structure-light based methods that can be used, this paper just utilizes the structure-light based process in the literature (Zheng et al. 2012) to extract the fragments' geometry information.According to the requirement of this paper, two main operations of the structure-light based process are chosen: one is the 3D point cloud acquisition and the other is 3D mesh reconstruction, which are shown in Figure 2 curves can be easily extracted from the outer contours of the 3D meshes, as is shown in Figure 2(d).What's more, we just make use of the original points instead of the curvatures and torsions to represent the contour curves, thus avoiding the complicated and unreliable computation of the curvatures and torsions.
Although our approach only needs the contours to match the fragments, we need the around points to estimate the normal of each point of the contours as is illustrated in next section, and we also need all the points to recover the original 3D shape after the matching.This is why we choose the structured light scanner for the data acquisition in the experiment.In fact, if possible, other 3D scanner can be also chosen to acquire the data.

INITIAL MATCHING AND ALIGNMENT
To measure the matching degree between one 3D contour curve and another, a similarity measurement is developed in this paper to find initial matching pairs of the fragments.With the similarity measurement, we utilize a pairwise strategy to find the initial matching and alignment of the fragments, i.e. we calculate the similarity measurement for every pair of the contour curves of the fragments, and then construct a graph to encode matching degrees of all pairs of the contour curves.In the graph, nodes represent individual contour curves of the fragments, and each pair of the nodes is connected by an edge weighted by the matching degree of the corresponding two contour curves.As a result, we can find the maximum likelihood matching of the fragments through generating the maximum weight spanning tree of the graph, and further more estimate the initial transformations of the fragments according to the maximum weight spanning tree.
When 3D contour curves are extracted successfully, every curve point is associated with a unique normal, which is estimated early during the course of triangulated mesh reconstruction, as is shown in Figure 3(a).In order to establish local Cartesian coordinates for the matching and alignment, the tangent at a curve point is further estimated by polynomial fitting with curve points located in the neighborhood of the point, and the tangential directions of one contour curve are chosen to make sure that the contour curve goes counterclockwise around the region of the corresponding point cloud, as is shown in Figure 3(b).What's more, the local Cartesian coordinate system can be successfully established by the normal and the tangent at the corresponding point, i.e. the position of the point is defined as the coordinate origin, the tangent is defined as the x axis, the normal is defined as the z axis, and the y axis is finally defined by the right-hand rule, as is shown in Figure 3(c).
Once the local Cartesian coordinates is established, the initial matching of fragments can be found pairwise by the local Cartesian coordinates of the contour curves.As is known, when we find a pair of matching points located in two different contour curves, we can estimate the three-dimensional rigid transformation between the two curves according to the local Cartesian coordinates of the matching points.Our pairwise matching algorithm is based on this principle, i.e. we define the similarity measurement from one fragment contour curve to another and design the initial matching and alignment of the fragments as follows: (1) For a pair of fragments, we assume two contour curve points of the pair matching to each other, and then a rigid transformation is estimated according to the corresponding local Cartesian coordinates.
(2) Then the two contour curves are converted to the same coordinate system by the estimated rigid transformation.
(3) For any point i P located in one of the converted curves, we search the nearest point (4) The arc length along the trajectory of the inliers signed i L is then estimated after the outliers and the inliers are distinguished.
The similarity measurement proposed in this paper is defined by the equation (1) as follows: where c is a constant, i.e. 0.3 in this paper, and N is the number of the inliers.
(5) The matching degree signed D of this pair of fragments is computed by finding maximum i G value of the contour curve pair as the follow equation: It should be noted that when we find the maximum i G value, we can also acquire the maximum likelihood alignment of the curve pair because of step (2) above.
(6) At last, we encode the matching in a graph, in which the node represent individual contour curve of fragments, and each pair of the nodes is connected by an edge weighted by the matching degree.We need and only need 1 n  pairs of pairwise matching to align the fragments to a common coordinate system, where n is the number of the fragments of an object.In other words, in order to find the maximum likelihood matching and alignment of the fragments, the maximum weight spanning tree of the graph is generated, and then the transformations of the fragments to a common coordinate system is further estimated by the spanning tree.An example of the matching and alignment corresponds to the data in Figure 3 above is shown in Figure 4(a) through (c).

GLOBAL REFINEMENT
As is shown in Figure 4, due to pairwise strategy approach is employed, the matching and alignment suffers from serious error accumulation.For the sake of a higher accuracy, the alignment errors are iteratively and globally adjusted by a least square method, which is similar to the Bundle Adjustment in Photogrammetry.
Suppose i R and i T are respectively the rotation matrix and the translation vector of the rigid transformation corresponding to the i-th fragment, which are estimated by the initial matching and alignment proposed in above section, i.e.In order to adjust the alignment errors effectively, it needs to be capable of searching the nearest points to a given point from other contour curves according to the current alignment.

Suppose that (i) l
P is one point in the i-th contour curve of the fragments, we can convert it to the coordinate system of the j-th contour curve as the follow equation:   In this case, the nearest point to (j) l P is easy to be found in the j-th contour curve according to the coordinate system of the j-th fragment.Then our least square method is equivalent to searching the minimum of the function: (1 ) where (0,1] w , (i) l P and (j) l P are the nearest pair of points in the i-th and the j-th fragment respectively, The main steps of our least square method for globally adjusting the alignment errors are as follows: (1) Updating the matching.Given a certain threshold, followed by searching the nearest points to every point (2) Error Adjustment.Employing the least square adjustment to search the minimum of equation ( 6) with the updated nearest point pairs.
(3) Checking the adjusted rigid transformations, if their variations are all small enough in comparison to the last iteration ones, the global refinement of the alignment is finished.Otherwise, go to (1).This global refinement can effectively adjust the matching and alignment errors, and significantly improve the alignment accuracy, as is shown in Figure 4(c) and (e). of our global refinement.As the original three-dimensional geometry model of Data Ⅱ has already been constructed before the object is broken, we will make use of Data Ⅱ to assess the accuracy quantitatively, as is shown in Figure 8.In order to assess the real accuracy of the proposed approach, we firstly make use of the ICP algorithm (Besl and McKay 1992) to align the 3D model reassembled by the proposed approach with the original 3D model acquired before being broken, and then the errors are calculated by estimating the distances from each point in the reassembled model to the original model's surface.As a result, we find that the root mean square (RMS) of the errors of Data Ⅱ reaches 0.47 mm.Additionally, the distribution of the errors is elaborately displayed in Figure 8, through which we can obviously find that the error is not a normal distribution.The reason is that the contour curves can provide very little information to the reassembling of the fragments.
Figure 8.Alignment Errors of DataⅡ

CONCLUSIONS AND FUTURE WORK
We have proposed an approach to reassemble the broken 3D thin fragments found at archaeology sites.In order to avoid the complicated and unstable computation of the curvatures and torsions for matching the fragments' 3D contour curves, this approach defines and calculates a local 3D Cartesian coordinate at every contour curve point, and develops a method to find the maximum likelihood matching pairs of the contour curves.Unlike other current methods, this work doesn't rely on any assumption of known geometry of original objects.Upon the maximum likelihood matching, the initial alignment of the fragments is also accomplished simultaneously.However, the initial alignment suffers from serious error accumulation.In order to avoid this, we introduce a global refinement method to adjust the errors and improve the reassembling accuracy.
Finally, experiments with two groups of fragments suggest that the proposed approach cannot only match and align fragments but also improve the accuracy significantly.To compare the reassembling 3D model with the original one, the final reassembling accuracy reaches 0.47 mm.Therefore, the proposed approach is effective and accurate enough to be suitable to reassemble the thin 3D fragments in cultural heritage.
However, further research is still needed if applying to more other realistic reassembling projects.For example, because the computation will increase with the increasing number of the fragments, we need to improve the pairwise initial matching and alignment strategy.And in case of the fragments are complete unorganized, it's necessary to incorporating other information, such as the texture and the colours, to determine whether the fragments can be reassembled or not, and to adapt to the case of the fragments of more than one objects.In a word, this method still needs improvement for more complex real conditions in the future.
(b) and Figure 2(c) respectively.After the 3D mesh reconstruction, the contour ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5, 2014 ISPRS Technical Commission V Symposium, 23 -25 June 2014, Riva del Garda, ItalyThis contribution has been peer-reviewed.The double-blind peer-review was conducted on the basis of the full paper.doi:10.5194/isprsannals-II-5-393-2014 (a) Original Fragments of One Broken Bowl (b) Point Clouds of the Fragments (c) Triangulated Meshes of the Fragments (d) Contour Curves from Triangulated Meshes Figure 2. Geometry Information of One Broken Bowl Acquired by Structure-Light System

Figure 3 .
Figure 3.To Establish the Local Cartesian Coordinates is calculated.Finally, we make use of a fixed threshold to distinguish the outliers from the inliers of the matching.The outliers and the inliers are respectively signed O and I in the follow sections.
corresponding normal vectors.

P
in one of the contour curves from other contour curves.At last, some of the nearest point pairs are eliminated when their distances to each other are larger than the given threshold.
(a) Some Pairwise Matching and Alignment (b) Initial Curve Result (c) Initial Point Cloud Result (d) Effect of Globally Refined Curves (e) Effect of Globally Refined Point Clouds Figure 4.An Matching and Alignment Example 5. EXPERIMENTS AND ACCURACY ASSESSMENT Except for the broken bowl in the above sections, this paper takes another two groups of thin fragments to verify the effectiveness of the proposed reassembly approach.As is shown in Figure 5, one of the groups contains 4 pieces of fragments (Data Ⅰ), and another contains 12 pieces of fragments (Data Ⅱ).What's more, the experiment results of the two groups of data are shown in Figure 6 and Figure 7 in detail.The two Figures suggest that the proposed approach is effective and the accuracy can be significantly improved due to the introduction This contribution has been peer-reviewed.The double-blind peer-review was conducted on the basis of the full paper.doi:10.5194/isprsannals-II-5-393-2014

Figure
Figure 5. Experiment Data