A TARGET AWARE TEXTURE MAPPING FOR SCULPTURE HERITAGE MODELING

In this paper, we proposed a target aware image to model registration method using silhouette as the matching clues. The target sculpture object in natural environment can be automatically detected from image with complex background with assistant of 3D geometric data. Then the silhouette can be automatically extracted and applied in image to model matching. Due to the user don’t need to deliberately draw target area, the time consumption for precisely image to model matching operation can be greatly reduced. To enhance the function of this method, we also improved the silhouette matching algorithm to support conditional silhouette matching. Two experiments using a stone lion sculpture of Ming Dynasty and a potable relic in museum are given to evaluate the method we proposed. The method we proposed in this paper is extended and developed into a mature software applied in many culture heritage documentation projects.


Background
Sculptures are one of the most important forms of cultural heritage with complex geometric structure and intricate texture information (Fontana et al. 2002).Producing high precision 3D digital sculpture models is a primary work for cultural heritage protection and visualization (Ikeuchi et al. 2007;Chow and Chan 2009).Laser scanning is a reliable and efficient way, which utilizes structured light or time of flight (TOF) technique to acquire high precision geometric information (Pavlidis et al. 2007).Though camera is settled to some laser scanner system, additional images collected from elaborately selected view angle and fine light condition are necessary to achieve high fidelity texture reconstruction (Andrade et al. 2012).
In order to create model with high fidelity texture, a precise image to geometric model mapping is a necessary step.For sculptures with complex structure, this work will be challenging task (Arbace el al. 2012).Texture mapping is a kind of 2d image to 3d model registration problem.The key point of this problem is finding correspondence between two data sources.Usually, a few pairs of control points are collected from image and geometric model to solve the image orientation parameters, as well as inner parameters.Image can be precisely mapped to model using the orientation parameters.This method works well for single image to geometric model registration.Sometime, there are a bundle of images with overlap enough to satisfy photogrammetry requirements.Then, texture to model relation can be solved by photogrammetric or computer vision solution (Tzur and Tal , 2009).
However, for many sculptures, it is not easy to pick control points precise enough to create accurate parameters by solving the collinearity equations, thus resulting to mis-matching problem.Silhouette matching is another way for image to model registration which was applied to texture mapping in (Hendrik P. A. et al., 2001).But this method requires that the image should has very clean background so as to easily detect the silhouette lines.It is suitable for processing texture mapping of portable sculptures in the museum, where it is easy to set up a single colour background while collecting images.For the sculpture in natural environment or importable objects, this method cannot be applied, due to the requirement of clean background image cannot be satisfied.In many modelling cases, the sculpture texture images that collected by photographer contain both target object information and complex background information.Intelligently extract the target area from source texture image and mapping it to geometric model could greatly save processing time.
Figure 1 (a) shows a stone lion sculpture of Ming Dynasty in the natural environment with grass and bushes around.Figure 1 (b) is 3d geometric model of this stone sculpture created using Geomagic software.The point cloud data was collected using Z+F imager 5006 laser scanner from different angles.During geometric model editing processing, we removed all the noise points from laser scanner and only kept the data of sculpture.To solve the texture mapping problem, we proposed a target aware image to model registration method using silhouette as the matching clues.User only need to put geometric model close to the target area in image, the silhouette can be automatically extracted and applied in image to model matching.Due to the user don't need to deliberately draw target area, the time consumption for precisely image to model matching operation can be greatly reduced.
The paper is organized as following: In the second section, methodology will be explained.The method of target area detection and one-cut optimization will be discussed in this section.The second part of this section explains the procedure of texture to model matching using silhouette information.In section three, two experiments will be introduced with analysis to evaluate the effectiveness of this algorithm.And, a detailed discussion will be given at the end.

Overview
In this study, we assume that precise geometric model has been collected using handhold laser scanner or structure lights system.The goal of this study is to reduce user interaction in texture mapping processing, especially in identifying the target.
As we know, fully automatic target detection without any predefined knowledge is a very challenging task.To simplify this problem, we utilize 3d geometric model as the clue for target detection.We developed a user interface using OpenGL which is able to show texture image and 3d geometry data simultaneously.
User can move 3d model to image close to the target area.Then, target detection using one-cut algorithm will be applied to find the target area, and sketch precise boundary of detected target area.The boundary of target is finally utilized as features for texture to model registration, using silhouette texture registration algorithm.

Target Detection using One-Cut
To decrease the level of difficulty, we ask the user put 3d model to image where the target located at.To make this task simple enough and can be finished in a few second, we visualize the 3d geometric model in 3d space, as well as image.From OpenGL system, we can find 3d model area and extract silhouette information using depth image in OpenGL visualization buffer.
Since we don't require user giving precise position, the operation can be done with very short time.

Experiment Environment:
In OpenGL system, the final visualization result in the screen can be seemed as a photo comes from projection of 3d model to plane.If we rotate the model, we can get the depth image of each frame with given 3d model, according to the camera parameters in OpenGL.Figure 2 shows the projection relation between 3d model and frustum.If we change the parameters of OpenGL, the depth image will change precisely follow the parameter we give.Figure 3 is the results of put geometric model into image in our experimental 3d visualization system.

Seed Area Extraction:
For a given 3d model in the depth image, the background area has a same invalid value, while the front parts contain meaningful number.We make the depth image into a binary image to find model block.It is very easy to find the target area in depth image, leading to a result as shown in Figure 4.For simple object, we can combine this binary image with colourful texture image to find out the target area, denoted in figure 4 white area as front and black as back ground.But the 3d model sometimes contains a few holes, using the binary image block directly to label the texture image into front and back is not sufficient.Meanwhile, 3d model cannot overlap all the target image area for we didn't require a precise 3d model position setting.
To make the algorithm robustly processing complex situation, a post processing using erosion operation is conducted, as shown in Figure 5. Two steps are involved in our binary image processing: (1) apply erosion operation at different level to binary image, getting two result images (left, middle); (2) apply exclusive-or to the two result images and get result (right) as the seed area for target detection.
Figure 5. Target Seed Area Extraction from 3D data.

Seed Background Extraction:
Since the background area is larger than target area, we need to find area outside the model boundary as background.In corresponding to target seed area extraction, an expanding algorithm is applied to the binary image to get a bigger background area.Similar to target seed area detection, two different level expanding operations and a consequent exclusiveor operation are applied to the binary image.In Figure 6, left and middle images are the results of expanding operation and right is the final background seeds area.
Figure 6.Seed Background Extraction using Expanding Operation.
According to the abover section, the seed of target area and background area are extracted.Combining the seed area to texture image, it is clearyly shown that target area and background area correctly surround the target sculpture (in Figure 7).The blue line means area that contain background, while read line cover front part, means target.The blue line and red line both are seed area.
Figure 7. Target seed and background seed surround the target sculpture.

Target Optimization using One-Cut:
Graph based algorithm is a powerful tool for image segmentation, especially for find object from background.Graph cut is one the classic graph based image segmentation algorithms (Boykov and Jolly, 2001).
Graph based image segmentation method normally change image segmentation into a min-cut problem in graph theory.An image can be seemed as a weight graph G=<P, B>, P is the set of nodes in graph, while B is set of edges.The rule of segmentation is keeping minimum similarity between sub-graphs, but maximum similarity inside each sub-graphs.The essence of graph cut algorithm is breaking unstable or weak edges inside graph.
Figure 8. Detected Target Area using One-Cut Algorithm.

Texture to Model Matching
Recovering the spatial position relationship of texture image and geometric model is the precondition of texture mapping.We get the relationship by matching texture image and geometric model.
In details, we try to translate and rotate the geometric model in 3d space (see in Figure 9), and then perspective projection is used to get the silhouette of model in screen.As introduced in Section 2, we have already gotten the silhouette of object in texture image, so we can match texture and model by matching their 2D silhouette information.

Similarity Measurement
We present a similarity measure that can be used in texture and model matching: closest point distance.Analogous to iterative closest point, one point set (boundary points), the reference, is kept fixed, while the other one, as the source point set.For each point in the source point set, matching the closest point to the reference point set and measuring their distances.The energy formulation is formulated as: Where (  ,   ) is the point in source point set, (  ,   ) is the closest point in reference point set, and   is their distance. integrates the distance error over all source points.If two silhouette images are similar, the sum of distances error is small.In order to reduce the consumption for searching closest point, kd-tree is used in our method to improve computation efficiency.Silhouette from Texture Image

Parameters Optimization
To minimize the distance error, the Nelder-Mead simplex algorithm is applied in this our method to optimize the energy function .The Nelder-Mead simplex algorithm is a classical very powerful local descent algorithm, making no use of the objective function derivatives.A "simplex" is a geometrical figure consisting, in n dimensions, of  + 1 points  1 ,  2 , … ,   ,  +1 (Nelder. And Mead, 2003) To select the appropriate transformation, the method only uses the values of the function to be optimized at the vertices of the simplex considered.After each transformation, the current worst vertex is replaced by a better one.
At the beginning of the algorithm, one moves only the point of the simplex, where the objective function is worst, and one generates an other point image of the worst point.This operation is the reflection.If the reflected point is better than all other points , the method expands the simplex in this direction, otherwise, if it is at least better than the worst, the algorithm performs again the reflection with the new worst point.The contraction step is performed when the worst point is at least as good as the reflected point, in such a way that the simplex adapts itself to the function landscape and finally surrounds the optimum.
If the worst point is better than the contracted point, the multicontraction is performed.At each step we check that the generated point is not outside the allowed reduced solution space (Pintus et al. 2011).The pseudo code of the algorithm as follows: Initialize the  + 1 points  1 ,

Results
In this experiment, user only need to set the 3d model to the target area in image, without giving precise overlapping.The algorithm automatically finds the target area and mapping texture to 3d geometric model.
To reach a proper model to image registration result, the algorithm need an iterative processing to optimize result step-bystep.Since the iterative registration algorithm is sensitive to initial position, we test is in many different distances and view angles.

Robustness Analysis
The robustness of this algorithm is another factor we evaluate in this experiment.For each test dataset, we select 64 different initial positions.Two indicators are measured to evaluate the processing speed and registration quality, which are times of iteration and registration error.
Figure 14 is the robustness analysis results of stone lion.Blue line is the iterative time (number at left side) for each initial position.Red line shows registration error after many times of iteration.Only one failed in 64 giving initial positions.Figure 15 is the test results of bronze tripod.In test dataset II, all the 64 texture mapping test successfully converge to a stable state, resulting to good registration quality.

Conclusion
High fidelity texture mapping is an important work for heritage modelling.In this, we propose a semi-automatic image to 3d model registration algorithm, which is able detect target object and mapping texture to geometric model using detected silhouette information.Experiments using data from real project proved the algorithm we proposed.
The benefit of this method includes: (1) image to model registration result is stable and insensitive to initial position; (2) only need very simple user interaction; (3) high efficiency.
This method works well on single image to model registration.If the texture images have abundant texture information enough for aerial-triangulation, mapping image to model according exterior parameters of image will be a proper solution.Currently, we didn't add multi-image global optimization into the algorithm, which might be done in the future.

Figure 1 .
Figure 1.Test Dataset I: a stone lion in natural environment.(a) Image of stone sculpture collected by Canon camera; (b) geometric model.

Figure 2 .
Figure 2. Camera system of OpenGL.In Figure2:W, H is the width and height of 3d depth image.D is the distance of view point to culling plane.

Figure 3 .
Figure 3. Overlap effect of 3d model with image.

Figure 4 .
Figure 4. Target block area in depth image from 3d model.
(a) Translation and Zoom In/Out (b) Rotation at Three Axis Figure 9.The Translation and Rotation of 3D model Silhouette:(a) Silhouette from 3D Model (b) Data set I: A stone lion sculpture, 3d model data contains 100k triangles, image collected using Canon EOS camera with resolution of 5616*3744 (texture image and geometric model are shown in Figure1).Data set II: Bronze tripod from Museum of Wuhan University ( in Figure11), 3d model data contains 1 million triangles, image collect using same camera as data set I.
Figure 12 (a) shows the initial position of 3d model and image; (b) is the result after geometric model adjustment; (c) is the result of textured model.It is clearly in this figure that the grass area is removed from source image.Only the target area is left using as texture.Results of Target Extraction and Texture Mapping.
(a) Detailed Texture Mapping Results of Stone Lion.(b) Detailed Texture Mapping Results of Bronze Tripod.

Figure 15 :
Figure 15: Details ofTexture Mapping Result.From these two groups of experiments, we can find that: though the initial positions are different, the final registration errors are almost same after the iteration processing.We also extend this algorithm and integrate this algorithm into a software named ModelPaitner.Matching image to 3d geometric model is the first step for high fidelity texture modeling.What beyond matching are color adjustment, best image selection, texture seam processing, etc.Each of these work is very time consuming.ModelPainter is a software we developed specially for texture processing, especially for culture heritage modeling.

Figure 16 :
ModelPainter UI and the Produced 3D Model Data.