function-based

: The paper presents a novel method to measure 3-D deformation of a large metallic frame structure of a crane under loading from one to several images, when the cameras need to be attached to the self deforming body, the structure sways during loading, and the imaging geometry is not optimal due to physical limitations. The solution is based on modeling the deformation with adequate shape functions and taking into account that the cameras move depending on the frame deformation. It is shown that the deformation can be estimated even from a single image of targeted points if the 3-D coordinates of the points are known or have been measured before loading using multiple cameras or some other measuring technique. The precision of the method is evaluated to be 1 mm at best, corresponding to 1:11400 of the average distance to the target.


INTRODUCTION
Image-based deformation measurement is typically based on a stationary camera set-up.The object points are measured before and after deformation, and the difference tells the object deformation.The cameras may also move but then there is something in the object space which remains the same so that the orientation of the cameras can be solved before and after deformation with respect to a common coordinate system.
However in this paper, we are facing a measurement task, where these assumptions are no longer valid.The task deals with measuring the deformation of a large metallic frame structure of a crane when it is loaded with a weight of 40000 kg.The size of the frame is 25 × 8 m and it is located at a height of 25 m. Figure 1 shows a view of one camera illustrating the frame structure and a carriage across the frame in the middle of the longitudinal direction.Due to physical limitations, the cameras need to be attached to the deforming frame so that they move when the object deforms neither there is anything stable in the scene which could be used to link the camera orientations to the same system before and after deformation.The structure also sways during loading so that the image capturing needs to be synchronized.The cameras are considered to move from one orientation to another and the measuring to occur in temporary static conditions in both orientations.
The lack of a stable datum for deformation estimation has been previously addressed by considering only relative changes determined, e.g., by aligning the point sets so that the difference in the direction of one coordinate is minimized (Fraser and Gustafson, 1986).Another approach is proposed by Papo (1986), where a free net adjustment is extended with additional parameters and constraints, which enable to remove the datum defect.Shahar and Even-Tzur (2014) use the same principle to extract a datum area, where changes are due to deterministic dynamics and can be estimated.Deformation of the rest of the network is obtained after cleaning the deterministic component from the data.We propose to address the crane deformation problem by modeling the deformation with appropriate shape functions, the parameters of which are solved in a least squares adjustment minimizing the difference between observed and modeled image coordinates of circular target points attached onto the object.The shape functions constrain the problem so that it becomes solvable.The key issue is to find shape functions which adequately describe the deformation and then, to determine the dependence of modeled image observations on the shape function parameters.
Shape functions have been previously used to constrain possible deformations in order to cope with weak imaging geometry or even estimation from a single image (Jokinen and Haggrén, 2011).They assume that the orientations of the cameras remain fixed while in Jokinen and Haggrén (2012), the method is extended to changing orientations.The problem can also be viewed as non-rigid registration of 3-D data sets, which thus assumes that 3-D point sets, surfaces, or volumes are first reconstructed before and after deformation, and then a non-rigid transformation is esti-mated to account for rigid changes in orientations and a non-rigid change or deformation of the object.To regularize the problem, constraints are typically included for the shape of the deformation (Hirshberg et al., 2012).For example, Chui and Rangarajan (2003) assume that the deformation can be presented as a thinplate spline, while Rueckert et al. (1999) apply B-splines, which can better describe local deformations.Small local deformations may be also recovered using elastic models (Bajcsy and Kovacic, 1989).If certain structures of the object should not change the shape, a penalty term may be included which forces the transformation to be rigid in that region (Loeckx et al., 2004).In many cases, the shape is represented as a linear combination of shape bases and the measurements include only image observations.The non-rigid shape and motion are then solved using rank constraints on the measurement matrix and orthonormality constraints on camera rotations (Bregler et al., 2000;Brand, 2001;Torresani et al., 2001), while Xiao et al. (2006) show that a unique solution is obtained when basis constraints are also included.Fayad et al. (2009) extend the method to a quadratic deformation model, which is able to cope with object bending, stretching, shearing, and twisting.Most of these approaches that utilize shape bases assume an orthographic camera projection model, while Hartley and Vidal (2008) propose an algorithm for a perspective camera.
For the crane deformation, however, we can do more than just non-rigid registration as changes in the camera orientations are not arbitrary but directly dependent on the frame deformation.Further, we need not reconstruct the 3-D points after deformation explicitly, but only the shape function parameters are solved, which can be used to evaluate the deformation at any point.As measurements, we also use only the image coordinates of the target points in a single image or optionally in multiple images after deformation.Ranta (2015) studies the same problem but registers the 3-D object points before and after deformation by assuming that one beam of the structure remains unchanged which leads to less accurate deformation estimation.Our main contribution is thus to couple changes in the cameras orientations with the object deformation for estimation of deformation with high precision.
The paper is organized as follows.The methods are described in Section 2 including description of the object geometry and camera set-up, introducing a mathematical model for the deformation utilizing shape functions, deriving the influence of deformation on the camera orientations, and solving the deformation by least squares adjustment.Experimental results are discussed in Section 3 and conclusions are presented in Section 4.

Object Geometry and Camera Set-up
The object geometry is illustrated in Fig. 2. It consists of a metallic frame supported by four columns at the corners.The frame is not anchored but stands on wheels.In the middle of the longitudinal direction, there is a carriage k across the frame, which is loaded with a heavy weight.Due to loading, the carriage bends downwards.This causes beams a and b to bend downwards and slightly towards the carriage since the beams are assumed nonstretchable.Beams a and b drag also beams c and d towards the carriage, while the columns bend slightly.Four calibrated industrial Mightex SME-C050-U cameras of resolution of 2560×1920 pixels are attached to the beams: one at beam a, one at beam b, and two at beam d.The camera viewing directions are towards beam c.The imaging geometry is not optimal for high measuring accuracy but dictated by physical limitations.The interior orientations of the cameras are assumed to remain stable during loading as the camera optics are locked.The image capturing is synchronized with a hardware trigger in order to cope with considerable swaying of the structure that occurs during loading.To measure the deformation, circular target points are evenly attached to the carriage and beam c (see Fig. 1).In addition, some bolts visible on beams a and b are used as measurement targets.For solving the scale of photogrammetric measurements, a scale bar with a known length is attached to the carriage.The scale bar moves but does not deform when the structure is loaded.The image coordinates of the targets are measured manually before and after loading using a centroiding tool of iWitness software (www.iwitnessphoto.com).The software solves also the image orientations and the 3-D object coordinates needed before loading.An object coordinate system shown in Fig. 3 is defined as follows.Planes are fitted to the points on the carriage and on beam c.The average of the normals of these planes defines the direction of the Y -axis.The direction of the X-axis is given by orthogonal projection of the normal of a plane fitted to the points on beam a onto a plane perpendicular to the Y -axis.The direction of the Z-axis is obtained by the cross product of directions of Xand Y -axes.The origin is located at the intersection of three planes defined as follows.The first plane is defined by the X-axis direction and the mean of points on beam a, the second plane is defined by the Y -axis direction and the location of the one of the cameras on beam d the projection center of which has a smaller Y -coordinate value, and the third plane is defined by the Z-axis direction and the lowest point on beam a.The 3-D points reconstructed originally in a camera coordinate system of one of the cameras are transformed to the object coordinate system and also, the exterior orientations are computed with respect to the object coordinate system.The lines in Fig. 3 help to illustrate the inner outline of the frame.The line between points A and B is the intersection of the XY-plane with a plane perpendicular to the Yaxis and passing through the center of points on the carriage.The same holds for the line below points on beam c.Point C is the Y-axis intercept of a plane perpendicular the Y-axis and passing through the center of points on beam c.

Model for Object Deformation
The object is assumed to deform according to the following mathematical model.The carriage bends downwards according to a shape function where α and β are unknown parameters and f (YA, α) is the deformation of beam a at point A.
Beam a deforms downwards according to another shape function (2) This model implicitly assumes that the maximum deformation in Z appears at YD = YC /2.More complex models such as piecewise defined polynomials of second degree (a different polynomial for the part of beam a which is in front of the carriage and which is behind of the carriage) should thus be used if the carriage is not placed in the middle of the longitudinal direction of the frame.Since beam a is assumed non-stretchable, the length |g(Y, ∆Y, α)| of the deformation curve ∆Z = f (Y, α) from the mid point to the deformed point equals the distance from the mid point to the measurement point before deformation.Thus, deformation ∆Y (α) is solved from the equation (3) Denoting the partial derivative of f by V = α(2Y − YC ) and denoting an initial estimate for ∆Y is obtained by finding the value h = L, where the sequence changes the sign.The initial estimate is then For the refinement of the initial estimate, Eq. 4 is linearized and at each iteration, the correction ∆VT is obtained from Further, the correction ∆(∆Y ) = ∆VT /(2α).Beam b is assumed to deform similarly as beam a.
Deformation of beam c is modeled as a rotation around a line parallel to the X-axis and having an equation Y = YC , Z = ZE, where ZE(α) and the rotation angle θ(α) depend on the deformation of beam a.The rotation angle is obtained from the slope of normal of curve in Eq. 2 at Y = YC + ∆YC .It is given by θ(α) = arccot(−1/(α(YC + 2∆YC ))). (8) The coordinate ZE is the intercept of the normal with line X = 0, Y = YC given after some simplification by Beam d is assumed to deform similarly as beam c except that the rotation is around line Y = 0, Z = ZE by angle −θ.

Changes in Camera Orientations
Due to object deformation, the projection centers of cameras on beams a and b change by ∆YCam and ∆ZCam similarly as the measurement points move according to Eqs. 2 -7.The camera rotations change by a rotation RX (ϕ) around a line parallel to X-axis and passing through the changed camera location.The rotation angle is obtained from the slope of tangent of curve ∆Z = f (Y, α) at the changed camera location and it is given by Small corrections by rotations RX (ωa) and RZ (κa) around lines parallel to the X-axis and twice rotated Z-axis, respectively, and passing through the changed camera location are included to compensate for swaying of the object during loading.The changed rotation matrix of the camera on beam a is obtained from where R a old is the rotation matrix before loading.A similar formula holds for the camera on beam b.
The cameras on beam d rotate by −θ(α) around a line parallel to the X-axis and passing through (0, 0, ZE).Small corrections of rotations around lines parallel to the X-axis and twice rotated Z-axis and passing through (0, 0, ZE) are also included so that The changed camera location is given by where t d old is the translation vector before loading and tE = [0, 0, ZE] T .

Least Squares Adjustment
According to the collinearity equations, the measured image coordinates x ik , y ik of target point i in camera k after deformation are related to the object coordinates X ik , Y ik , Z ik in the camera coordinate system of camera k after deformation by where c k is the camera constant of camera k, the image coordinate origin is in the principal point, and v xik , v yik are residuals in the x and y directions, respectively.Further, we have where the deformed object coordinates and the changed rotation matrices R k new and translation vectors t k new depend on the shape function parameters α, β and on the rotation angles ω k , κ k (two parameters for each camera k).For K cameras, there are thus 2 + 2K parameters, which are solved by minimizing the merit function (17) where w ik are weights.The weight w ik equals one if the differences between the measured and modeled x ik and y ik coordinates are both below an adaptive threshold.The threshold is based on the mean and standard deviation of differences in the image coordinates of points having positive weight at the previous iteration, and it gets tighter as the iteration proceeds similarly as proposed for registration of curves and surfaces by Zhang (1994).Otherwise, the weight equals zero, and also when the point is not visible in camera k.
The minimization problem is solved using the Levenberg-Marquardt algorithm.Consequently, the derivatives with respect to the unknown parameters are needed.These can be derived using elementary calculus after figuring out how the modeled image coordinates depend on the parameters.For points on the carriage, (18) For points on beams a, b, and c, The derivative ∂(∆Yi)/∂α is obtained for points on beams a and b by differentiating Eq. 4 with respect to α.It yields from which we can solve where VT = α(2(Yi + ∆Yi(α)) − YC ).

RESULTS
The results of deformation estimation with all the four cameras are illustrated in Fig. 4. It shows that the points on the carriage and beams a and b move mainly downwards while the points on beam c move mainly towards the carriage.The cameras on beam d move also mainly towards the carriage and the cameras on beams a and b move mainly downwards.In order to verify that the solution is correct, the 3-D points after deformation are projected onto the images using the changed orientations.Figure 5 shows the projected points as red plus signs, which correctly hit the image targets visible by this camera.
The method was then tested with different number of cameras  ranging from one to four cameras and with all the possible 15 combinations of them.The method worked in all cases except in two, where the deformation in the middle of beam a and in the middle of beam b was larger than in the middle of the carriage.
The parameter β was negative in these cases, which is unnatural.The mean and standard deviation of quantities that characterize the deformation are given in Table 1 calculated using all the combinations of the cameras.The maximum deformations are reported in the middle of the carriage and in the middle of beam a or beam b.The RMSE gives the difference between the measured image coordinates and the 3-D points projected to the image after deformation using all the points, while the square root of the merit function χ 2 is based only on points with a positive weight.The standard deviations describe the variation due to different camera configurations.For the carriage with well-defined target points, a standard deviation of 1 mm is achieved while for beams a and b, the standard deviation is larger as the bolt targets are more uncertain to measure and they are also located farther away from the cameras on the average.These standard deviation values are regarded as experimentally estimated precisions of the quantities mentioned, although the number of camera combinations is quite small.No ground truth for the deformations was available so that the accuracy of the method could not be verified.The swaying of the structure also limits alternatives that might possibly be used for acquiring such reference data.The dependence of the method on the number of cameras is further studied in Figs. 6 -8, which show the deformation in the middle of the carriage where it is at its maximum, the deformation in the middle of beam a or beam b, and the rotation angle of beam c, respectively.The red crosses are the deformation or rotation angle values averaged over all combinations of cameras possible for the given number of cameras.For 1, 2, and 3 cameras, there are thus 4, 6, and 4 combinations, respectively.The blue lines show the uncertainty given by the standard deviation of the deformation values.For four cameras, there is only one combination so that the standard deviation is not defined.It can be seen that the estimation of deformation is possible using only a single camera although the lowest uncertainty is obtained for three cameras.The precision (square root of variance) of the quantities listed in Table 1 was also estimated analytically through error propagation from the covariance matrix of the parameters given by the inverse of the coefficient matrix of the normal equations multiplied by the reference variance estimated during least squares adjustment.The precision values on the average for different numbers of cameras are shown in Figs. 9 -10.The precision clearly increases with increasing number of cameras as expected.The values in Figs. 9 -10 are considerably lower than the standard deviations in Table 1 calculated experimentally with different com-binations of cameras.Possible reasons for this include that the method is sensitive to image viewpoint or there are systematic errors caused by inaccuracies in interior or exterior orientation, or that the mathematical model for deformation does not perfectly describe the true deformation.The considerable swaying of the structure during loading is considered to be the main reason as it causes deformations which have not been modeled although the orientations of the cameras are corrected by small rotations.
According to the specifications of the cameras used, the camera trigger delay is less than 0.2 ms, which describes the uncertainty in synchronization of image capturing.However, the obtained standard deviation of 1 mm for the maximum deformation of the carriage is better than the measuring tolerance of 4 mm set by the manufacturer of the crane.When three cameras are used, the standard deviation of deformation in the middle of beam a or beam b is 0.7 mm, which is also below the tolerance.For comparison, the iWitness software reports overall accuracies of 7 mm and 6 mm for the 3-D object points before and after deformation, respectively, so that estimating the deformation as a difference between the object coordinates does not give satisfactory results.
According to iWitness, the imaging geometry is moderate and the smallest angles of intersection of 3-D points are 10.7 • .The estimated accuracy of image referencing before deformation is 0.16 pixels, which also affects the precision of the scale determined by measuring the image coordinates of circular targets at both ends of the scale bar.The length of the scale bar is known with an accuracy of 0.001 mm.However, we note that for the proposed method, all random errors in image measurements, orientations, scale, and 3-D points are included and fully accounted for in the covariance matrix of the shape function parameters, which is used to estimate the analytically derived precision values.

CONCLUSIONS
The paper has presented a new method to estimate the deformation of a large frame structure in challenging conditions, where the imaging geometry is not optimal, the structure sways during loading, and the cameras move along with the deforming body without anything stable in the scene.The proposed solution was based on modeling the object deformation with two quadratic shape functions and assuming that the beams of the structure are non-stretchable.A careful analysis was then performed to derive how the measurement targets move and the exterior orientations of the cameras change as a function of the shape function parameters estimated in a least squares adjustment.The experimental results show that the deformation of the carriage can be measured with a precision of 1 mm on the average using any combination of one to four cameras, while a similar precision is obtained for beams a and b, when three cameras are used.These values are better than the measuring tolerance requested by the manufacturer of the crane, which verifies that the mathematical model proposed for the deformation is adequate.Future research includes continuous monitoring of the frame deformation with one or more video cameras during crane operation.For such an application, it is advantageous that the method works also with a single camera so that the need to synchronize the image capturing is eliminated.However, the precision is not so high as with multiple cameras.Continuous monitoring would also allow to learn how the frame deforms in different situations, which could be used to improve the mathematical model for deformation and to lessen the influence of measuring uncertainties on the precision of the results.Different configurations of targets should be also tested since the distribution, number, and distances of target points might play a crucial role in determining a correct deformation shape.

Figure 1 :
Figure 1: Frame and carriage of a crane to be measured.

Figure 3 :
Figure 3: Measured target points and camera locations in the object coordinate system before loading.

Figure 4 :
Figure 4: Direction and 100 times the magnitude of movement of target points and camera locations due to object deformation.

Figure 5 :
Figure 5: 3-D points (red plus signs) projected onto the image after deformation.The image has been cropped (zoom in to see the red plus signs).

Figure 6 :
Figure 6: Deformation in the middle of the carriage ± sample standard deviation as a function of the number of cameras.

Figure 7 :
Figure 7: Deformation in the middle of beam a or beam b ± sample standard deviation as a function of the number of cameras.

Figure 8 :
Figure 8: Rotation angle of beam c ± sample standard deviation as a function of the number of cameras.

Figure 9 :
Figure 9: Analytically derived precision of deformation in the middle of the carriage and in the middle of beam a or beam b as a function of the number of cameras.

Figure 10 :
Figure 10: Analytically derived precision of the rotation angle of beam c as a function of the number of cameras.

Table 1 :
Mean and standard deviation of deformation results of all 15 combinations of one to four cameras.