CALIBRATION STUDY OF A TRIMBLE ACx 4 SYSTEM FOR DIRECT GEOREFERENCING MAPPING APPLICATIONS

The Trimble Aerial Camera x4 (i.e., TACx4) is a photogrammetric multi-head system manufactured by Trimble Inc.© in 2010. It has four cameras mounted together in the main structure allowing the simultaneous acquisition to generate a single synthetic image with much larger ground coverage. In addition, the cameras are also integrated with a GNSS/INS to perform “Direct” or “Integrated” Sensor Orientation. The main condition to obtain photogrammetric mapping products with high accuracy using a direct sensor orientation procedure is to execute a step known as “geometric system calibration”. In general, the photogrammetric multi-head system manufacturers perform this step using laboratory methods to obtain the parameters of cameras interior and relative orientation. Accurate mounting parameters (lever arms and “boresight misalignments”) are fundamental requirements to generate the synthetic image when georeferencing of images is applied. This paper shows a “full field” calibration method to perform the geometric system calibration of the TACx4 system and its evaluation for direct sensor orientation mapping applications. The developed method involves two steps using only aerial images: (1) estimation of the cameras interior and relative orientation parameters to generate the synthetic image and (2) estimation of the synthetic image interior orientation and the mounting parameters between the synthetic image and GNSS/INS reference systems using two different methods. The obtained results in the conventional photogrammetric project show that the proposed method allows performing the geometric system calibration of the TACx4 system achieving around 50cm (5 pixels) in horizontal and vertical accuracies. The obtained results can be used for large-scale mapping requirements using direct sensor orientation according to Brazilian accuracy standards.


INTRODUCTION
In conventional aerial photogrammetric mapping projects, the exterior orientation parameters of a block of images have been indirectly determined by the bundle block adjustment using collinearity equations and ground control points.Today, due to the GNSS/INS integration technologies, the Direct Sensor Orientation has been frequently used to perform the direct georeferencing of images, reducing the cost and time involved to establish a set of control points at the ground.However, to use this technology, a system calibration procedure is a fundamental prerequisite to integrate GNSS/INS sensors with a digital camera.Consequently, the system accuracy for the georeferencing procedures is largely discussed in the worldwide photogrammetry community.
Several studies at the geometric accuracy of multi-head camera systems have been carried out with focus at the synthetic (or virtual) image generation process.The first commercial system, known as Digital Mapping Camera (DMC), was developed by Z/I Imaging.The principle of virtual image generation was shown in 1999.The camera has a charge-coupled device (CCD) array sensor with high interior geometric stability.Four panchromatic and multi-spectral modules are used.During the last two decades, several authors have investigated the DMC geometric calibration process using laboratory and in-flight methods.They have also investigated the quality of the virtual image generation process for photogrammetric applications as well its influence in the mapping accuracy.In 2001, the properties of the DMC were described that uses a modular design to achieve high geometrical resolution together with multispectral capabilities.The aspects involving the calibration of panchromatic camera head system using a standard goniometer technique are also discussed (Hinz et al., 2001).In 2002 and 2003, several authors shown the post-processing steps of DMC image data to generate virtual central perspective images and mosaicking procedure to transform four individual high-resolution panchromatic images to one virtual image into a normal perspective projection (Heier, 2002, Zeitler et al., 2002and Dorstel et al., 2003).In 2006 and 2008, the authors discussed the DMC accuracy focusing at the role of selfcalibration parameters and the assessment of automatic DEM (Digital Elevation Model) quality using an appropriate set of self-calibration parameters to achieve theoretical accuracy and precision (Alamús et al., 2006 andAlamús andKornus, 2008).Other authors investigated the geometric accuracy of DMC and the calibration process to generate a distortion-free virtual image (Madani and Shkolnikov, 2008).
The UltraCamD (UCD), developed by Vexcel Imaging, was shown in 2003.It has a different image acquisition principle and virtual image generation process.The solution consists a sensor unit using multiple area array sensors and four lenses for the high-resolution panchromatic image.A large image is built by combining the four separate component images, using one master cone to define the image coordinate system.Additional optical cones produce color image separated in red, green, blue and near-infrared.In this year, the authors shown the potential of the UCD camera system for mapping applications and its geometric calibration process using laboratory methods for the virtual image generation (Leberl et al., 2003).Also, some authors investigated the geometric issues of the digital large format aerial camera UltraCamD.They show the concept of the geometric calibration by means of a bundle adjustment based at the specific design of the camera additional parameters to reduce the remaining systematic errors caused by environmental conditions (Smith et al., 2005 andGruber andLadstädter, 2006).
Another similar research involving commercial and noncommercial camera systems and calibration procedures were developed.They introduced the SWDC-4 (Siwei Digital Camera-4) large format digital aerial camera system detailing the calibration of non-metric CCD cameras, correction, and rectification of unit images, internal bundle adjustment and generation of a virtual image (Li et al., 2008).In 2009 and 2012 some authors investigated an approach based at simultaneous calibration of two or more cameras using constraints and experiments with images acquired by an arrangement of two Hasselblad H3D cameras (Tommaselli et al., 2009 andTommaselli et al., 2012).
The use of GNSS/INS integrated with a single or multi-head camera system allows for the direct determination of the position and orientation of the platform at the image acquisition time, process known as direct sensor orientation.In 1996 and 1999, a study about the method of INS/DGPS integration was presented to provide exterior orientation parameters for direct georeferencing of airborne imagery with more reliability and better accuracy.The operational aspects of airborne mapping with INS/DGPS was analysed and strategies for minimize the effect of the hardware integration errors at the process for direct georeferencing (Skaloud et al., 1996 andSkaloud, 1999).In 2000 and 2001, several authors discussed the topic of image orientation by combined aerial triangulation with GPS/IMU, process also called as Integrated Sensor Orientation.The analysis of the direct determination of the exterior orientation parameters via GPS and IMU as a complete substitute for aerial triangulation was investigated as the use of integrated systems in airborne environments with the main emphasis at the combination with standard analog frame cameras.Additionally, a combined GPS/Inertial-AT (Aerial Triangulation) or integrated sensor orientation approach was presented which allows the In-Situ calibration of certain system parameters even without ground control and therefore provides the highest flexibility to overcome the most limiting factor of direct georeferencing (Cramer et al., 2000, Heipke et al., 2000, Cramer, 2001b, Cramer and Stallmann, 2001and Heipke et al., 2001).In 2002 and 2004, the authors investigated the direct geo-referencing of sensors based at a combination of relative kinematic GPS-positioning and inertial measurement units (IMU) with high accuracy level and discussed the aspects of Camera/IMU boresight calibration as a critical element in the mapping process when using GPS/IMU in the direct georeferencing mode of mapping (Mostafa, 2002 andJacobsen, 2004).
The use of DG (Direct Georreferencing) in photogrammetric mapping applications has been viable for medium and large scale, where accuracies in the range of 5-20cm for the horizontal and 10-25cm for the vertical component (Cramer et al., 2000).However, to get these accuracies, a geometric calibration is necessary.This procedure involves the determination of the mounting parameters relating the IMU body frame and the camera reference system as well the camera interior orientation parameters.In 2002, some authors shown a new single-step calibration procedure.In this study, the calibration parameters are explicitly inserted in the collinearity equations and the IMU/GPS data were considered as pseudoobserved quantities, replacing the exterior orientation parameters as unknowns in the block adjustment (Pinto and Forlani, 2002).In the same year, another author investigated the calibration aspects in the direct georeferencing of frame imagery.The study focuses at the estimation of the exterior orientation by a combination of an inertial measurement system (IMU) with relative kinematic GPS-positioning and boresight misalignment (Jacobsen, 2002).In 2003 and 2004, several authors show the results of eleven system calibrations of four GPS/IMU/optics combinations performed.In addition to the boresight parameters, the interior orientation parameters appeared to be significant calibration quantities and the variation of the interior orientation parameters were quite large in comparison with boresight misalignment variations.Also, they investigated the effects of the system calibration at direct sensor orientation based at data set calibration flights; using bundle block adjustments, the correction for interior orientation, 3 shifts and 3 misalignment angles between IMU and imaging sensor have been determined (Honkavaara, 2003 andYastikli, 2004).
In this paper, the calibration of the Trimble ACx4 system was performed using a "full field" method.The determination of the cameras interior and relative orientation parameters to generate the synthetic image was done using aerial images.The determination of mounting parameters between the synthetic image and GNSS/INS reference systems was done using twostep and single-step methods.In the two-step method, the mounting parameters are determined by comparing the GNSS/INS position and orientation information at the image acquisition time and the corresponding camera EOP (Exterior Orientation Parameters) obtained from the traditional bundle block adjustment using ground control points.In the single-step method, the mounting parameters and camera IOP (Interior Orientation Parameters) determination are performed simultaneously by the least square adjustment (bundle block adjustment with additional conditions) using appropriate flight and control configuration.Every calibration step will be detailed in section II.Section III outlines the equipment and dataset used in this research work while section IV presents the experimental results and discussions.Finally, section V shows the conclusions and recommendations for the future work.

THE CALIBRATION METHOD
The calibration method, proposed in this study, estimates all parameters involved in a geometric calibration process of the multi-head camera systems using the aerial image.This approach can acquire more realistic results from the calibration process due to it uses images close to operational condition and offer more flexibility to calibrate the system considering its use in the airborne environment.The method is composed of two steps, as follows: 1. Estimation of cameras interior and relative orientation parameters to generate the synthetic image; and 2. Estimation of synthetic image interior orientation and mounting parameters.

Estimation of cameras interior and relative orientation parameters and synthetic image generation
The estimation of interior orientation parameters of multi-head camera system is an essential step that enables the mathematical representation of the physical camera internal geometry known as the focal length (f), principal point coordinates (xp, yp), radial symmetric and decentering distortion coefficients (k1, k2, k3, p1, p2).Generally, this estimation is performed by laboratory methods using a standard goniometer technique (Hinz, 2001).However, the methods that use aerial images can offer a more robust solution for aerial photogrammetric cameras since the influence of external conditions (such as changes in pressure, temperature, humidity, among other factors) can model the parameters close to the operating conditions.In this work, the interior and relative orientation parameters of each camera the composed the imaging system was incorporated in a bundle block adjustment using the collinearity equations with additional parameters (Brown, 1971). Where: • = calibrated focal length; •   ,   = coordinates of point "A" in the image space reference system; •   ,   ,   = coordinates of point "A" in the object space reference system; •  0 ,  0 ,  0 = perspective center coordinates in the object space reference system; During the estimation of the interior and relative parameter, the focal length of each camera was fixed for the nominal value of 74mm to normalize scale variation in each image.The parameters such as the principal point coordinates, radial symmetric and decentering distortion coefficients were computed for each camera.The relative orientation parameters were determined in the same bundle block adjustment and refined using tie points extracted in the overlapping areas.To generate the synthetic image, the relative orientation parameters were converted from the mapping to the synthetic image reference system using the Z component of the camera 1 as a reference and fixed equal to 74 mm.
The synthetic image generation is an essential step used by commercial manufacturers of multi-head camera systems to generate a "virtual image" resulted from the images of each camera.This paper uses a similar approach considering as "input" the interior and relative orientation parameters determined in the previous step.Before starts the synthetic image generation, to eliminated small differences, tie points in the overlapping areas area extracted (Figure 1).This step was performed using Harris-Stephens algorithm based at a corner detection principle and RANSAC (RANdon Sample And Consensus) algorithm to detect and eliminate inaccurate correspondences.
Figure 1.Imaging system scheme with overlap areas.(Ercolin Filho, et al., 2017) Using the coordinates of the approximately 5000 matched tie points in the overlap area of each image pair (img1 and img2), the relative orientation parameters of each camera were refined using the average differences in the x and y directions.The last step consists to use the refined relative orientation parameters and interior orientation parameters obtained in the previous step to generate the synthetic image.During this process and using the same equations as presented in (1) to ( 11), all image coordinates of the synthetic image were converted from the image coordinate system (mm) to the bitmap coordinate system (row and column) and projected onto the images of each camera to get RGB values using bilinear interpolation algorithm.The described steps were performed by SIGP (Synthetic Image Generation Program) software developed in the MathWorks Matlab® platform and using Harris-Stephens and RANSAC Matlab libraries.

Estimation of the mounting parameters between the synthetic image and GNSS/INS reference systems
The estimation of the mounting parameters includes the lever arms (   ), which is the vector connecting the IMU body frame to the camera perspective center and the boresight misalignment angles (   ) relating the camera coordinate system and the IMU body frame.For direct georreferencing applications, the refinement of the synthetic image interior orientation parameters is essential and can be performed using Two-Step or Single-Step methods (Kersting, 2011).
In the "Two-Step" method, the mounting parameters are computed using the position and orientation of the IMU body frame from the GNSS/INS trajectory and the exterior orientation parameters of the camera perspective center obtained through traditional bundle block adjustment using equations ( 12) and ( 13).The interior orientation parameters are obtained in the bundle block adjustment.
In the "Single-Step" method, the mounting parameters and the interior orientation parameters (   ,    and    respectively) are considered as unknowns in a bundle block adjustment that incorporates the position and orientation information derived from the GNSS/INS trajectory using the mathematical model presented in equation ( 15).This model is also known as "modified collinearity equations" and involves the sum of three vectors after applying the rotation matrix and a scaling factor (  ).

𝑟
The Single-Step method is considered more robust than the Two-Step method since the interior orientation and mounting parameters are determined simultaneously in a unique adjustment process.However, this procedure requires a photogrammetric block of images with strips in opposite directions and at two different flight heights to obtain the conditioning of system equations and to minimize the correlation between parameters (Cramer, 2001b).
In this research, two methodologies are used to perform the geometric calibration.In the first method, the interior orientation parameters are computed with a conventional selfcalibration bundle block adjustment and the mounting parameters (only misalignment angle) with the Two-Step method.The software used were developed at the Photogrammetry Department of University of Paraná State.In the second method, the interior orientation and mounting parameters (lever arms and boresight misalignment angles) are computed simultaneously with the Single-Step method using the software MSAT (Multi Sensor Aerial Triangulation) developed by Digital Photogrammetry Research Group at the department of Geomatics, University of Calgary.

The TACx4 system
The Trimble Aerial Camera x4 developed by Trimble Germany TM , which is classified as large-format frame photogrammetric camera, is composed by four photogrammetric digital cameras model PhaseOne TM P65+ with 60.5 megapixels (image with 8984 x 6732 pixels); 6μm geometric resolution; visible range (RGB) spectral resolution; 16-bit radiometric resolution per band and Schneider© Apo-Digitar lenses with 72mm nominal focal length (Figure 2).The set of cameras is mounted at the main structure for simultaneous image acquisition with overlap areas that are used to extract tie points and generate the synthetic image.Integrated with the main structure and the cameras, a GNSS/INS position and orientation system model POS-AV510 TM is used (Figure 2).This system is manufactured by Applanix© Corporation and it has an IMU model LN-200 from Northrop Grumman Corporation used to perform direct sensor orientation of aerial sensors.This system operates integrated with a flight management system (FMS) which allows to record the position and orientation of the platform at the image acquisition time and to obtain the platform trajectory during the aerial survey with accuracy around ±0.05-0.30m in the threedimensional position coordinates and orientation accuracy for roll and pitch angles around ±0.005° and ±0.008° for heading angle in the post-processing mode.To transform the trajectory from the GNSS antenna in the aircraft to the IMU body frame, the IMU-GNSS offset was determined by conventional topographic surveying with three-dimensional accuracy better than 1cm.

Data used for calibration
Two datasets were used to perform the geometric calibration experiments of the TACx4 system.The first dataset comprises four image blocks used to compute the interior and exterior orientation parameters of each camera to be used in the synthetic image generation step.The second dataset comprises an image block of synthetic images used to determine the interior orientation and the mounting parameters relating the synthetic image and the IMU body frame reference systems.A specific calibration field was implemented to acquire the images and both datasets were captured at the same flight.The image acquisition configuration in both datasets has two flight heights to minimize the correlation between the interior and exterior orientation parameters in the calibration procedure.The calibration field has 56 signalized control points and the three-dimensional coordinates (X, Y, Z) were determined using GNSS (Global Navigation Satellite System) surveying technology using relative static positioning techniques.The obtained horizontal and vertical accuracies were approximately 1 and 2cm, respectively.The targets were implemented at the ground using circular design with 60 cm diameter (Figure 4

Estimation of interior and relative orientation parameters of the cameras
The Table I presents the results obtained in the first step of the geometric calibration procedure.The results include the estimated interior and relative orientation parameters of the cameras with the respective precision.During the estimation of the parameters, the focal length of the four cameras was fixed using the nominal value of 74mm to normalize the image scale variations.The combination of the principal point coordinates (xp, yp), the first radial distortion coefficient (k1), and decentering distortion coefficients (p1, p2) was significant in the variance and covariance matrix.The relative orientation parameters of the four cameras (XC, YC, ZC, C, C, C) presented in Table 1 were computed using camera 1 (CAM1) as a reference and refined using the average differences in the x and y directions.The radial symmetric distortion of four cameras presented similar behavior and the maximum distortion at the border of each image is approximately 15 pixels.The correlations between IOP and EOP computed from the variancecovariance matrices of the four cameras calibration were below 10%.The correlation between the focal length and the vertical coordinate of each cameras perspective center (Z0) was insignificant since the focal length was fixed during the calibration.
After the calibration step, using the estimated interior and relative orientation parameters of each camera, the synthetic image was generated considering the main parameters as shown in Table 2.In this step, the pixel's color information (RGB) of the synthetic image was obtained by the direct projection of the pixel to the image space of each camera using the collinearity equations.Additionally, the bilinear interpolation, equalization, and mosaicking algorithms were also applied to generate the final synthetic image as shown in Figure 5. Table 2.The synthetic image main parameters.
Analysing the synthetic image, the geometric quality and the residuals resulting from the process were around 3µm (½ pixel).
The decentering distortion values were not significant and the maximum radial distortion at the border of the synthetic image was close to 2 pixels.This value can be attributed to the measurement errors in image points used in the geometric quality analysis.

Estimation of the mounting parameters between the synthetic image and GNSS/INS reference systems
As described, two methods were used to compute the mounting parameters and refine the interior orientation parameters of the synthetic image.In the first method, the interior orientation parameters were computed using a conventional self-calibration bundle block adjustment and only the misalignment angle was computed using the Two-Step method.In the second method, the interior orientation including lever arms and boresight misalignment angles was computed simultaneously using the Single-Step method.The obtained results are shown in Table 3. 2.57 -07 +/-2.14 -08 2.42 -07 ± 2.05 -08 K2 (mm -4 ) -1.04 -10 +/-1.53 -11 -9.30 -11 ± 1.47 -11 K3 (mm -6 ) 2.28 -14 +/-3.34 -15 2.06 -14 ± 3.19 -15 P1 (mm -2 ) 1.73 -06 +/-6.12 -08 1.70 -06 ± 5.85 -08 P2 (mm -2 ) 2.46 -07 +/-7.72 -08 2.03 -07 ± 7. The Table 3 shows that the radial and decentering distortions parameters have similar values in both experiments and the misalignment angles were also similar.However, there were significant differences among values of the focal length and principal point coordinates computed in the two experiments.These differences are expected because the relative positions of the synthetic image were not computed in the Two-Step methodology.Consequently, the focal length and principal point coordinates absorbed the variation of these parameters due to the high correlation between the focal length and the image perspective center coordinates.In the next section, the estimated calibration parameters from the two methods are used for direct georeferencing and the object space reconstruction accuracies are evaluated.Direct georeferencing experiments using the same image block used in the calibration process as well as an independent image block are performed.

Geometric system calibration validation for large-scale mapping using direct sensor orientation
To validate the geometric system calibration results obtained with both methods, the experiment where performed using an independent image block.The independent image block has 268 images in twenty linear strips, flown with 1300 m of flight height above the ground resulting in a GSD of approximately 10 cm.The longitudinal overlap in all strips were approximately 60%, as in a traditional photogrammetric project.In this image block, 36 non-signaled control points (natural targets) were available.All control points have three-dimensional coordinates (X, Y, Z) determined using GNSS (Global Navigation Satellite System) surveying technology using relative static positioning techniques.The obtained horizontal and vertical accuracies of the control points were approximately 1 and 2 cm, respectively, as required by the international accuracy standards.Figure 6 shows the layouts of the strips and the control points.For the validation, the 3D coordinates obtained by the photogrammetric intersection were compared with the surveyed 3D coordinates of the natural targets.The RMS analysis results are presented in Table 4.  Analysing the obtained accuracies in Table 4, it can be observed that approximately 40% of the 3D discrepancies in the control points were smaller than 3xGSD in three-dimensional coordinates.In the t-student and qui-square analysis, it can be demonstrated a very similar performance for both procedures for direct sensor orientation.Even though the experiments achieved smaller horizontal and vertical accuracies, the obtained accuracies are acceptable for 1:5000 to 1:2000 mapping scale according to the Cartographic Accuracy Standard for Digital Cartographic Products (PEC-PCD) of the Technical Specification for Vector Geospatial Data Acquisition (ET-ADGV) approved by the National Cartography Commission of the Brazilian Institute of Geography and Statistics (CONCAR/IBGE).

CONCLUSIONS
This paper has presented a method for the geometric calibration of the TACx4 system.Different from the proposed methods in the previous literature, this paper has presented a "full field" method to estimate the calibration parameters under operational conditions.The presented method consists two main steps: (1) estimation of the cameras interior and relative orientation parameters to generate the synthetic image and (2) estimation of the mounting parameters between the synthetic image and GNSS/INS reference systems, and the refinement of the synthetic image interior orientation parameters using two different methods.To validate the parameters obtained in the calibration procedures, an experiment using photogrammetric intersection were carried out.The experiment was performed using synthetic images from an independent project.
The proposed methods were utilized to obtain the necessary information to generate the synthetic images.The quality of the synthetic image was assessed and residual of the approximately ½ pixel and insignificant values for radial and decentering lens distortions coefficients were encountered.The performed experiment has demonstrated that using the estimated parameters from both methods can be used to produce for largescale mapping requirements in scales ranging from 1:5000 to 1:2000 in according with Brazilian accuracy standards.One should note that the proposed methodology also allows the refinement of the interior orientation and mounting parameters of the synthetic image under operational conditions.The presented methodology is an alternative approach to existing laboratory methods and can be applied to all aerial multi-head camera systems.

Figure 2 .
Figure 2. The TACx4 system.(Ercolin Filho et al., 2017) The first image block acquired for each camera has 31 images in 4 strips.One strip has 10 images acquired with approximately 2600m of flight height above the ground, which provides a ground sampling distance (GSD) close to 22cm, and three strips with approximately 7 images each were acquired with 1600 m of flight height above the ground resulting in a GSD close to 13cm.The Figure3(a)shows the configuration of the first image block.The longitudinal and lateral overlaps in all strips were 80% and 50% respectively.Due to the position and orientation of the four cameras in the main structure, the strips in the lower flight height were flown in the same direction to obtain the same lateral and longitudinal overlap for each camera simultaneously.The Figure3(a) also shows the flight direction of the strips.The second image block has 52 synthetic images in 5 strips.Three strips with 10 images in each one were flown in opposite directions with 1000m of flight height above the ground resulting in a GSD near to 8cm.Two strips with 11 images in each one were taken in opposite direction with 1500m of flight height above the ground resulting in a GSD nearly to 11cm at the ground.The Figure3(b) shows the directions of the strips.The longitudinal overlap in all strips was close to 80%.The lateral overlap of the three strips with 1000m of flight height is approximately to 50% and for the strips with 1500m of flight height the lateral overlap is 100% (coincident strips).(a) First block layout configuration.(b) Second block layout configuration.(Ercolin Filho et al., 2017) (a)) and in the layout as shown in Figure 4(b).

Figure 5 .
Figure 5.The synthetic image generation process.(Ercolin Filho et al., 2017) Parameter Value Width dimension in pixels 12000 Height dimension in pixels 16200 Geometric resolution in µm 5.6 x 5.6 X dimension in mm 67.20 Y dimension in mm 90.72 Nominal focal length in mm 74.000 Interior orientation and mounting parameters.

Figure 6 .
Figure 6.The layout of the independent image block.

Table 1 .
Interior and relative orientation parameters.

Table 4 .
Direct georeferencing validation results in an independent image block.