CAMERA CALIBRATION MODELS AND METHODS FOR CORRIDOR MAPPING WITH UAVS

Camera calibration refers to the modeling of the relationship between the coordinates of object points and their projections on the image plane. This is usually done by parametric models that describe the physical properties of the lens systems and camera assemblies, such as the camera principal distance, the principal point, and various types of optical distortions. In photogrammetry, accurate knowledge of the parameters of such models, often referred to as Interior Orientation (IO), is of ultimate importance. In this work, we target advanced corridor mapping applications with UAVs. In this scenario, the camera calibration is not completely observable due to the unfavorable geometry of the flight trajectory (e.g., no cross flight lines available and a single altitude) and needs to be determined beforehand. Further challenges are introduced by the limited mechanical stability of UAV-grade cameras. This may cause slight variations in the IO that need to be recovered while processing production flights. We review and compare two well known camera models, the Brown-Conrady and the Ebner’s self-calibration functions, in 36 calibration setups and provide a discussion of the results, where sub ground sampling distance accuracy in the checkpoints was achieved for some, but not all, configurations.


INTRODUCTION
Camera calibration, or the camera Interior Orientation (IO), is an essential prerequisite to determine precise and accurate three dimensional, metric, information from images. It models the relationship between the coordinates of object points in the camera reference frame and the corresponding image coordinates. The knowledge of such relationship allows, among others, to recover three dimensional object coordinates from multiple views by means of a bundle adjustment algorithm (Triggs et al., 1999). The IO is specific to each particular assembly of camera and lenses and may change in time (Läbe, Förstner, 2004).
A wide range of digital cameras commonly employed in photogrammetry are equipped with lenses designed to obey the perspective projection law (i.e., the collinearity equations). However, this is seldom true in reality because of at least four effects related to imperfections in their construction (Fraser, 1997): i) symmetric radial distortion, ii) decentering distortion, iii) image plane unflatness and iv) in-plane image distortion. These can cause non negligible deviations from collinearity. At least two families of models have been proposed to account for this: Brown-Conrady (Brown, 1971) (abbreviated to Brown in the following) and Ebner's self calibration functions (Ebner, 1976). These models are characterized by a variable number of parameters that need to be determined, through a process called camera calibration (Remondino, Fraser, 2006). Camera calibration is typically approached by acquiring a large set of images (typically more than 50) of an object (such as a chessboard or a calibration field composed of several Ground Controls Points: GCPs) in order to describe accurately the camera behavior. The parameters of the chosen camera model (e.g., Brown) are then estimated to best fit the observations. A very * Corresponding author well known approach in computer vision relies on the use of easily recognisable targets (e.g., checkerboards) whose metric dimensions are known. Several established tools are available to determine Brown's distortion coefficients in this setup, e.g., (Bouguet, 2000), however, these often yield correlated estimates for the calibration parameters. One reason is that all the known points lie on the same plane. Even in three dimensional laboratory calibration setups, the depth of field is typically much smaller with respect to target applications, such that erroneous estimates for the distortion coefficients may be obtained (Lichti et al., 2008). Therefore, photogrammetrists typically prefer to perform camera calibration in conditions that are similar with respect to the target application. For example, in aerial photogrammetry, dedicated flights are performed over a field equipped with several GCPs: the orientation of the camera, along with the IO, are recovered during the bundle adjustment. This approach also relies on additional points identified automatically (tie-points) whose object space coordinates are not known.
Further issues related to camera calibration arise in the emerging field of aerial photogrammetry based on Micro Aerial Vehicles (UAVs), which are becoming an essential tool for surveyors, engineers and scientists (Colomina, Molina, 2014). Here, consumer-grade cameras and lenses are employed due to payload size, weight and cost limitations of UAVs. The mechanical stability of such sensors is uncertain: for example, the µm level positioning and alignment of the lens might change when vibrations, bumps during landing, temperature variations, etc. occur during the operation of a UAV. This implies that the IO determined from one calibration flight may not be directly applicable, as it is, in subsequent production flights (Cramer et al., 2017). However, in general it is not possible to determine the IO from scratch in production flights as i) the geometry of certain mapping missions, e.g., for corridor mapping, is such that the IO is not fully observable, ii) practitioners strive to reduce the number of GCPs since they are time and cost intensive. This calls for sound methods to exploit the IO from dedicated calibration flights but at the same time recover minor variations due to the mechanical instability of consumer-grade cameras.
In this work we focus on long corridor mapping missions with UAVs where no GCPs are available. Here, a high quality apriori IO is essential. We consider five instances of models from the Brown's and Ebner's families, differing by the number of IO parameters, which we determine using two different calibration setups. Next, we employ these IOs in a 2 km long corridor mapping flight, first using only values determined during calibration, and then using three different strategies to estimate corrections for camera mechanical instabilities during the bundle adjustment. Practically unbiased Check Points (CPs) errors and sub-Ground Sampling Distance (GSD) root mean squared error (RMS) were achieved, in certain cases, approaching the accuracy to which CPs are known. We've also found combinations of models and calibration strategies which do not provide satisfactory results, for which we investigate the causes and provide an extensive discussion.
This work is organised as follows: in Section 2 we review the Brown and Ebner's calibration models in a unified formulation. In Section 3 and 4 we review the calibration state of the art in Integrated Sensor Orientation (ISO) and we present three strategies to re-calibrate the IO during production flights. In the last sections we present the results of a rigorous experimental evaluation of the different models and strategies presented in a real-world corridor mapping application.

CAMERA MODELS
In this section we review two approaches to model the departures from collinearity typical of many narrow angle cameras and lenses. The first is the physical oriented approach, the Brown function, while the second, more numerically oriented, is given by the Ebner self-calibration functions, and later extensions. Whereas the principles behind those are well known, several slightly different formulations have been presented in the literature. Thus, we report those used in this study in the following analysis.
We define the projection function π : R 3 → R 2 as: where [X, Y, Z] T are the object coordinates of a point with respect to the camera reference frame (in meters) and [x, y] T is the corresponding (unit-less) projection on a plane at unitary distance from the optical center of the camera. The well known pinhole camera model is found by multiplying π by the principal distance of the camera, e.g., in pixels. Many lens systems are designed to best follow this relation.
More complex camera models, able to account for different kinds of distortion effects introduced by real lens systems, build on π as follows: with ξ : R 2 → R 2 . Image coordinates [x , y ] T are typically expressed in pixels. The function ξ involves a set of parameters which are generally referred to as Additional Parameters (APs), or camera Interior Orientation (IO).

Brown distortion model
In Brown's distortion model (Brown, 1971), the function ξ(·) is defined as the composition of two other functions ξ = ξ2 • ξ1: the output of the function ξ1 is used as input of the function ξ2.
In this work we consider three specific instances of the Brown model, referred to as Brown10, Brown15 and Brown18 in the following, which differ in the total number of parameters employed. Whereas f , ppx, ppy, B1 and B2 are always considered, the models differ in the order of the polynomial expansions for the radial and tangential distortion (with respect to r 2 ):

Orthogonal polynomials
A second family of models for camera distortions was introduced in (Blázquez, Colomina, 2010) building on a typical example of Orthogonal polynomials: the Ebner's self-calibration functions (Ebner, 1976) (extensions of Ebner's Orthogonal polynomials such as Grün polynomials: (Grün, 1986) are also referred in (Blázquez, Colomina, 2010) but increase the number of parameters). We refer the reader to the original publications for the derivations. The function ξ reads as follows: ξ : where bx and by are the width and the height of the image (in pixels) divided by twice the nominal focal length (in pixels). This model is fully defined by the set of 18 parameters aij and bij and will be referred to as OrthoPoly18 in the following.
Brown's and Orthogonal Polynomial models are intrinsically different and it is not possible to give a closed form expression that would map parameters from one model to the other, except for the following:

CAMERA CALIBRATION
In order to properly compensate for the distortions introduced by the lens system, one has to first choose which model is best suited for the camera in use, and second determine the values for the parameters defining the model (the IO, or the APs). The first step is often left to user experience, even though modern photogrammetry software is able to automatically select the most significant parameters to be considered within a model family (e.g., choose between Brown10 and Brown15). The second step is called calibration. If the calibration is performed without the use of an external instrument to characterize the sensor, we speak of self-calibration. The methods are well known, and we are going to summarize them in the following.
Camera calibration via aerial photogrammetry is commonly achieved by establishing a dense network of accurately georeferenced ground control points (GCPs) over a calibration field. Dedicated flights are executed to collect several images in a favourable geometrical configuration (e.g., cross flight lines, different elevations and a mix of nadir/oblique views). The images are then oriented by means of the bundle adjustment algorithm, which determines the camera exterior orientation (EO), the parameters of the distortion model (IO), and possibly their precision and reliability. This approach is commonly referred to as Indirect Sensor Orientation (InSO).
In InSO, it is sometimes difficult to achieve optimal decorrelation between the camera EO and IO, and within the IO itself, even when the calibration have been performed with several camera orientations and flight heights. This is the case when the GCPs lie on the same plane, such as when the computer vision approach to camera calibration, based on checkerboards, is employed. In this case the parameters corresponding to the focal length (f in BrownXX and a21, b12 in OrthoPolyXX) remain correlated with the camera position.
If the aerial mapping platform is equipped with a survey grade GNSS receiver, and (optionally) an IMU, the additional information made available by these sensors can be introduced in the bundle adjustment. This approach is called Integrated Sensor Orientation (ISO), see for example see (Rehak, 2017) for a comprehensive description focused on UAVs.
In ISO, GNSS and inertial observations are fused together by means of a Kalman filter/smoother in a pre-processing step. This calculates positions and orientations for the camera that can be used as prior information in the bundle adjustment (aerial control, in photogrammetry jargon). More modern approaches consider a single step where both image and raw inertial/GNSS observation are adjusted together (Cucci et al., 2017). The availability of the extra information from GNSS and inertial sensors helps to de-correlate the camera model parameters from the camera exterior orientation.
In UAVs, only MEMS IMUs can be employed due to space and take-off weight limitations. These are substantially less accurate with respect to tactical or navigation grade IMUs commonly employed in airborne photogrammetry. Thus, the GNSS/inertial solution might be biased, or not sufficiently accurate. In this case, a very effective approach is to use relative, instead of absolute, position/orientation control in the bundle adjustment, as proposed in (Blázquez, Colomina, 2012) and further investigated for the UAV scenario in (Rehak, Skaloud, 2016). This technique exploits the fact that the GNSS/inertial solution may not be sufficiently accurate, but is locally precise, and thus it can be effectively employed to constrain the relative change between subsequent image orientations. Furthermore, relative orientation control eliminates the need to determine the camera boresight with respect to the IMU. We refer the reader to the original publications for the details.

Orthogonal Polynomials over-parameterization
In InSO, OrthoPoly18 suffers from over-parameterization: some of the parameters defining the model fully correlate with the exterior orientation of the camera. This means that if no absolute aerial orientation control is available, unreliable estimates or worse, numerical instabilities and singularities will be obtained during the bundle adjustment.
To address this issue, in (Blázquez, Colomina, 2010), the authors introduced six constraints on the parameters aij, bij: If these are used to simplify Equation 5, the original 12 parameter Ebner's self-calibration functions are obtained.
In this work we've chosen to omit the first two constraints in Equation 6, since they correspond to f and pp in BrownXX. These are important for modeling consumer-grade cameras where the lens system can not be considered fully geometrically stable in time, or in large camera systems where they depend on environmental factors such as temperature and air pressure. Simplifying Equation 5 with the remaining three constraints gives OrthoPoly15, which requires at least absolute aerial position control during calibration.

RE-ESTIMATING THE INTERIOR ORIENTATION
In UAVs, consumer grade digital cameras are commonly employed. These are not built for photogrammetry applications and can not be considered mechanically and geometrically stable. In particular, the alignment of the lens with the imaging sensor is subject to change due to external stresses such as vibration or temperature change. The lens radial distortions are in general considered to be more stable, but can still depend on environmental factors. This means that the IO may slightly change with time and thus be different between calibration and production flights.
To account for this, IO is often re-calibrated during production flights. This means that the bundle adjustment algorithm is initialized with the IO determined from calibration flights, and estimated corrections which account for camera instabilities are applied while processing subsequent production flights. Indeed, particular care needs to be taken as the geometry of production flights may not guarantee the observability of the full IO (e.g., in corridor mapping). In this work we consider three strategies which are discussed in the following.

Leading (Lead)
The parameters f , ppx and ppy in BrownXX and the corresponding parameters a11, a21, b11 and b12 in OrthoPolyXX, are estimated, while the remaining parameters are kept fixed. This option is well known and implemented in professional aerial photogrammetry softwares such as Pix4D Mapper and Agisoft Metashape.

A-posteriori covariance (APC)
Let Θ be the vector of the interior orientation parameters, e.g., for BrownXX: The a-posteriori covariance matrix of this vector, ΣΘ, is available as an additional output from the bundle adjustment run on the dedicated calibration datasets. The square root of the diagonal of ΣΘ is typically reported as the error bound for the IO vector. On the off-diagonal terms, often neglected yet significant to experienced practitioners, we find the covariance between two parameters. Non-complete observability of some parameters is spotted by computing the correlation matrix from ΣΘ and looking for off-diagonal terms which are close to one in absolute value.
We can exploit the prior information on the IO available from the calibration flight, including the residual correlations between the single parameters, for the re-estimation of the IO. During the bundle adjustment for the production flight we reestimate the entire IO vector, where we include an extra observation equation as follows, where Θ0 have been determined in a calibration flight.

A-posteriori covariance inflated (APCI)
While ΣΘ is the best estimate of uncertainty for the camera IO just after the calibration flight, the use of this information in production flights does not account for possible changes in IO due to camera instability.
We thus propose to scale up the diagonal elements corresponding to the leading parameters defined in Section 4.1. The degree to which each parameter is up-scaled is determined by the amount that they are expected to change with time.

EXPERIMENTAL EVALUATION
In this work we address challenging corridor mapping applications where we seek to achieve a ground accuracy that is better than the GSD using no ground control points. In these cases, it is essential to determine a reliable IO beforehand since the geometry of such production flights does not allow for selfcalibration. Indeed, in corridor mapping missions we typically do not observe at multiple elevations and/or orthogonal flight-lines. Furthermore, in our case no ground control point is available. Another important element is the correct choice of the camera model, which needs to be able to compensate for lens distortion while minimizing the number of parameters employed.
For all the flights, we have employed an aerial photogrammetry payload tailored for a small fixed wing UAV (Rehak, Skaloud, 2015). This payload is composed of: i) a custom, 20 Mpx camera for aerial photogrammetry developed by IGN, France (Martin et al., 2014) with Zeiss Biogon 35 mm lens, ii) a Gecko4Nav redundant IMU board with two Intersense NavChip MEMs IMUs (Clausen, 2019), iii) a Topcon B110 GPS/GLONASS L1/L2 receiver.
We determine the IO of the camera during the bundle adjustment as described in Section 3. Two different strategies are considered: 1. CF1. A set of close-range images of a calibration field densely covered with mm-accurate GCPs (surveyed with theodolites) is oriented without the use of any aerial control (Indirect Sensor Orientation: InSO). The images are taken with converging geometry and from distances between 8 and 12 m.
2. CF2. A block mapping mission is flown over a large area equipped with several cm-accurate GCPs (surveyed by post-processed GNSS). The images are oriented with aerial control (Integrated Sensor orientation: ISO with absolute position and relative orientation control). Two flight altitudes are considered, i.e., 120 and 150 m AGL. See Table 1 for a detailed description of the considered flights.
For both CF1 and CF2, we consider five choices for the camera models: i) Brown10, ii) Brown15, iii) Brown18, iv) OrthoPoly15 and v) OrthoPoly18. See Section 2 and Section 3.1.
We test the IO determined with each combination of calibration strategy and camera model on a challenging 2 km N-S oriented corridor mapping mission, referred to as the Production Flight (PF) in the following, see again Table 1. This can be done by either keeping the IO fixed as it has been determined from the calibration flights (Fix, in the following), or by correcting for estimated camera instabilities in the bundle adjustment starting from the known initial values, according to any of the strategies presented in Section 4, i.e., Lead, APC, APCI.
For each of the 5 (camera models) × 4 (re-calibration strategies) × 2 (calibration setups) − 4 (because OrthoPoly18 cannot be applied in CF1 where no aerial control is available) = 36 experiments, we compute the statistics of the CPs error, in terms of mean, maximum and root mean squared error (RMS). See Table 2. To ease the interpretation, the cells are color coded according to the RMS, where white and red are associated with the lowest and highest values, respectively.

Camera calibration strategy
The IO parameters determined in CF2 perform better than those determined in CF1. In fact, when the first ones are directly applied in PF with no correction, (Fix), they always yield better results.
In Figure 1 we have reported the correlation matrix for the Brown10 IO parameters as determined from CF1 and CF2. The matrix for CF1 shows substantially higher correlations between parameters as compared to that of CF2, notably for the lead parameters. Note that almost complete correlation exists between the principal distance and the y component of the principal point (ρ f,ppy = −0.97). This means that the values for the IO parameters could not be fully resolved, and that one could manipulate those to a relatively large extent (e.g., along the direction of the first eigenvector of the covariance matrix) and obtain similar values for the image observation residuals.
We argue that the reason for this is that aerial control in CF2 brings extremely valuable information to de-correlate IO parameters. This is an argument for the use of aerial control in camera calibration flights.

Camera models
We observe that the physical models BrownXX outperform OrthoPolyXX in the considered case. Indeed, some aspects of the distortions specific to the considered lens system could not be compensated using any of the OrthoPolyXX models.
To show this, we attempt to compare the departures from collinearity as observed from tie-points and as implied by the estimated IO model. For each tie-point Pi we define a vector vP i as the difference between its image projection implied by the pinhole camera model and the actual tie-point image coordinates: where Pi, f , ppx and ppy are taken from the output of the bundle adjustment for PF with OrthoPoly18. A vector vM , i.e., the corrections implied by a given IO model, can be defined analogously for any point of the image plane.
v can be decomposed in two components, radial, in the direction of the principal point, and orthoradial, i.e., orthogonal to the first, see Figure 2. Figure 2. Decomposition of the departure from collinearity vector, vP i into its radial (vρ) and orthoradial (v θ ) components for a tie-point observed at image coordinates zP i .
As an example, in Figure 3 we have plotted the radial component of vM along the diagonal of the image plane (blue curve) and the observed vP i for the tie-point observation close to such diagonal, for all the considered IO models. It is readily evident that BrownXX fits the observations better than OrthoPolyXX. Note the overfit in Brown18 in the extremities of Figure 3c. In Figure 4 and Figure 5 we have plotted the radial and the orthoradial components of vM , respectively, as a function of the image coordinates. We can observe that no clear radially symmetric correction is implied by OrthoPolyXX, see especially Figure 4d and 4e.
These results may hold for the specific lens at hand, a high quality Zeiss Biogon, and may not be generally applicable. However, the numerical models based on Ebner's self-calibration functions were developed in the seventies when aerial photogrammetry was performed by means of film cameras and additional corrections due to film out of plane deformations were important. This is why numerically inspired models were developed. We suggest that this may no longer be appropriate with modern digital cameras. In fact, with digital cameras, even consumer grade, the distortion effects are dominated by a component that is radially symmetrical. For this, no explicit term is present in Ebner's functions.

Re-calibration
With respect to the re-calibration strategy, we argue the following: 1. In general, it is not optimal to directly employ the IO as determined in calibration flights (Fix), at least with consumer grade cameras, since it is well known that the leading IO parameters may slightly but significantly change due to camera mechanical instability.
2. The well known strategy of re-estimating the leading IO parameters (Lead) improves results with respect to Fix in all the considered experiments and yields some of the best results.
3. APC and APCI further improve upon Lead and such improvement is more marked when the available IO parameters from calibration are correlated, as in CF1. Unfortunately, these strategies are not implemented in commercial aerial photogrammetry software.
4. It is not clear whether APCI is better than APC. We note that APCI has one tuning parameter which affects how much the diagonal components of ΣΘ corresponding to the leading IO parameters are inflated. This is related to how much those are expected to change with time and is an empirical parameter that relies upon user experience and the specific camera being utilized.
Finally, we note that no matter what re-calibration strategy we employ, better results are obtained when IO from CF2 are employed. This is related to the fact that the geometry of corridor mapping missions enables correction for imperfect IO, even when the full a-priori uncertainty ΣΘ is employed.

CONCLUSIONS
In this work we have investigated different calibration strategies and IO models targeting corridor mapping applications. In this scenario, a-priori knowledge of the IO is essential. We've seen that certain choices may lead to poor results and should be avoided, while others allowed us to obtain sub-GSD residuals at checkpoints using no GCP.
Our results can be summarized in the following guidelines useful to UAV practitioners who are targeting delicate mapping missions in which redundancy and ground control are limited: 1. prefer calibration flights in similar configurations (e.g., altitude) with respect to production flights, 2. use aerial control during camera calibration, as opposed to laboratory setups or Integrated Sensor Orientation, 3. allow some form of re-estimation of the camera calibration during the bundle adjustment of production flights (unfortunately only Lead is available in commercial bundle adjustment software), 4. strive to reduce the number of calibration parameters, as long as they are sufficient for modeling the lens at hand, 5. prefer the Brown family model over Ebner's for lenses with a predominant radial distortion component.