ADAPTIVE WEIGHTING OF IMAGE OBSERVATIONS FOR SELF-CALIBRATION WITH FISHEYE IMAGES

Fisheye cameras have been widely used in photogrammetric applications, but conventional techniques must be adapted to consider specific features of fisheye images, such as nonuniform resolution in the images. This work presents experimental results of an adaptive weighting of the observation in a self-calibrating bundle adjustment to cope with the nonuniform resolution of fisheye images. GoPro Fusion and Ricoh Theta dual-fisheye systems were calibrated with bundle adjustment based on equisolid-angle projection model combined with Conrady-Brown distortion model. The image observations were weighted as a function of radial distance based on combining loss of resolution and blurring in fisheye images. The results were compared with a similar trial by considering the same standard deviation for all image observations. The use of adaptive weighting of image observations reduced the estimated standard deviation of unit weight by 30 % and 50 % with GoPro Fusion and Ricoh Theta images, respectively. The estimation of relative orientation parameters (ROPs) was also improved (~50 %) when using adaptive weighting for image observations.


INTRODUCTION
The advantages of using 180° field of view (FoV) cameras to collect single-shot images have motivated their use in terrestrial mobile mapping and robotic applications. The potential applicability of these cameras has motivated many studies focusing on the internal geometry modelling of lenses with a large FoV. There are many differences when comparing fisheye and perspective images since the former has high radial symmetric distortion, large-scale variation in the image and between scenes, nonuniform spatial resolution and does not fit the perspective projection (Ray, 2002). A rigorous treatment of these differences needs to be considered to achieve suitable results.
Regarding the nonuniform spatial resolution, most works using large FoV images for close-range applications consider all image observations with the same quality, which is not realistic, due to compression and blurring in the border of fisheye images. Perfetti et al. (2018) discussed the effects of resolution degradation across the fisheye images in the 3D reconstruction. The ground sampling distance (GSD) was used as a parameter to analyse the scale and resolution variation in a scene. A mathematical model describing the resolution distribution in the fisheye images was used to evaluate GSD values. As a result, areas in the image limits were discarded in the 3D reconstruction process due to the loss of resolution, generating a less noisy dense point cloud. To avoid eliminating these observations in the image limits, one alternative is to classify the fisheye image observations as a function of the image resolution, considering the image point's radial distance as a criterion. Lourenço et al. (2012) and Puig et al. (2014) considered different weights for the image observations as a function of the radial distance from the image centre, to describe the omnidirectional image geometry in the image matching process. This approach can be extended to other * Corresponding author photogrammetric processes with fisheye lenses, such as camera calibration. Menna et al. (2018) discussed the degradation of image quality in underwater environments. Their analysis showed that the measurement errors in the image corners were more significant than in the image centre, due to refracted rays' inclination. The quality of measurements in the image centre is five times better than in the corners. Therefore, the image observations were weighted with an empirical function based on MTF analysis (Menna et al., 2017). Standard deviations assigned to the observations in a self-calibrating adjustment varied from 0.1 in the image centre to 5 pixels in the corners. Comparative analysis was performed with a reference experiment considering the same weight for all observations (0.1 pixels). Despite Menna et al. (2017) did not achieve significant improvements with this variable weighting approach, better results were reported in a later work (Menna et al., 2018) with the self-calibration of the GoPro HERO4 Black and Nikon D750 camera in underwater environments and using different standard deviations for the observations. This paper focuses on the mitigation of the effects of the nonuniform spatial resolution of the fisheye images in the accuracy of the image observations. An adaptive weighting of image observations was proposed and assessed in the selfcalibration process. The experimental assessments were performed in two steps. First, the quality of fisheye images was analysed using images of a circular bar-target chart taken with two cameras: Ricoh Theta and GoPro Fusion. This first step aimed to assess the loss of resolution and contrast as a function of radial distance, to enable an objective criterion for the observations weighting. In the second step, self-calibrating bundle adjustment with GoPro Fusion and Ricoh Theta S images was performed using equal and different weights for the observations, the latter using the criteria defined in step 1. The self-calibration was performed using the equisolid-angle projection equations combined with Conrady-Brown distortion model (Conrady, 1919, Brown, 1971, Schneider et al., 2009. Some contributions of this work can be mentioned: the weighting function, combining loss of contrast and resolution; use of dualfisheye images; simultaneous calibration of the dual head cameras with constraints in the stability of relative orientation; results analysis using both inner and relative orientation parameters.

LOSS OF CONTRAST AND RESOLUTION IN DUAL-FISHEYE CAMERAS
An experimental analysis of the quality of the images acquired by GoPro Fusion and Ricoh Theta S dual-fisheye systems was performed. Both GoPro Fusion and Ricoh Theta S systems are composed of two hyperhemispherical lenses (190° FoV). Both dual-fisheye systems are lightweight and compact, which is an advantage for mobile mapping systems. The fisheye image size acquired with the Ricoh Theta system is 1080 x 980 pixels for each sensor, while the GoPro Fusion system has a fisheye image size of 3104 x 3000 pixel for each sensor. Some differences in the internal optics of both cameras can affect image quality. The Ricoh Theta S dual-fisheye system has a very compact design with an internal reflecting prism to redirect the incoming light rays to the sensors ( Figure 1a). This configuration enables the lenses to be aligned and with closer entrance pupils, reducing the occlusion areas since perspective centres are almost coincident. However, optical reflective elements can affect image quality.
GoPro Fusion system uses two single cameras separated, to avoid internal prisms ( Figure 1b). Still, such lenses displacement causes parallax for closer objects and occluded areas, resulting in further problems in the image stitching, when producing panoramas. Figure 1a shows the Ricoh Theta dual-fisheye system with side and top views, and the path of light rays to the sensors, passing through the prisms. Figure 1b shows the GoPro Fusion system with top and side views.
A 360° circular bar-target, with a radius of 30 cm, composed of black and white bars (width step = 4 mm) was built ( Figure 2a). Fisheye images covering this 360° structure were acquired with Ricoh Theta (Figure 2b) and GoPro Fusion 360 (Figure 2c) systems. Only images from one sensor of each camera were used in this experiment. Figure 2b presents a fisheye image taken with the Ricoh Theta S system, and Figure 2c shows an image from the GoPro Fusion dual-camera. Details of the images of the bartarget from Ricoh Theta and GoPro Fusion are presented in Figure 2d and Figure    and (e) details of the images acquired, respectively.
In Figure 2d and Figure 2e, the compression, blurring and scale variation due to fisheye imaging geometry can be seen. The image contrast and resolution decrease on both camera images as a function of the radial distance with reference to the image centre. For instance, in the Ricoh Theta image, the black bar, which has 4 mm of width in the object space, is mapped as 4 pixels in the image centre. The resolution gradually decreases to 3 pixels in the intermediate area, and 2 pixels in the border of the image. Even considering the differences in the image resolutions, it is relevant to extend these analyses to GoPro Fusion images. A 4 mm feature in the object space is mapped as 16 pixels in the image centre, decreasing to 13 pixels in the intermediate area, and 11 pixels in the image border. Figure 3 depicts the grey scale profiles of the black and white bar target evidencing the loss of resolution and contrast in the image limits of (a) Ricoh Theta S and (b) GoPro Fusion images. The differences between maximum and minimum values of the DN (digital number) were calculated to analyse the loss of contrast for both Ricoh Theta (Figure 4a-b) and GoPro Fusion (Figure 4c-d) images. The calculated difference values (red points) and a fitted exponential function (blue line) for Ricoh Theta and GoPro Fusion data are presented in Figure 4a and Figure 4c, respectively. These differences were also presented in percentage, as shown in Figure 4b (Ricoh Theta S) and Figure 4d (GoPro Fusion). The loss of contrast in the limit of the Ricoh Theta image was 73% and, in the GoPro Fusion image, the loss was 41.5%. A similar study was performed for image resolution by computing the ratio between the bar width in pixels and the corresponding value in the object space (in mm). Figure 5 shows the ratios, in pixels/mm, for (a) Ricoh Theta and (b) GoPro Fusion images, in which exponential functions were fitted (blue line). The coefficients (a, b, c) of the fitted exponential functions are presented in Table 2 for both the Ricoh Theta S and GoPro Fusion images.   Table 2. Coefficients (a, b, c) of the exponential functions modelling loss of contrast and resolution.
The resolution decreases in the image border for both cameras ( Figure 5), as well as the image contrast ( Figure 4, Table 2). The problems involving image observations geometry can be handled with proper stochastic treatment, stating the standard deviations of image observations proportional to the loss of contrast and resolution. Equation 1 averages the normalised response of two exponential functions modelling loss of contrast (c index) and loss of resolution (res index). Normalisation was done as the ratio of the function for each radial distance ri to the minimum value (achieved for the maximum radial distance valuermax).
(1) Figure 6 depicts the standard deviation values resulting from this equation, using the coefficients presented in Table 2. These values will be scaled with the a priori sigma to compose the observations' weights.

EXPERIMENTS ASSESSMENTS ON A SELF-CALIBRATION OF DUAL-FISHEYE SYSTEM
This section presents the assessment of the proposal adaptative weighting in a self-calibration process using images of a 360° calibration field acquired with the GoPro Fusion and Ricoh Theta S systems. The results were compared with the self-calibration using all image observations with the same weight in the bundle adjustment.

Data set
Thirteen dual-fisheye images (a total of 26 images) were taken in an indoor 360° Calibration field (Campos et al., 2018) with each dual-fisheye system (GoPro Fusion and Ricoh Theta). The 360° calibration field has 160 ArUco squared coded targets and 100 circular coded targets ( Figure 6). In this work, only ArUco targets (Muñoz-Salinas, 2012) were used, in which the ground coordinates of three points (labelled as 113, 313 and 343) were used as constraints in a self-calibration with bundle adjustment. ArUco targets were automatically measured with sub-pixel accuracy (Tommaselli et al., 2014).
The calibration of both cameras was performed considering a local reference system with origin in point 313 (X =10 m, Y =10 m, Z = 10 m). Figure 7 shows the fisheye images (GoPro Fusion) of sensor 1 (S1) and sensor 2 (S2) taken in the 360° calibration field with ArUco targets detected. The local reference system and the three points used as constraints were also presented in Figure 7.

Self-calibrating dual-fisheye systems method
The simultaneous calibration of dual-fisheye cameras is a recent research topic. Different approaches were proposed to calibrate Ricoh Theta systems, such as Aghayari et al. (2017) and Campos et al. (2018). Castanheiro et al. (2020) presented the selfcalibration of the GoPro Fusion dual-fisheye system using Agisoft Metashape software, in which the image observations were equally weighted. The bundle adjustment was performed for each sensor separately since this software does not estimate the relative orientation parameters (ROPs). In this paper, the calibration of sensor 1 and sensor 2 of both GoPro Fusion and Ricoh Theta dual-fisheye systems were performed simultaneously in the same process with the stability constraints in the relative rotation matrix and base elements between sensors (Tommaselli et al., 2013). The benefits of using stability constraints of ROPs in multi-camera and dual  Castanheiro et al. (2021) demonstrated that among the available fisheye projection models, the equisolid-angle model is the one that can successfully handle points in the hyperhemispherical FoV, which have an incident angle larger than 90°. Considering that both GoPro Fusion and Ricoh Theta dual-fisheye systems are composed of hyperhemispherical fisheye lenses, the calibration processes were based on the equisolid-angle model (Equation 2) combined with Conrady-Brown distortion model. The calibration parameters were estimated based on the unified approach to least squares (Mikhail and Ackerman, 1976) with the CMC (Calibration of multiple cameras) software (Tommaselli et al., 2010) adapted to include fisheye camera models (Marcato Junior et al., 2015).
Equation 2 shows the equisolid-angle projection model, in which (x0, y0) are principal point coordinates; (Δx, Δy) are effects of lens distortion, c is the principal distance; and (XC, YC, ZC) are the point coordinates in the imaging system, which origin is the perspective centre. Equation 3 shows the Conrady-Brown distortion model (Conrady, 1919, Brown, 1971, in which K1, K2 and K3 are the radial symmetric coefficients; P1 and P2 are decentering distortion coefficients; r is the radial distance. (2) The constraints in the stability of relative orientation components were imposed with standard deviations of 0.0005° for each relative rotation angle, and 0.001 mm for base elements. Standard deviations of initial values for the exterior orientation parameters (EOPs) were set to 0.5 m for camera positions and 10° for attitude angles. In a first trial (Experiment A), the self-calibration was performed considering a standard deviation of 0.5 pixels in columns and rows components for each image observation. This value was defined based on the subpixel accuracy achieved with the software for ArUco targets extraction. However, it was observed that the measurement accuracy decreased due to both the deformation caused by the fisheye sensor geometry and the nonuniform image resolution, as discussed in Section 2. Therefore, in a second trial (Experiment B), the weights of the image observations were based on Equation 1, which assigns weights to the image observations based on the analysis performed in Section 2.
In summary, two self-calibrating trials (Experiment A and B) were performed with both GoPro Fusion and Ricoh Theta images to evaluate the influence of different observations weighting in the calibration parameters estimated for dual-fisheye systems. The bundle adjustment trials were performed with 2748 and 2460 automatically measured points, for the GoPro Fusion and Ricoh Theta images, respectively. The experiments were assessed with check points and distance. Since the 3D coordinates of all ground points in the 360° calibration field were known, those which were not used as constraints in the calibration, were used as independent check points to evaluate the accuracy of the bundle adjustment. Furthermore, eight well-distributed distances directly measured in the 360° calibration field, using a calliper with a precision of 0.2 mm were compared with the distances calculated from the ground coordinates of points estimated in the self-calibrating bundle adjustment. The standard deviation of c, x0 and y0 were much less than 1 pixel in both experiments, and the results were similar. Figure 8 shows the curves of (a) radial symmetric distortion and (b) the curves with the differences in the effects of radial and (c) decentering distortion curves, with distortion parameters estimated in Experiment A and B for each GoPro sensor. In both experiments, the radial distortions presented high values when comparing to perspective cameras. This behaviour is expected for fisheye images (Schneider et al., 2009), requiring a suitable model to achieve accurate results. Although the radial distortions coefficients were similar for both experiments, the effects near the image borders were different (approximately 8 pixels), which can be explained by the lower weight assigned to the observations in the image limits. The maximum decentering distortion effects were much smaller than those of radial distortion, approximately 1 pixel (0.002 mm) for sensor 1 in both experiments. For sensor 2, the decentering coefficients could be neglected in the photogrammetric processes, since its effects were much lower than 0.5 pixels. In order to maintain a standard set of parameters, both distortions were considered in the calibration process.

GoPro Fusion
Improvements were observed in the standard deviations of camera orientation angles (σω, σφ, σκ), as can be seen in Figure 9, which shows the average values. The improvements in the estimated precision of the camera orientation were approximately 5%. The standard deviations of camera position improved in the same proportion. The most significant improvements were noticed in the estimated ROPs, mainly in the relative angles, when using the proposed adaptive weighting for observations. Table 5 shows the ROPs calculated from the EOPs estimated in the bundle adjustment and the standard deviations. ROPs are the base components (BX, BY, BZ), distances between cameras perspective centres (DPC) and relative orientation angles between each sensor (Δω, Δφ, Δκ). The standard deviations of base elements have been improved, mainly for the BZ component, with a 22% reduction. The standard deviation of DPC decreased by 20% when using the proposed adaptive weighting function. Noticeable improvements occurred in the relative orientation angles with the standard deviation of Δω decreasing to less than 1" (almost 50% of reduction), indicating stable results. The standard deviations of Δφ decrease by 39%, while for Δκ remained almost the same.

ROP
The bundle adjustment accuracy was evaluated both with distances measured in the 360° calibration field and check points. The same distances were calculated with the estimated 3D coordinates of the points. The RMSEs of the discrepancies between control and calculated distances for both experiments were approximately 5.3 mm, which is compatible with the pixel size in object space units (varying from 3 to 5 mm). In addition, the RMSEs of the check point coordinates were also calculated. In Experiment A, the RMSEs were 3.8 mm, 1.5 mm, and 2.8 mm in X, Y and Z coordinates, respectively. In Experiment B, the RMSEs of check point coordinates were reduced to 2.9 mm, 1.3 mm, and 2.3 mm in X, Y and Z coordinates, respectively. Thus, these results show the improvements with this adaptive weighting.

Ricoh Theta S
The analysis of the results obtained in GoPro Fusion's selfcalibration was repeated for the Ricoh Theta dual-fisheye system. Table 6 shows the statistical analysis (mean, standard deviations and RMSE), in pixels, of the resultant component of image coordinates residuals, and the estimated standard deviation of unit weight (0) for each experiment, considering the a priori value ( 0) as 1. The improvements in the estimated precision of the camera orientation (σω, σφ, σκ) were approximately 14%. Figure 10 shows the estimated standard deviations of the camera orientation angles in Experiment A (same weight) and Experiment B (adaptative weight). As in the GoPro Fusion experiments, significant improvements were observed in the estimation of ROPs.  (Table 8). A reduction of standard deviations can also be seen in the DPC (24%). As for GoPro Fusion experiments, the significant improvements were noticed in the estimation of the relative orientation angles, in which the standard deviation of Δω and Δφ decreased by 55% and 32%, respectively. The RMSE of the discrepancies between control and calculated distances for both experiments were 7 mm. The RMSEs of the 3D coordinate of the ground points were 5.4 mm, 3.6 mm, and 4.4 mm for X, Y, and Z coordinates, respectively, in Experiment A. These RMSEs were reduced to 4.0 mm, 2.7 mm, and 3.3 mm for X, Y, and Z coordinates, respectively, in Experiment B.

CONCLUSION
This paper presented an experimental analysis of dual-fisheye cameras' calibration, using different strategies for observations weighting. First, the image quality of both Ricoh Theta and GoPro Fusion were evaluated. Blurring and compression in the fisheye images' border were noticed, resulting in loss of contrast and resolution. A black and white bar target was used to assess the magnitude of these losses and to estimate a percentage of degradation, which was used to set an adaptive weight for the observations in self-calibration adjustment, depending on the radial distance. The proposed treatment of the fisheye image observations was tested in a self-calibrating bundle adjustment of the GoPro Fusion and Ricoh Theta dual-fisheye system. The image observations were weighted based on an exponential function obtained from the analysis of the loss of contrast and resolution of the images. In general, the self-calibrating bundle adjustment based on the equisolid-angle projection combined with stability constraints of relative orientation angles and base elements presented relevant results. Improvements were achieved when using the adaptive weighting compared with a self-calibration considering the same standard deviation for all image coordinates.
The weighting of the image observation can be done with other objective criteria, such as loss of resolution estimated with the MTF. The equation used to assign weights in this work averaged two exponential functions modelling loss of contrast and resolution, but it should be recommended to study the effects of using both criteria isolated and with different combinations. More accurate quality control is also recommended to assess the weighting strategy's effects in the coordinates of the estimated points in the object space. The technique used in work (constraints in the stability of ROPs) requires the assignment of standard deviations for the constraints, and the effects of varying these values should be experimentally studied.
More studies are recommended on the influence of image observations weights in other photogrammetric processes, such as 3D reconstruction. Further criteria could be used to weight the fisheye image observations, such as parallactical angle in the intersection of rays in the object space.