ASSESSMENT OF CHROMATIC ABERRATIONS FOR GOPRO 3 CAMERAS IN UNDERWATER ENVIRONMENTS

With underwater photogrammetric mapping becoming more prominent due to the lower costs for waterproof cameras as well as lower costs for underwater platforms, the aim of this research is to investigate chromatic aberration in underwater environments. Chromatic aberration in in-air applications is to be known to systematically influence the observations of up to a few pixels. In order to achieve pixel-level positioning accuracy, this systematic influence needs further investigation. However, while chromatic aberration studies have been performed for in-air environments, there is a lack of research to quantify the influence of chromatic aberration in underwater environments. Using images captured in a water tank from three different GoPro cameras in five datasets, we investigate possible chromatic aberration by running two different adjustments on the extracted red (R), green (G) and blue (B) bands. The first adjustment is an adjustment that calculates the interior orientation parameters for each set of images independently in a free network adjustment. The second adjustment solves for all interior orientation parameters (for R, G, and B channels) in a combined adjustment per camera, constraining the point observations in object space. We were able to quantify significant chromatic aberrations in our evaluation, with the largest aberrations observed for red band followed by green and blue.


INTRODUCTION
With more affordable water-proofed sensors (e.g. GoPro, 2018) and more affordable underwater remotely operated vehicles (ROVs; e.g. BlueRobotics, 2018) operating in water and unmanned water vehicles (UWV; Sardemann et al., 2018) operating on the water surface available, underwater photogrammetry utilising low cost systems is becoming more prominent and is being utilised for photogrammetric mapping. The applications of underwater photogrammetry range from heritage preservations such as mapping of ship wreck sites (Balletti et al., 2015;Diamanti et al, 2017;Ktistis et al., 2017;Van Damme, 2015) and mapping of archaeological sites and objects D'Amelio et al., 2015;Denker and Oniz, 2015), to the mapping of organic environments such as underwater meadows (Rendea et al., 2015) and coral reefs (Burns andDelparte, 2017, Drap et al., 2017), and to the mapping and monitoring of engineering structures (Przybilla, 1988). However, underwater photogrammetric mapping faces a number of challenges compared to traditional in-air photogrammetry starting from lighting conditions to moving objects such as organic growth and fish in the field of view of the camera, and the introduction of location information.
For instance, while sufficient sun light is available to illuminate the scene in shallow water, constantly moving wave patterns can create issues during the image matching. While data capture during overcast days can reduce the effects, it is also possible to reduce or even eliminate it through image processing, e.g. using Convolutional Neural Networks (CNN) applied to the image histograms (Agrafiotis et al., 2018). In contrast, artificial light, often fixed to the ROV, is used in deep water environments. The changing position of the ROV leads, and the installation of the light on the ROV will impact the light conditions (Gupta et al., * Corresponding author 2008), which leads to poor visibility, lack of contrast and colour casting (Mangeruga et al., 2018). Changing lighting conditions are also known to impact image matching. Another challenge is the establishment of control, which is especially relevant to the monitoring of structures in an environment where every component is prone to changes. Neyer et al., (2018) overcame this issue applying the geodetic principles of trilateration and height difference measurements.
Another important aspect compared to air is the different refraction and chromatic characteristics in the water. For instance, the index of refraction depends on the medium, i.e. the refractive index of water is known to change by up to 2% with wavelength, depth, temperature and salinity and the shape of the camera housings and port may change with depth due to changing pressure levels (Shortis et al, 2009). Regarding this challenge, the majority of the research focus is on the radiometric calibration of the images in order to implement colour correction and enhancement of image matching approaches (Agrafiotis et al., 2017). Another aspect is the geometric model for the handling of refraction effects on the optical path. Ray tracing can be applied to airborne images, enabling the derivation of bathymetry (Mandlburger et al., 2017) and for geometric correction of underwater images (Harvey and Shortis, 1998). A modular geometric model for underwater photogrammetry was introduced in Maas (2015).
However, one aspect to which little attention has been paid is chromatic aberrations. Considering that especially low cost cameras are used for underwater photogrammetry, and considering the different characteristics of light rays underwater leading to different chromatic aberration characteristics from in air to underwater operation., it is important to not only qualitatively evaluate chromatic aberration but to quantify the effect. Based on the results, it will be possible to improve the processing of underwater images, e.g. considering strategies for accurate image point observations and improving the results of colour mapping for the creation of ortho-mosaics. Hence, this paper aims to focus on this topic, and aims to quantify chromatic aberrations for a low cost camera system (GoPro Hero 3) in a controlled underwater environment. A comparison of in air and underwater comparison is outside of the scope of this paper. This paper is structured as follows. The next section will focus on related work regarding chromatic aberration and will also include above-water studies due to the limited number of researchers working on this topic. Then, after the background is introduced in section three, section four will introduce the data and the method used to quantify the chromatic aberration. The results are presented in section Five, and the paper concludes in section Six.

RELATED WORK
The existence of chromatic aberration in underwater images is shown in a number of publications. For instance, Menna et al. (2017) analysed the optical aberration in underwater photogrammetry with flat and hemispherical dome ports utilising a customised calibration field introduced in Menna et al. (2016). While the corners of the dome port show some blur effects due to field curvature, the flat port shows not only blur effects but also chromatic aberration. Nevertheless, only the green channel was utilised in the processing of the images. Hence, a quantification of the chromatic aberration was not performed. Based on the findings in Menna et al. (2017), further research published in Menna et al. (2018) employed empirical weighting to consider image degradation towards the borders of the image, i.e. a stochastic model for image observations that penalises those that are more affected by aberrations and departure from the pinhole model were applied within a self-calibration (Fraser 1997) using the Brown model (Brown, 1971). Kaufmann and Ladstaedter (2005) did not quantify chromatic aberration explicitly as their main aim was to eliminate the effect of chromatic aberration for images. In their paper, images captured in-air are used in a least squares adjustment in which points from the green image are transferred to the red and blue images. Based on the transformation parameters displacement vectors are computed in order to eliminate the effect of the chromatic aberration. The conclusion was that the chromatic misregistration is generally more or less radially-symmetric, the amount of the radial image displacement is not necessarily a linear function of the radial distance, the geometric centre is likely not to coincide with the principal point and, finally, that the chromatic misregistration may in some instances be of a general geometric type (e.g. non-radial-symmetric).
While Kaufmann and Ladstaedter (2005) were able to remove the chromatic shift, van den Heuvel et al., (2006), who applied a similar approach, were not able to remove the misalignment. The correction calculated were in the sub-pixel range while the actual shift was up to 5 pixels. It was concluded that further assessments are required. Matsuoka et al. (2012) and Rudakova and Monasse (2013) had similar aims, and only corrected for the chromatic aberration. Rudakova and Monasse (2013) first detected key points represented in pixel coordinates using the red, blue and green bands separately. The lateral misalignments between the red (or blue) and the green planes are corrected by identifying the parameters of polynomials. A dense correction vector field is then deduced by a general polynomial model to correct for the chromatic aberration. Matsuoka et al. (2012) concluded that the correction method, which is based on the assumption that the magnitude of chromatic aberrations can be expressed by a cubic function of the radial distance from an image frame centre, is the best method to correct chromatic aberrations satisfactorily in many cases. Furthermore, they could show clearly that if green is set as the one in focus, then a barrel distortion of the red channel and a pincushion distortion of the blue channel can be observed. Luhmann et al. (2006) was able to quantify the chromatic aberration of two camera systems in-air by calculating separate sets of interior orientation parameters per image band. Image observations were taken in three colour channels instead of just utilising the green band. While the higher quality rated camera (Canon EOS D1 Mark II) showed little deviation in the principle distance (4m), the effect was much stronger in the low cost system (Fuji S2 Pro, 27m), which exhibited longitudinal chromatic aberrations. Introducing observations from the red, green and blue channel in the bundle adjustment, and introducing additional constraints in object space, numerically improved RMS values and, therefore, a better relative precision could be achieved. Overall, a significant accuracy enhancement of about factor 1.3 could be achieved. Robson et al. (2014) describe the effect of wavelength on camera calibration parameters in their research on low cost monochromatic camera systems to. Results demonstrate not only a dependency of the principal distance on wavelength, but also of the principal point offset caused by physical misalignment of lens elements and the sensor cover glass. Furthermore, the radial distortion increases as a function of the radius, and the greatest distortion is observed around 480 -500 nm (blue light) with less for both lower and higher wavelengths.
As the motivation of this paper is to quantify chromatic aberrations in underwater environment using a low-cost sensor system (GoPro) a similar strategy as used by Luhmann et al. (2006) will be applied in this research.

BACKGROUND
While this section is only a brief introduction, more details about chromatic aberration is available in ASPRS (2004). Indeed, the explanations provide in in this section are based on this reference.
In general, image imperfections are called aberrations. Next to spherical aberration, coma, astigmatism, curvature of field and distortion, in colour images there are also chromatic aberrations. A ray of white light in object space refracted at a lens surface becomes a small spectral fan of rays owing to the dependence of the index of refraction on wavelength. Aberrations can be expressed as longitudinal and lateral (transverse) measure.

Longitudinal chromatic aberrations
Longitudinal aberration is measured in the direction parallel to that of the optical axis. The variation of the principle distance is caused because wavelengths on the shorter side (i.e. red) of the mean (green) are more strongly refracted by a positive lens and hence, focus closer to the lens than the mean focus. On the other hand, wavelengths on the longer wavelength side of the mean (blue) focus long ( Figure 1). Hence, if green is be set as the one in focus, then a barrel distortion of the red channel and a pincushion distortion of the blue channel can be observed. The effect can be reduced using a combination of lenses, so that two colours focus on the mean. The effect cannot be eliminated as three colours cannot be focus on the same plane. In some cameras the effect can be reduced through the design of the lens. However, this is usually not the case in low cost sensors due to costs.

Oblique (lateral) chromatic aberrations
Lateral aberration is measured perpendicular to the optical axis and is caused because the ray of white light in object space arrives at the image plane as a spread of light ( Figure 2). Therefore, lateral aberrations are also called chromatic differences of magnification. As the illumination of an object point and its reflectivity may vary with wavelength, the mean position in the image plane the aberration magnitude also depends on the effective colour. Therefore, a calibration would also depend on the material of object points. Hence, when modelling lateral aberrations as radial lens distortion parameters per band (R, G, B), the assumption has to be made that the reflectivity of the object points is similar. While the secondary spectrum effect caused by lateral aberration can be eliminated using a combination of two lenses using design control, this is usually not done in low-cost camera systems.

Test setup
For this research three cameras of the type GoPro Hero 3+ (Black series) are used; the specifications of the camera are provided in Table 1. The cameras were used in their water proofed housings. Due to the waterproof housings, the light of rays were not just affected by the medium water and the lenses/air in the camera but also the plastic of the housing. However, as discussed in Shortis et al, (2009), implicit inclusion of the refraction effects requires no assumptions to be made concerning the refractive indices of the air, glass or water media, and modelling of the optical components of the underwater housing is unnecessary. The cameras used have the serial number #305E917, #3064F7C, and #3064F72 and are respectively referred to as "GB", "GD" and "GT". For the testing the cameras were mounted on a GoPro Extender Handle. The images were captured in a controlled environment, i.e. a water tank (3 m long, 2 m wide and 1.5 m in depth). The tank was large enough to submerge a calibration frame which was placed in the centre of the tank (Figure 3). The calibration frame is a steel frame with 52 targets placed in the frame. The GoPro cameras were set to time-lapse and submerged with the extender arm into the tank. The cameras were moved around the calibration frame whilst maintaining a constant distance. A convergence angle of 125 ± 10 degrees surrounding the frame could be achieved, and one camera position was tilted by 90 degrees per sequence. All settings of the cameras were kept fixed in all sequences. While multiple images of the calibration frame were taken for each calibration sequence, only 8 to 10 images were used in the processing, eliminating those with motion blur or poor image quality. Nevertheless, this was not always possible and a few images still contained motion blur in (Figure 4). However, for all images a comparable and strong geometry was achieved. Further details about the data capturing is available in Helmholz et al., 2016. For the processing five datasets were used. Out of those five, three were captured with the same camera (GB) in order to check the consistency within one sensor, and one dataset was captured for each of the other cameras (GD and GT) in order to check if the results are also visible in similar platforms.

Pre-Processing
While it is understood that the resolution of the images is influenced and correlated by the Bayer pattern used in the cameras, three colour band images (Red, Green and Blue) were extracted from the captured RGB images. When the images were extracted, no further image enhancement steps were applied. Hence, the number of datasets processed increased from 5 to 20 (five image sequence were processed using its RGB, R, G, and B images).
While the targets of the calibration frame are retroreflective in air, they are not in water as the retro-reflective properties are lost underwater. However, as the targets are white dots on a black frame they were easily detectable because of the high contrast. The targets on the frame were automatically detected using a centroiding fitting approach, and when required manually adjusted (also using a centroiding fitting). Initially the labelling of points was done manually in the first three images, and then using the drive back function. Both, the target detection and the referencing in all images (five datasets times four bands (RGB, R, G, B)) was done using the software iWitnessPro. The derived image observations were used in a least square adjustment. For this least square adjustment, the EO and IO values calculated by iWitnessPro were used as initial values for the adjustment.

Bundle Adjustment
The aim of the least-squares adjustment was to determine the camera parameters of the 20 datasets using the Brown's camera model. More specifically, we solved for the principal distance c, the principal point offset (xp, yp), radial lens distortion (k1-k3) and decentring distortion (p1-p2). Tests with four radial lens distortion parameters showed that k4 is not significant and, therefore, was not considered in the results.
A number of adjustments were performed, adding different constraints to the adjustment. Firstly, all datasets were processed independently from each other using a free network adjustment, i.e. a total of 20 adjustments were run. The image observations were weighted equally for x and y observations for the adjustment of the RGB composite images. For the adjustments using the R, G, and B images the y coordinate were up to 4x worse in terms of rmse. Therefore, an approximate variance component estimation was used to set the weights once outliers were removed. When talking about this adjustment we referred to it as Independent Adjustment.
Secondly, the same weights and camera models as for the R,G,B independent adjustment were used performing a combined adjustment per camera, i.e. constraining the points in object space to be the same (Lichti and Chapman, 1997). For this test we did not process the RGB images, but only the R, G and B images. A total of 5 adjustments were performed (i.e. for each image sequence one adjustment solving for all camera parameters in one adjustment). When talking about this adjustment we refer to it as Combined Adjustment.

Independent adjustment results
The analysis of the result will focus first on the principal point offsets and the principal distances followed by radial lens distortion parameters and the decentring distortion parameters.

Principal Point Offsets and Principle Distance
The estimated principal distance and the estimated principal point offset of the independent adjustment utilising the GB camera 1 st repeat only are shown in Table 2. The principal distance using the RGB composite is the shortest, followed by G, then R and finally B. However, none of the differences are significant. It is interesting to see that the standard deviations of the B band for all three parameters (principle distance and principal point offset) are always the largest. Similar (even if sometimes not as strong) results can be also observed for the other datasets.  Table 2. Calibration parameters (c, xp, yp) for the individual calibrations using the GB camera (1st repeat) independent adjustment. All units in mm. The largest standard deviation value for each dataset is in boldface text.
In order to assess possible longitudinal chromatic aberrations Table 3 shows the principal distance and their standard deviations for all 20 independent adjustments. The repeats for the GB camera (1 st , 2 nd and 3 rd repeat) show relatively stable results; the magnitude of c and its standard deviation are not significant different. Looking at the results of one camera and the results between the RGB composite and the different bands, it is also true that the magnitude of c and its standard deviation are not significantly different. However, a small trend (even if not significant) shows that blue usually has the longest principal distance, while the R and G principal distances are very similar. While it is possible that two wavelengths show similar longitudinal chromatic aberration (see background), the blue light should produce the smallest and not the largest corrections. However, as already pointed out, the differences are not significant, and the trend could be caused by the larger standard deviations in the results of the blue band images. Finally, the principal distance of the RGB composite is usually close to the R and G value for each dataset.  Table 3. Comparison of the principle distance for the individual calibrations of all assessed cameras (1st repeat). All units in mm. The largest values and standard deviations are highlighted.

Radial Distortion
In order to assess any lateral chromatic aberration, the radial lens distortion parameters are analysed. All radial distortion parameters were confirmed to be significant in the adjustment. The analysis is reduced to a smaller area as the calibration field did not cover the whole image format. The maximum radial distance of 3.875 mm was reduced to 1.9 mm. Table 4 shows the correction values in m for the radial distance of 1.9 mm.
While the distortions of the repeats with the GB camera are similar in magnitude, they are significantly different from the other two camera models. As, the differences between the cameras are large, each sensor has to be treated independently.
Furthermore, there are clear differences visible between the bands in each dataset. Firstly, the RGB composite values are again close to the R value. Secondly, there is the clear trend that R has the largest corrections, followed by G and then by B. This disagrees with the explanations provided in the background section, and cannot be explained yet. While some wavelengths have higher absorption in water (e.g. red light) than others (e.g. blue lights), the wavelengths do not change, and therefore the lateral chromatic aberration behaviour should not change, too.  Table 4. Comparison of radial distortion correction in 1.9 mm radial distance. All units in m. The largest correction value for each dataset is highlighted.
For better visualisation, the profile of the GB camera (1 st repeat) is presented in Figure 5. All other profiles look similar. The figure highlights that the RGB composite radial distortions are similar to the R and G distortions, and that B shows the largest distortions. Figure 5. Radial distortion profile of camera GB camera 1st repeat (RGBblack, Rred, Bblue, G -green bands) for the individual adjustment. X axis in mm, y axis in m.

Decentring Distortion
To further analyse any lateral chromatic aberrations, the decentring distortions (also up to 1.9 mm radial distance) are presented in Table 5, and the profile of the GB camera (1 st repeat) is shown in Figure 6. All decentring distortion parameters were confirmed to be significant in the adjustment.   Table 5. Comparison of the decentring distortion correction in 1.9 mm radial distance, independent adjustment. All units in m. The largest correction value for each dataset is highlighted.
As to be expected, the magnitudes of the decentring distortion corrections are much smaller compared to the radial lens distortion corrections. Similar to the radial lens distortions, the decentring distortions of different cameras are significantly different. While the GD camera had the smallest radial distortion correction, it has the largest decentring distortion correction. The GT camera has relatively high radial and also decentring distortion corrections. Overall, there is no clear trend which band produces the highest corrections. Figure 6. Decentring distortion profile of camera GB camera 1st repeat (RGB and Rblack, Bblue, G -green bands) for the individual adjustment. X axis in mm, y axis in m.

Combined adjustment results
Again, the principal point offsets and the principal distances will first be analysed followed by radial lens distortion parameters and lastly the decentring distortion parameters.

Principal Point Offsets and Principle Distance
When analysing the principle distance and principal point offsets parameters with their standard deviations of the GB camera (1 st repeat) of the combined adjustment (Table 6), the first observation is that the standard deviations of all parameters are similar; the previously largest standard deviations for the blue band is not present in the data anymore. Similar results also can be observed for the other cameras.  Table 6. Calibration parameters (c, xp, yp) for the individual calibrations using the GB camera (1st repeat) combined adjustment. All units in mm. Table 7 shows the principal distance and its standard deviation of the different test datasets. Firstly, the repeats using the GB camera showed relative stable results with 3 rd repeat being a bit "off". However, the differences are not significant. Furthermore, the standard deviations compared to the independent adjustments are smaller, and the standard deviations for each dataset for the different bands are basically the same. There is no significant difference visible between the bands, again only a small trend (even if not significant) that blue has usually the longest principle distance. This agrees with the results from the independent adjustment, but again disagrees with the explanations provided in the background section. However, as the differences are not significant, a final conclusion cannot be drawn.

Radial Distortion
The analysis of the radial distortion corrections was again reduced to a smaller area of 1.9 mm radial distance. The results are presented in Table 8.
Again, differences between the bands are clearly visible for all tests. While the magnitude of the correction is comparable to the independent adjustment, the additional constraint led to more comparable results between the repeats using the GB camera system. Again, there is a clear trend that R has the largest magnitude, followed by G and then B which is the opposite of what is expected following the explanations in the background section.
[  Table 8. Comparison of radial distortion correction in 1.9 mm radial distance. All units in m. Figure 7 shows the profile of the same camera (GB 1 st repeat) as presented in Figure 5 for the independent adjustment. As discussed, the profiles are similar. Figure 7. Radial distortion profile of camera GD (RGBblack, Rred, Bblue, G -green bands) for the combined adjustment. X axis in mm, y axis in m.

Decentring Distortion
The corrections for the decentring distortion are presented in Table 9. Compared to the independent adjustment, there is a clear difference between the B and the R/G band visible. In this test, it also shows clearly that B has always the largest correction values. While this trend disagrees with the radial lens distortion parameters, it agrees with the information provided in the background section. Figure 8 makes Table 9. Comparison of the decentring distortion correction in 1.9 mm radial distance, combined adjustment. All units in m.
The largest correction value for each dataset is highlighted. Figure 8. Decentring distortion profile of camera GB camera 1st repeat (Rred, Bblue, G -green bands) for the combined adjustment. X axis in mm, y axis in m.

CONCLUSION AND OUTLOOK
We were able to detect significant corrections for the radial and decentring distortion parameters indicating lateral chromatic aberration. However, while the literature specifies that the largest correction values for lateral chromatic aberration should be present in the blue band followed by green and red, we could observe the largest corrections for red followed by green and blue. This trend is unclear and will have to be confirmed with further tests. Furthermore, a significant trend of different principal distance indication longitudinal chromatic aberration was not able to be observed.
For the next tests, a more suitable calibration frame covering the whole field of view of the camera will be used. A larger calibration frame also would allow to analyse the correlations between the different groups of calibration parameters and the estimated EO parameters. In addition, the bands have been extracted out of an image which was generated by the Bayer filter. However, due to the Bayer filter, the different bands are highly correlated with each other. Future work will analyse the impact of those correlations to the derived chromatic aberration. In addition, it is aimed to compare the results produce with the same camera for underwater and in-air environments.
Finally, we are going to add further constraints to the adjustment, more specifically we will add constraints to the exterior orientation parameters assuming that the location of the camera is the same for all image bands.
The different behaviours of different camera models should be also investigated, in order to produce a guideline supporting underwater photogrammetric applications.