INVESTIGATION ON MULTI-SENSOR FUSION STRATEGIES FOR IMPROVED ORIENTATION DETERMINATION IN MOBILE PHONE IMAGING APPLICATIONS

Thanks to the rapid technological progress in the field of mobile devices, smartphones are increasingly becoming valuable for science. They can serve as photogrammetric measurement devices with built-in cameras, micro-electro-mechanical systems for orientationand position assessment, as well as powerful processing units allowing field-based data acquisition and processing. This paper outlines a comprehensive investigation focusing on the accuracy and stability of smartphone camera rotation parameters determined by builtin smartphone sensors. For that purpose, the rotation parameters were measured under a range of different conditions. Four test scenarios were defined considering indoorand outdoor measurements using three different devices being in static and dynamic modes. Furthermore, the influence of magnetic perturbations was investigated. The rotation parameters were determined from the measurements applying different sensor fusion approaches. Reference values for accuracy assessment were provided by a superior precision inertial measurement unit that measured the rotation parameters simultaneously to the smartphone in each experiment. The analysis of the smartphone-based rotation parameters, separated in the Euler angles azimuth, pitch and roll, shows average accuracies below 2° for pitch and roll. In comparison, azimuth shows significantly lower accuracies of more than 30° especially when the smartphone is in motion and when it is exposed to magnetic perturbations. In this regard, advanced multi-sensor fusion approaches were examined that handle such interferences to considerably improve the accuracy of azimuth measurements. In conclusion, a summary of accuracies and stabilities to be expected from smartphone sensors is given referring to ambient conditions and investigated sensor fusion strategies.


INTRODUCTION
Smartphones are ubiquitously available in modern society. They are not only pure communication tools, but also equipped with cameras and a variety of sensors to measure the device position and orientation. Thus, smartphones are increasingly used as versatile measurement devices, which has been demonstrated in different studies. For instance, Bruinink et al. (2015) and Elias et al. (2019) developed photogrammetric smartphone applications for water level observation and flash flood prediction. Novakova and Pavlis (2017), Kröhnert et al. (2017) and Kehl et al. (2019) use smartphones for field-based interpretations of geological features. Bianchi et al. (2017) have shown that smartphones are suitable for the determination of forest inventory parameters. Muratov et al. (2016) introduced a method to perform 3D reconstruction based on structure from motion (SfM) on smartphones. Each of these applications requires precise information about smartphone camera's position-and rotation parameters, i.e. the exterior orientation. Referring to this, stateof-the-art global navigation satellite system (GNSS) receivers, implemented in smartphones, promise accuracies of less than 1 m for the position parameters using real-time precise point positioning (PPP) strategies (Chen et al., 2019a). Despite numerous technical innovations, the smartphone-based determination of absolute rotation parameters using built-in micro-electro-mechanical system (MEMS) inertial measurement units (IMUs) is still challenging due to sensor instabilities and external magnetic field interferences, e.g. Poulose et al. (2019a), and requires a comprehensive investigation. * Corresponding author For that purpose, typical problems in rotation parameter determination must be simulated in relation to the abovementioned applications, e.g. when the device is exposed to magnetic perturbations during field-based measurements. Elias et al. (2019) address the challenge of smartphone-based rotation parameters in their description of a smartphone application enabling the determination of water levels in flood events. The presented approach is based on image-to-geometry registration and needs good approximations for smartphone camera's exterior orientation. They point out that the success of water level determination highly depends on the rotation parameters and may be impeded when the determined azimuth (also known as heading or compass angle) deviates by more than 40° from the actual value, which is quite possible according to the work of Blum et al. (2013). They determined deviations of about 30° regarding azimuth with significant drifts after a few minutes in an augmented reality (AR) application where the smartphone is used in a static mode. Similar observations were made by Kok et al. (2017). Novakova and Pavlis (2017) even determined azimuth-related errors up to 90° using the smartphone as a compass in geological applications.

Related Work
It is obvious that the precise determination of rotation parameters using built-in smartphone sensors is not trivial and a subject in manifold research. However, most studies are focussed on the measurement of relative rotation parameters, necessary for e.g. indoor navigation, using the output directly from physical sensors or from virtual sensors that implement standard sensor fusion approaches, e.g. Ma et al. (2013), Aicardi et al. (2014), Gikas and Perakis (2016) and Kuznetsov et al. (2018). This paper affiliates these investigations with a particular focus on the abovementioned issues regarding photogrammetric applications. For that purpose, the rotation parameters were determined in relation to a world reference frame under a range of different conditions utilising three Android devices, equipped with low-cost IMUs based on MEMS technology, and implementing three inertial multi-sensor fusion approaches. Of course, inertial sensors can also be coupled with GNSS, e.g. Hide et al. (2009) and Sheta et al. (2018), cellular signals or visual sensors, e.g. Angelino et al. (2012), Solin et al. (2018) and Qin et al. (2018), to improve the rotation measurements, but only provided that the device is in motion. In view of the mentioned applications predominantly requiring the smartphone in static mode, however, this paper is focused on the accuracy and stability assessment of rotation parameters (f)using only inertial sensors. The accuracy was determined comparing the rotation parameters from smartphones to those from a superior precision IMU and the stability was determined observing the variability of the rotation parameters when the investigated smartphone was used in a static mode.
Smartphone camera, MEMSs and GNSS module are closely mounted on a single circuit board. With respect to the expected reduced accuracies in comparison to high-end sensor systems used in, for example, airborne laser scanning applications, the determination of the boresight alignment was neglected in the following investigations.

Paper structure
Before starting with the measurements, Section 2 provides basic knowledge about physical and virtual sensors that are commonly used to determine rotation parameters. Furthermore, the relation between the smartphone sensor coordinate system and a world coordinate system is described. As the paper investigates the impact of magnetic perturbations on the measurement of rotation parameters, a short outline is provided dealing with magnetic effects and how to calibrate them. Section 3 provides information about the conducted experiments including a description of the test scenarios, of the applied hardware and sensor fusion techniques and of the experimental setup. Furthermore, the synchronisation of smartphone and IMU is described. The measurement results are given in Section 4 separated in investigations on accuracy and on stability. Finally, a short summary is provided in Section 5.

Sensors systems for orientation estimation
As the term sensor fusion indicates, the measured values from multiple physical sensors are combined to generate advanced virtual sensors. Such sensor fusion concepts are commonly implemented by state-of-the-art IMUs and increasingly by smartphones with the purpose of rotation parameter determination.

Physical Sensors are MEMSs built in smartphones that
can be classified into three groups (Google, 2020):


Environmental sensors, e.g. barometer 1 natural device orientation of smartphones and phablets: portrait mode; tablets: landscape mode (valid for most devices)  Position sensors, e.g. magnetometer (+ GNSS)  Motion sensors, e.g. gyroscope and accelerometer By default, the sensor data of magnetometer, gyroscope and accelerometer is used to determine the rotation parameters. In this regard, the accelerometer measures linear accelerations and gravities along three axes, the magnetometer measures the ambient magnetic field and the gyroscope measures angular velocities and accelerations in 3D space.
2.1.2 Virtual sensors are a result of fusing the data from physical sensors. They show significant advantages over the exclusive usage of single physical sensors. One the one hand, integrating position sensors in addition to motion sensors allows to determine the absolute device orientation in Earth-North-Up reference frame. On the other hand, sensors with complementary strengths and weaknesses can be combined to overcome sensorspecific issues. As an example, accelerometers measure inertial forces at the device, but they are susceptible to smallest vibrations and non-gravity accelerations that result in noisy angular measurements. Gyroscopes accurately measure the angular acceleration of a device, but they are prone to drift effects due to bias instability and high-frequency noise called angular random walk (ARW). Fusing the measurements from both sensors helps to reduce noise occurring from non-gravity forces and drift effects. More details are given in Beavers (2017)

Smartphone coordinate systems
The orthogonal smartphone sensor coordinate system is defined in relation to the natural orientation of the device 1 where  x points to the right edge of the device  y points to the top edge of the device  z points out of the display The world coordinate system is expressed as orthogonal coordinate system where  X (Y×Z) is tangential to the ground pointing to East  Y is tangential to the ground pointing to magnetic North  Z is perpendicular to the ground pointing in direction of the sky Smartphone-based rotation parameters will be zero when the smartphone is flat on the ground with the screen facing the sky and with the top edge pointing to magnetic North (see Figure 1). More details about the different coordinate systems are given by Google (2020). Note: Using the device upright, e.g. in AR-and photogrammetric applications, requires remapping of the sensor coordinate system in order to align camera's principal axis with the Y-axis.

Magnetic perturbations
Measuring the azimuth in relation to the world reference system requires measurements of Earth's magnetic field that can be described by three components. Declination and inclination describe Earth's magnetic field orientation where the declination provides information about the deviation from magnetic North to true North. 3D magnetometers are able to measure the third component, the magnetic field intensity, which is used to determine the device direction in relation to magnetic North. Unfortunately, magnetometers are prone to errors occurring from inferences with the geomagnetic field, called magnetic perturbations. These can be classified in soft-and hard iron effects. Details are given in Caruso (2000), Seco and Jiménez (2018) and Muraccini et al. (2019).

Hard iron effects
arise from fixed-installed magnetised objects close to the sensor system, which applies to almost all components built in smartphones. These error sources will cause a constant bias to the sensor measurements resulting in a 3D offset of the measured magnetic field intensities that need to be calibrated.

Soft iron effects occur from flexible magnetised objects
close to the sensor system that can distort and deflect Earth's magnetic field measurements. This results in bias towards the sources of interference. Soft iron effects can occur indoor and outdoor, for example when the device is exposed to changes of the physical environment.

Magnetometer calibration
implies the correction of hard-and soft iron effects including, on the one hand, the determination of the offset referring to hard iron effects and, on the other hand, the establishment of the complex relationship between the magnetic field and soft-iron-induced distortions (Vishwatheja et al., 2016). Magnetometer calibration is mandatory and must be performed immediately before the actual measurement. However, soft iron effects are highly variable and will thus influence the measurements even if the sensor has been calibrated in advance. For that reason, they were considered in the experiments to quantify their impact on the determined rotation parameters with a special focus on the measurements of the compass angle azimuth.

EXPERIMENTS
Three sensor systems, built in three different Android devices, were used to investigate the accuracy and the stability of smartphone rotation parameters determined from three virtual sensors. For that purpose, four indoor and outdoor test scenarios were defined, where the device was in: 1. static mode 2. static mode being exposed to soft iron effects 3. dynamic mode 4. dynamic mode being exposed to soft iron effects

Hardware
Details on the investigated devices are given in Table 1. Each investigated smartphone has a 6-axis MEMS gyroscope and accelerometer (see Table 1, Inertial sensor system) and a 3-axis electronic compass (see Table 1, Magnetometer). The reference values, given in Euler angles, were determined using the IMU Spatial v6.1 by Advanced Navigation. It comes with a virtual sensor that fuses the measurements from built-in 3D accelerometer, gyroscope and magnetometer. Table 2 shows IMU's specifications, provided by Advanced Navigation (2020).
LG  Table 2. Manufacturer's information to IMU Spatial.

Experimental setup
The experiments were carried out with a smartphone and the IMU being fixed on a stable bar at a distance of 1 m to ensure that both sensor systems do not disturb each other (see Figure 2).

Figure 2. Experimental setup.
Smartphone and IMU were aligned in such a way that the sensor coordinate systems of both devices were parallel. Note, Spatial measures the pitch as rotation around the y-axis whereas the smartphone measures the pitch as rotation around the -x-axis, which resulted in opposite directions of rotations that must be considered in the data analysis.
Three repeated measurements were performed with measuring times of about 90 seconds. At the beginning of each experiment, both sensor systems were calibrated with regard to the magnetic field following the explanations given in Google (2020) and Advanced Navigation (2020) while information about the calibration validity was given by the operating system. After that, the bar, being parallel to the ground, was roughly aligned to true North. Regarding experiments where the device was used in a dynamic mode, the bar was moved in a slight rowing motion. Soft-iron effects were induced by means of magnetised objects that were placed close to the smartphone with varying positions.

Smartphone implementation:
The experiments were conducted using a self-developed smartphone application that implements the virtual sensors:  Android Rotation Vector (ARV)  Improved Rotation Sensor 1 (IRS1)  Improved Rotation Sensor 2 (IRS2) to determine the rotation parameters to be compared with those from the reference IMU. The default-implemented ARV measures the rotation parameters in relation to a world coordinate system (see Figure 1) fusing the measurements from accelerometer, gyroscope and magnetometer by means of an EKF. In this regard, the gyroscope gives the orientation, the accelerometer provides the correction measuring the gravity forces at the device and the magnetometer provides the correction to magnetic North (Pacha, 2015). Unfortunately, ARV shows residual drift effects that seem to be related with the gyroscope, as ARV uses the raw gyroscope measurements directly from the physical sensor. The multi-sensor fusion approaches IRS1 and IRS2, developed by Pacha (2015), promise solutions for this issue fusing ARV with the Android-implemented virtual sensor Calibrated Gyroscope. According to the developer, IRS1 combines the angular acceleration measurements of both sensors and is recommended for most AR applications when the device is used in a static mode. In contrast to this, IRS2 fuses the angular accelerations as well as the angular velocities measured by the Calibrated Gyroscope and is recommended for applications where the device is used in a dynamic mode. The implementation of both virtual sensors, IRS1 and IRS2, used here, is based on the source code provided by Pacha (2015).
In the experiments, the virtual smartphone sensors ran with a frequency of 25 Hz that allowed for motion tracking when the device was moved with moderate speed and provided reasonable numbers of about 2250 measurements in a 90-second measurement period.

Observing magnetic perturbations:
Magnetic perturbations can be detected comparing the measured magnetic field intensities with expectation values given by the World Magnetic Model 2015 (WWM-2015) that provides superior information about the Earth's magnetic field depending on the device location and current time. Further details can be found in Chulliat et al. (2015). In this study, the measured intensity must equal the expected intensity with a tolerance of ±5 μT, otherwise it was assumed to be affected by uncalibrated soft iron effects. As the smartphone application informs the user about the ambient magnetic field intensity, deviations could be provoked to simulate soft iron effects that ranged between 10 μT and 15 μT in the respective experiments. Stronger magnetic perturbations were avoided as they are likely to heavily impair the magnetometer so that this could not work properly.

Data logging and synchronisation:
Regarding data logging, the smartphone application recorded the measurements including the time stamp in milliseconds given by Unix time, the rotation parameters azimuth (to true North), pitch and roll measured by three virtual sensors and the deviation of the measured magnetic field intensity referring to magnetic perturbations. Similar to this, the IMU recorded azimuth (to true North), pitch and roll together with a Unix time stamp at a frequency of 40 Hz.
Even if both datasets can be synchronised by Unix time, the sensor systems showed different latencies that must be corrected to make them comparable. Considering the experiments, the lag was determined and fixed for each investigated device by means of sample cross correlation on the trajectories of azimuth, pitch and roll considering the dynamic-mode measurements. Then, it was applied to the corresponding measurements when the device was used in a static mode.

RESULTS
In summary, 216 measurement series were conducted whereby one data set contains the rotation parameters from three virtual sensors (including four test scenarios performed indoor and outdoor applying three devices and three repeated measurements per setting). The accuracies were determined for each setting and for each device calculating the root mean square errors (RMSEs) between the synchronised smartphone-and IMU measurements for azimuth, pitch and roll, respectively. In relation to this, the RMSEs were individually determined for the measurements of ARV, IRS1 and IRS2. The sensor stability was assessed by means of the standard deviations that were calculated for the smartphone-based rotation parameters involving each measurement series where the device was used in a static mode.
The results are visualised in Figure 3-5 with regard to accuracy and stability, respectively, taking account of the measurement settings, the used devices and the individual virtual sensors.

Accuracies
As illustrated in Figure 3, devices used in dynamic mode show higher deviations to the ground truth in all rotation parameters compared to the measurements performed with devices used in static mode. Calculating the median values of the RMSE values of a measurement series for azimuth, pitch and roll results in 24.0°, 0.9° and 0.6° when the device was used in a static mode. In contrast, median deviations of 41.9°, 2.1° and 2.8° occurred when the device was used in a dynamic mode. This is particularly evident for azimuth when the device is exposed to magnetic perturbations. In this regard, the medians show deviations of 45.6°, 1.6° and 1.4° for azimuth, pitch and roll irrespective of whether they were captured in dynamic-or in static mode or whether the experiment was performed indoors or outdoors. On the contrary, devices that were not affected by magnetic perturbations show deviations of only 22.0°, 1.7° and 1.4°. Focussing on pitch and roll, the impact of magnetism is negligible as it results in a loss of accuracy of less than 1.0°. Comparing the results from indoor-and outdoor measurements, the azimuth angles measured outdoor are significantly more accurate as when they were measured indoor.
One reason for this may be stronger interferences occurring inside a building due lamps, radiators, pipes etc. Calculating the median deviations results in 51.2°, 1.7° and 1.4° as well as 23.7°, 1.6° and 1.7° for azimuth, pitch and roll with respect to indoor and outdoor measurements. Despite the rather small deviations for pitch and roll that were measured in both environments, azimuth shows maximum deviations up to 50.8° when it was measured outdoors and even up to 90.9° when it was measured indoors.
Comparing the results of IRS1, IRS2 and ARV, no significant differences became apparent between the respective measurements of pitch and roll. However, significantly better results could be achieved for azimuth using IRS2, especially in case of magnetic perturbations.   Focussing on Figure 5, 25% of all IRS2 measurements show deviations of less than 15° from the ground truth. In contrast, the percentage of measurements differing less than 15° amounts to only 18% for IRS1-and 13% for ARV measurements. Beside this, 50% of all IRS2 measurements show deviations of less than 30°.

Stability
The sensor stability was investigated when the devices were used in static mode. As visualised in Figure 4, ARV shows slight, negligible variations in all measurements. In contrast, IRS1 and IRS2 show variations of 0.6° and 1.8° for azimuth measured by LG Google Nexus 5 when the device was exposed to magnetic perturbations. These effects may be related with device's sensor type that might be more susceptible to interferences than the other sensor types. However, stabilities similar to the results of pitch and roll could also be observed for azimuth considering the measurements from Samsung Galaxy S8 and Samsung Galaxy S10+.

CONCLUSION
This paper addresses the complex issue of the determination of absolute rotation parameters using built-in smartphone sensors.
For that purpose, their accuracy potential and stability were investigated from different perspectives considering indoor-and outdoor measurements, magnetic perturbations, smartphones running in static and dynamic modes, as well as different (multi-) sensor fusion concepts. In each experiment, the rotation parameters were simultaneously measured from a superior precision IMU whose measured rotation parameters served as ground truth.
The experiments have shown that the rotation angles pitch and roll can be measured with accuracies of a few degrees under different conditions using built-in smartphone sensors. In contrast, the accuracy of the azimuth varies between 1° and 90° whereby the highest errors occurred when the smartphones were used in dynamic mode or when they were exposed to magnetic perturbations. However, it was detected that the measurements of the azimuth angle could be significantly improved applying multi-sensor fusion approaches like IRS2 that provide errors of less than 30° in 50% of all measurements.
It is obvious that these accuracies, especially the accuracies determined for azimuth, impede the direct georeferencing of smartphone images in photogrammetric applications. Poulose et al. (2019b) have shown that implementing sensor fusion by means of an Unscented Kalman Filter (UKF) could further improve the direct measurement of the azimuth, which, however, must be evaluated in future investigations. Nevertheless, using these values as rough estimates enables the application of imageto-geometry registration to precisely determine the azimuth with the aid of landmarks within spatial resection, e.g. Schwalbe and Maas (2017), Eltner et al. (2018), Kehl et al. (2019) and Elias et al. (2019).
New advanced approaches integrate deep learning in sensor fusion and promise significantly more accurate measurements of the rotation parameters combining also (low-cost) inertial sensors, provided that the smartphone is used in dynamic mode, e.g. Silva do Monte Lima et al. (2019). In this regard, Chen et al. (2019b) have shown how to use deep learning to improve the robustness of visual-inertial odometry to deal with imperfect sensor data, i.e. feature tracking errors and noisy IMU data. Shamwell et al. (2019) even trained a neuronal network with RGB-depth imagery and IMU data to enable the determination of absolute camera trajectories merely using consecutively captured camera images. Such approaches could be very helpful to improve photogrammetric applications like 3DCapture (Muratov et al., 2016), where precise information about the camera trajectory is mandatory to perform 3D reconstruction directly on the smartphone.