APPLICABILITY OF NEW APPROACHES OF SENSOR ORIENTATION TO MICRO AERIAL VEHICLES

This study highlights the benefits of precise aerial position and attitude control in the context of mapping with Micro Aerial Vehicles (MAVs). Accurate mapping with MAVs is gaining importance in applications such as corridor mapping, road and pipeline inspections or mapping of large areas with homogeneous surface structure, e.g. forests or agricultural fields. There, accurate aerial control plays a major role in successful terrain reconstruction and artifact-free ortophoto generation. The presented experiments focus on new approaches of aerial control. We confirm practically that the relative aerial position and attitude control can improve accuracy in difficult mapping scenarios. Indeed, the relative orientation method represents an attractive alternative in the context of MAVs for two reasons. First, the procedure is somewhat simplified, e.g. the angular misalignment, so called boresight, between the camera and the inertial measurement unit (IMU) does not have to be determined and, second, the effect of possible systematic errors in satellite positioning (e.g. due to multipath and/or incorrect recovery of differential carrier-phase ambiguities) is mitigated. First, we present a typical mapping project over an agricultural field and second, we perform a corridor road mapping. We evaluate the proposed methods in scenarios with and without automated image observations. We investigate a recently proposed concept where adjustment is performed using image observations limited to ground control and check points, so called fast aerial triangulation (Fast AT). In this context we show that accurate aerial control (absolute or relative) together with a few image observations can deliver accurate results comparable to classical aerial triangulation with thousands of image measurements. This procedure in turns reduces the demands on processing time and the requirements on the existence of surface texture. Finally, we compare the above mentioned procedures with direct sensor orientation (DiSO) to show its potential for rapid mapping.


INTRODUCTION
Unmanned aerial vehicles (UAVs) have become an important tool for surveyors, constructions engineers and scientists worldwide.Thanks to their affordability and recent advances in guidance, autonomy and easiness of use, they spread among wide public.The number of available system is increasing rapidly (Colomina and Molina, 2014).This progress is accelerated by accompanied software bundled with the platforms that makes image processing as easy as never before.Despite this progress, indirect sensor orientation is still the most common way of sensor orientation (SO) in spite of a gradual rise up of commercial platforms with embedded systems offering at least accurate aerial position control (Mavinci, 2015, senseFly, 2015).

Concepts of sensor orientation
The benefits of aerial control for larger platforms have been extensively studied in past (Schwarz et al., 1993, Skaloud et al., 1996).For a comprehensive summary and evolution of SO approaches see for instance (Colomina, 1999, Colomina, 2007) or (Skaloud, 2006, Legat et al., 2006) for the challenges in direct sensor orientation.The effort of introduction of absolute position and attitude aerial control on MAVs is relatively recent, see for instance (Eling et al., 2014, Rehak et al., 2014, Mian et al., 2015) for block operations or (Rehak and Skaloud, 2015) for corridor mapping.Such approaches allow to significantly reduce, or even completely eliminate the requirement on ground control points.That brings not only significant savings in the mapping operations, but also extends their applicability over inaccessible areas or regions with poor image texture.Each orientation method represents a certain trade-off between operational efficiency and resulting accuracy.The most important aspect is the control of the resulting quality that is related to geometrical redundancy and mitigation or detection of systematic effects.In this respect, relative aerial position and attitude observations between successive images allow addressing some issues in aerial control while reducing the effect of others.What is even more interesting, is the fact that in case of integrated sensor orientation (ISO), the replacement of absolute position and/or attitude observations with the relative ones, leads to similar mapping accuracy.This was demonstrated by (Blazquez and Colomina, 2012b) in case of mapping with precise GNSS/IMU on a conventional aircraft and by (Skaloud and Lichti, 2014) for mapping from a MAV.The latter study focused, however, only on the relative position control in a block-scenario with a vertical take-off and landing platform.Here, we study the benefits of absolute versus relative aerial control in a block as well as in a corridor using a fixedwing MAV equipped with a Micro Electro Mechanical System (MEMS) IMU and a multiple freq.GNSS receiver.We investigate such orientation schemes also with respect to direct sensor orientation as well as to an intermediate scenario: quasi direct orientation (Blazquez and Colomina, 2012a).So called Fast AT is bundle block adjustment (BBA) with either absolute or relative aerial control and a very few image observations.Indeed, the latter are reduced to ground control (GCP) and check points (ChP).As it will be shown, Fast AT is a very relevant concept for MAVs in situations where the terrain texture is limited and/or the automatically generated tie points are of poor and inconsistent quality, or their distribution is not regular.Moreover, it is also substantially faster than ISO since the automatic tie point detection can be skipped.Table 1 overviews the most pertinent methods of sensor orientation used in photogrammetry ordered from none to the most stringent requirements in terms of the availability of aerial control observations.Second possibility is to employ additional GNSS receiver(s) while maximizing the separation of antennas, (Eling et al., 2014).In most cases such separation is below 1 m level.Hence, the resulting quality of azimuth initialization is likely comparable with that of a calibrated magnetometer.Another possibility is to reduce the noise level of an integrated system by employing several IMUs in parallel (Waegli et al., 2010).Apart noise mitigation, this allows determination of the actual noise level that may vary in time (due to vibrations) as well as the possibility of detecting faults (Guerrier et al., 2012), which increases the redundancy of the system.Although an IMU of such type is employed in this study, the presented content uses data separately from its sub-system, i.e. one IMU.Last but not least, low-noise of carrier-phase in the satellite signal is a prerequisite for cm-level differential positioning.

Paper structure
In the following part we present the aerial absolute and relative observations models together with their stochastic models.The third section concentrates on performance analysis in real mapping scenarios.The MAV platform and its sensor payload are presented.The system calibration is then followed by the test data characterization.This section describes in detail the challenges in processing due to the weak geometry and low texture quality of the testing site.The fourth part is devoted to the practical evaluation during which we present the results of a variety of SO scenarios.Finally, the last part draws conclusions from the conducted research work.

Absolute position and attitude control
The observation equation that models the relation between the imaging sensor and the IMU body frame, for which absolute po-sition is derived, takes the form: the superscript m denotes a Cartesian mapping frame and the subscript c describes the camera sensor frame, X m is the GNSS/INS (Inertial Navigation System)-derived position for one epoch.X m 0 is the camera projection centre, R m c is the nine-elements rotation matrix from c to m frame, A c is the camera-GNSS antenna leverarm and S m is the possible positioning bias in GNSS-derived positions.Note, that time is a parameter for all components in Eq. 1 with the exception of A c .The absolute attitude observations can be expressed by following equation: where R m b is the GNSS/INS-derived attitude and R b c is the IMU -camera angular misalignment called boresight.

Relative position and attitude control
Relative observations relate the position and attitude parameters of two consecutive epochs (Li andStueckmann-Petring, 1992, Blazquez andColomina, 2012a).Differencing two sensor positions results in an observation equation for coordinate differences: Where ti and tj distinguish the two epochs.In comparison to Eq. 1, the term S m is canceled for certain (tj − ti) < ∆t.
The application of differencing makes use of the fact that certain effects of neighboring GNSS positions within strips or blocks are of a systematic nature.The attitude relative observations can be expressed as follows: Note, that the boresight parameter vanished in Eq. 4 compared to Eq. 2.

Stochastic models of relative aerial control
Proper observation weighting is very important in bundle block adjustment.The variances on absolute position and attitude observations are usually derived from the corresponding diagonal elements of a covariance matrix from a Kalman filter/smoother.Despite IMU error modeling and IMU calibration, the GNSS/INSderived observations remain time-correlated.Absolute positions might be affected by wrongly estimated ambiguities or multipath effects while IMU observations suffer from remaining unmodeled systematic errors as well as from residual effects due to initialization.The typical approach for handling these drawbacks is to introduce additional shift parameters.In the case of relative position control, the relation of variance propagation can be used if we assume a constant GNSS shift/bias within a strip or during a flight, depending on its duration.In Eq. 5, the parameter σ X(t ij ) represents the standard deviation of a relative position calculated according to Eq. 3 from two consecutive epochs σ X(t i ) and σ X(t j ) . (5) In the case of aerial attitude control, its stochastic models are somewhat more complicated.Given the quality and error characteristics of MEMS based IMUs, the on-line calibration process does not completely eliminate all systematic errors due to observability issues.This may lead to unrealistic covariance matrices for the attitude estimates and incorrect stochastic modeling in BBA (Martinez et al., 2007).However, over a short time interval within a flight time (i.e.limited acceleration and orientation changes), the accuracy of relative attitude aerial observations can be predicted by applying stochastic models for gyroscopes.In many cases this can be approximated as a superposition of a random walk (i.e.integrated white noise) ωRW and uncalibrated gyro drift ω b .
From our practical experience, the standard deviations of a relative kappa angle shall differ from those of omega and phi.It is usually sufficient to multiply ω b by a constant k = 1.5.Again, the approximation of Eq. 6 holds only within a short time ∆t between two consecutive images within the same flight line.In our evaluation, the maximal ∆t was set to 10 seconds.This constraint eliminates the observations between the separate flight lines of the trajectory.

Implementation
The relative observation models were implemented into a custom BBA software.The latter was developed as the offer of commercially available software for BBA allowing relative position and attitude observations is very limited.In addition, custom implementation allows having full control of observation stochastic modeling that is particularly very important when using accurate aerial control.The nonlinear triangulation problem is reduced to an optimization problem which is solved using an iterative Gauss-Newton process.The solution of all observation equations for one set of values for unknown parameters is obtained by solving normal equations: Where, dx is a vector of corrections to unknowns parameters, A is a model matrix containing partial derivatives of unknowns, P is a weight matrix and l is a vector of observations, for further details see (Triggs et al., 2000).

PERFORMANCE ANALYSIS
In the following paragraphs, the properties and performance of the MAV sensor system are presented and evaluated.The processing chain is introduced with focus on testing different sensor orientation scenarios on two data sets.Particularly, we focus our effort on evaluating the contribution of relative aerial control and on finding the optimal setup and geometry configuration.The performance of the different sensor orientation approaches is analyzed through empirical accuracy of ground point determination.The mapping accuracy is measured at independent check points by Root-Mean-Square (RMS) indicator of the coordinate differences.

MAV platform and sensors
The drone used for this study is a custom made airplane equipped with the open-source autopilot Pixhawk (Meier et al., 2012).The maximal payload capacity is around 800 g.The operational weight varies between 2200-2800 g and flying endurance is around 45 minutes.Thanks to its lightweight construction, the launching can be done from hand and it requires only a small place for landing.The payload comprises the 16 Mpx Sony NEX 5R camera, geodetic grade GNSS receiver and a redundant IMU (R-IMU) comprising four MEMS chips called Navchip (Intersense, 2015).
The sampling frequencies are 10 Hz for the GNSS receiver and 500 Hz for the IMU, respectively.A second GNSS receiver and an antenna can be placed in the nose of the fuselage in the distance of approximately 0.7 m from the first antenna.This allows the possibility of providing azimuth aiding, subject of which is, however, out of the scope of this contribution.

System calibration
System calibration is a process of determination of unknown sensor parameters and spatial relations between them.The sensor calibration includes a camera calibration and IMU sensor error modeling.These procedures are well established and documented and will not be described here, for more details see for example (Fraser, 1997, Syed et al., 2007).In our case, the camera calibration was performed during a separate flight over a dedicated field.The camera interior parameters remained fixed in the BBA for certain SO scenarios, particularly for DiSO and Fast AT.
In configurations with higher redundancy of observations, a selfcalibration was performed.The IMU sensor error modeling comprised a determination of random errors such as white noise, bias instability etc. and was performed using a method of Allan Variance (Hou, 2004).The deterministic part was estimated by multiposition calibration.
Additionally, it is necessary to determine spatial offsets, so-called lever-arms, between the GNSS antenna and the IMU and between the IMU and the camera.This can be a very complicated task due to the usage of consumer grade cameras on unmanned aerial platforms.There, we followed the approach suggested by (Rehak and Skaloud, 2015) where camera EO are recovered in static conditions and differentiated with the positions of the GNSS antenna surveyed by tachymetry.
When working with MEMS IMUs and imaging sensors not originally foreseen for mapping, one has to very carefully select a proper calibration method.The boresight estimation can be done either within self-calibration adjustment (so-called one-step) or by comparing the GNSS/INS-derived attitude of images for which EO are estimated via (aerial position-aided) calibration block (socalled two-step procedure).In our case, the boresight misalignment was calibrated during a special calibration flight using selfcalibration adjustment.

Test data
This study was conducted over agricultural fields and roads.The testing area has a size of approx.70 ha and is equipped with 20 permanent markers, location of which is accurately surveyed.Two flights were conducted over the field in two different years.First data set contains data from a block consisting of 7 parallel lines and 7 lines perpendicular to them, flown in two separate flight heights.The second data set, acquired one year earlier, contains just two lines simulating corridor mapping.The flight trajectories are depicted in Fig. 1.
The flight for block scenario was performed during crop vegetation stage and thus the surface suffers from strong homogeneity.The image observations of tie-points were automatically measured in the images using Pix4D mapper (Pix4D, 2015), the observations of GCPs and ChP were obtained manually using its rayCloud engine.The quality of detected key-points and tiepoints, respectively, is significantly degraded in certain areas.The Fig. 2 shows the distribution of tie-points and the gradient map represents a number of observations between these points and images.The distribution of tie-points is rather regular with a very dense area in the central part.In the gradient map, shades of grey represent a number of observations.Darker the grey value, less image observations for a given area.It is clear that the border areas have less observations due to the missing overlaps.However, the central part has also significantly less observations despite being flown from two directions at two different heights, as shown in Fig. 1.These weak areas make employment of indirect SO particularly very difficult as the tie-points cannot deliver sufficient precision due to the limited redundancy in observations.The aim of this study is to present the novel approaches of SO on the MAV on realistic data sets.Indeed, surface homogeneity is very common in agricultural areas and given the expansion of precision farming, a number of applications need to deal with such surfaces on regular basis.
Regarding the accuracy assessment, 5 points were used as GCPs and 15 as independent ChPs.The placement of GCPs was such that it simulates mapping of badly accessible area with GCPs placed only in the vicinity of the launching area, Fig. 1.It is important to say that the very same points were used for all testing scenarios and in the case of DiSO, the GCPs were completely excluded.This procedure was identical for both tested data sets.
The corridor data set was collected over the same calibration field.The length and width of the latter were 1200 m and 180 m respectively.The differences in topology of this particular corridor were around 30 m between the lowest and highest point.
Out of 9 visible points in the corridor, the first three were used as GCPs and the rest as ChPs.

Processing strategy
The data recorded during the flights were pre-processed in a way similar to mature mapping systems.The statistics on the number of tie points and corresponding observations is in Tab. 2. It is important to notice that the quality of tie points was somewhat lower than what can be expected from the state-of-the-art matching algorithms (e.g.1-1.5 pix).This was particularly due to the We can also observe repetitive patterns caused by correlation in attitude, e.g.due to initialization/alignment or by residual boresight.This somewhat less accurate attitude determination is of a less importance in the strong block configuration as its effect is mitigated by parallel strips, especially in the case of ISO with good distribution of tie-points.However, the influence on ground accuracy is more important further we go towards direct sensor orientation.
The solution for this problem offers relative orientation.By differentiating the attitude, we eliminate the effects of boresight as proved in Eq. 4 and mitigate the correlated part at the same time.To highlight this further, Fig. 4 shows the differences between initial relative attitude observations and relative attitude computed from adjusted attitude.It can be seen that the effects of residual boresight have vanished and the residuals have lower values, random distribution and low RMS.The practical evaluation was done by calculating different SO methods.The particular combinations are listed in Tab. 3.   Looking at the results of the Block 2015 data set, Tab. 4, and given the average spatial resolution of 4.5 cm/pix, the achieved accuracy varies significantly between different SO approaches.
As expected from the nature of the data, indirect SO fails completely.The noisy image measurements together with weak ground control resulted in huge residuals at the check points.Next, three ISO projects were processed.Again, due to the lower quality of image measurements and poor distribution of GCPs, the accuracy is mainly driven by the aerial control.There, the relative attitude control slightly outperforms the absolute by eliminating the relatively high attitude residuals as seen in Fig. 3.When removing image observations of tie-points from the adjustment and using Fast AT method, we can obtain similar results to that of ISO.There, the usage of absolute and relative aerial control together with a few GCPs and a very limited number of image measurements is sufficient to provide an accuracy of 1.5 pixel both in position and height while being significantly faster in processing.
The inclusion of relative attitude control improved the accuracy more than the absolute control.On the other hand, relative position control weakens the geometry in Fast AT as a significant shift is present in the X axis.The accuracy of DiSO corresponds to less than 2 pixels in position and around 3.5 pix in height.
The second data set is the corridor, where the GCPs were placed to one of its corner.Indeed, this represents a corridor mapping of an inaccessible area.The same testing scenarios to those of Block 2015 data set were processed.The type of control influences similarly the ground accuracy, but due to the missing side overlap, the relative aerial control in the Fast AT and ISO delivers slightly worse results in height and in X axis than the absolute.This was expected from the orientation of the strips that are almost aligned with the coordinate system as depicted in Fig. 1.The image observations quality was in this data set significantly better than in the Block 2015 and thus, ISO with absolute aerial control delivers the best accuracy.The latter is around 1.5 pixel in position and 2 pixels in height.Here again, the indirect SO cannot deliver acceptable results without a strong ground control which establishment may be, however, not feasible over inaccessible regions.

CONCLUSION AND PERSPECTIVES
This paper aimed at testing novel observation models in the context of MAVs.The initial part discussed the general problematic of sensor orientation with a specific focus on accurate mapping of difficult areas.The following part focused on theoretical description of mathematical models of absolute and relative aerial control and their implementation into a custom bundle adjustment software.An in-house developed MAV platform was used as a carrier of imaging and navigation sensors.Two data sets were acquired and served for quality evaluation.The flights were executed over a dedicated calibration field and the accuracy was evaluated at independent check points.Several mapping projects were presented with focus on scenarios with accurate aerial control.
It was shown that thanks to the aerial control, accurate mapping is possible even with areas with badly distributed GCPs.The main contribution of attitude absolute and relative control is the elimination of AT block structure and reduction of ground control in mapping configurations with weak geometry.Contrary, the contribution of attitude observation in strong AT blocks with well distributed tie points is rather limited.With relation to UAVs and MAVs in particular, the relative observations have great potential since the need of boresight calibration is omitted and therefore the process of sensor integration facilitated.The best achieved accuracy lies in the level of approx. 1 pixel, both in position and height.This accuracy is usually hardly achievable with conventional methods of sensor orientation i.e. indirect SO or ISO with absolute aerial control.
For certain projects with lower demands on accuracy, DiSO represents a very convenient and rapid way of mapping.Finally, Fast AT proved to be an excellent compromise between ISO and DiSO particularly in the context of relative attitude control.This method provides accuracy close to ISO while by speed and demands on configuration, e.g.image overlaps, is close to DiSO.Fast AT also offers considerable higher degree of robustness than DiSO.

Figure 3 .
Figure 3. Absolute attitude residuals; influence of IMU residual boresight on attitude accuracy In order to evaluate the accuracy of directly measured EO parameters, a separate project with all GCPs was created for each data set.The residuals of EO parameters in BBA serve as indicators of accuracy of the directly measured positions and attitude in flight.The estimated accuracy of GNSS/INS-derived absolute positions was 3.5 cm horizontally and 3 cm in height.Regarding the attitude, the RMS of omega and phi angles was 0.045 deg and 0.125 deg for kappa.

Table 2 .
Summary of test data Table 2 summarizes the most pertinent facts of both data sets.

Table 3 .
Test configurations and their properties

Table 4 .
Accuracy assessment at independent check points; Block 2015 represents the block with 5 GCPs and 15 ChPs, the Corridor 2014 has 3 GCPs and 6 ChPs; the Test acronyms correspond to those in Tab. 3