GENERATION OF REFERENCE VEHICLE TRAJECTORIES IN REAL-WORLD SITUATIONS USING AERIAL IMAGERY FROM A HELICOPTER

: Highly accurate reference vehicle trajectories are required in the automotive domain e. g. for testing mobile GNSS devices. Com-mon methods used to determine reference trajectories are based on the same working principles as the device under test and suffer from the same underlying error problems. In this paper, a new method to generate reference vehicle trajectories in real-world situations using simultaneously acquired aerial imagery from a helicopter is presented. This method requires independent height information which is coming from a LIDAR DTM and the relative height of the GNSS device. The reference trajectory is then derived by forward intersection of the vehicle position in each image with the DTM. In this context, the influence of all relevant error sources were analysed, like the error from the LIDAR DTM, from the sensor latency, from the semi-automatic matching of the vehicle marking, and from the image orientation. Results show that the presented method provides a tool for creating reference trajectories that is independent of the GNSS reception at the vehicle. Moreover, it can be demonstrated that the proposed method reaches an accuracy level of 10 cm , which is defined as necessary for certification and validation of automotive GNSS devices.


INTRODUCTION
Recent technological developments in the automotive domain have led to successive and increasing levels of autonomous operation. Critical technological components should and must be continuously tested, validated, and certificated on public roads with regard to their reliability, their safety as well as the overall functionality and system accuracy. A key parameter in this context is the accuracy of the absolute vehicle position.
The absolute position of a vehicle can only be determined by GNSS (Global Navigation Satellite System). A delta to the absolute position can be identified, especially in GNSS denied environments, by auxiliary sensors, e. g. INS (Inertial Navigation System), and odometer, which support the successful localization of the automotive GNSS. For autonomous driving, absolute positioning capability with, at least, lane accuracy in combination with high integrity in all driving environments is required. The demanded positioning accuracy in the field of autonomous driving is 20 cm horizontal and 2 m vertical, respectively 1 .
The performance of automotive GNSS devices is normally assessed by using a high-grade GNSS receiver with assumed higher accuracy and performance. This approach tends to ignore the fact that both the reference and the test device suffer from the same underlying error problems, since they operate according to the same working principles. As the real or intrinsic accuracy of GNSS receivers is unknown or quite difficult to estimate (as they work with the same principles and data) using an independent approach to assess the intrinsic accuracy of such devices is highly desirable and necessary, in particular in GNSS-denied areas and in safety-of-life applications as in autonomous driving cases (SAE L4 and L5).
Therefore, new methods for the determination of the reference trajectory need to be developed facilitating the characterisation of GNSS-INS-based reference solutions. There exist relative positioning methods with robotic total stations (Vaidis et al., 2021) which require three synchronised robotic stations at measured positions that can track prisms mounted on the target vehicle. Based on three reference positions with synchronised relative measurements of distance, azimuth and elevation to the target prisms the reference trajectory can be calculated. An alternative for this approach would be the relative positioning through visual odometry, e. g. (Lin et al., 2019).
In this paper, the concept and results of a new method to generate reference vehicle trajectories in real-world situations using simultaneously acquired aerial imagery from a helicopter is presented. Special attention is paid to the suitability of the proposed method for the validation and certification of automotive GNSS receivers, where an accuracy level of 10 cm for the horizontal position is demanded.

CONCEPT AND DATA
Absolute positioning of moving objects using airborne imagery is enabled by forward intersection of image objects. In our case, the image ray of the visible GNSS antenna at the target vehicle with position xv, yv in the image is intersected with a digital terrain model (DTM, see Figure 1 Based on precise rotation parameters rij ∀ij ∈ 1..3 and position coordinates X0, Y0, Z0 of the aerial images which are acquired with a high frame rate, the position Xv, Yv of the reference GNSS receiver antenna can be determined with the collinearity equation: (2) Further parameters are the focal length c and the image principal point xp, yp, which are output of a preceding camera calibration.
A helicopter is utilised as the flying platform, since the operation of a helicopter is far less regulated compared to a drone, especially with regard to capturing aerial imagery of real-world situations such as in densely populated regions 2 . The helicopter is following the target vehicle simultaneously in terms of speed and direction so that the vehicle can be continuously imaged by the airborne camera system during the measurement campaign. Aspects like vehicle speed, flight control, reaction time, and flying permits must be considered during the campaigns. The basic idea is to provide the additional height information of the GNSS antenna, with which the vehicle position independently of the vehicle GNSS determination can be derived. The absolute height of the vehicle GNSS antenna is the key parameter, which is simply the sum of the absolute road surface height and the height of the GNSS receiver relative to the ground. The first summand is derived from the LIDAR DTM, the second is measured directly before the campaign (see Figure 2).

Aerial camera and test site
For the aerial image sequences acquisition, two of the three cameras of the DLR 4k system (Kurz et al., 2014) with different focal lengths, 35 mm and 50 mm, (see Figure 3) are used. The third one with looking direction in nadir is used for the operator's live view that displays the image similar to the camera with the smaller focal length. The operator is able to check whether the target vehicle is still in the field of view of the camera system and can advise the pilots to change the speed or the flight path in order to improve tracking the vehicle. The footprints of the images cover the area around the test vehicle with different ground sample distances (GSD) of 7 cm resp. 14 cm based on the two focal lengths. The footprint sizes are 320 m × 240 m resp. 457 m × 342 m assuming a flight height of 500 m above ground. Assuming also that the test vehicle is at perfect nadir position below the helicopter, the environment around the vehicle is mapped 120 m resp. 171 m each in forward and backward direction. The image repetition rate is set to 1 Hz during the whole campaign. Each aerial image exposed generates a synchronised data frame of the GNSS/INS capturing the GNSS position and the image attitudes (i. e., the exterior orientation of the camera system) at the time of exposure. Additionally, 46 stationary measured GNSS points are used for the georeferencing of the aerial images, which are distributed along the test tracks.
In order to collect data for a representative automotive GNSS validation, test tracks are defined which fulfill certain requirements in terms of length, vehicle speeds, GNSS occlusions, etc. The selected test tracks in the south of Germany have a length of 30 km and are classified according to their purpose in calibration (CAL), rural road (RURAL), motorway (MOTO), and city area (URBAN) (see Figure 4). Minimum numbers of valid images are defined for the certification process of automotive GNSS devices for each scenario: 400 images for rural, 240 for motorway and 600 for the urban scenario. Moreover, an upper threshold of 10 cm was defined for the accuracy of the horizontal vehicle position derived from the aerial images (the determination of these thresholds and the certification process is not part of this paper).

Data acquisition
Two flight campaigns with the DLR 4k camera system were performed on 14/07 and 15/07/2021. A target vehicle equipped with two different GNSS receivers was tracked by the helicopter Figure 4. Map of test tracks with four scenarios (blue: calibration and rural, green: motorway, orange/red: urban area) and GNSS measured ground control (red) and check points (blue).
pilots as the vehicle drove along the test tracks. A decisive factor for the success of the campaign was that a defined number of aerial images were acquired containing the vehicle, i. e. the vehicle must be located within the footprint of the camera. In addition, the vehicle was required to travel at a minimum speed of 10 km/h. Finally, the helicopter flight succeeded very synchronously with the target vehicle. Thus, the number of valid images were higher than the thresholds defined at the beginning, in particular for the camera with 50 mm focal length. For further analysis, only images with 50 mm focal length are used. There are also other advantages of longer focal lengths which will be addressed in subsequent sections.

GNSS devices and DGPS services
In the test campaign two different types of GNSS receivers were used sharing the same antenna. The first is a high-grade multi constellation GNSS receiver 3 with multi-path and interference mitigation as well as ionospheric correction. The second is an automotive-grade GNSS receiver 4 . The first receiver is often used to measure a reference trajectory, the second one represents the GNSS devices installed in vehicles. The installation and wiring of the two GNSS receivers is illustrated in Figure 5. The GNSS devices were operated with different DG-PSs (Differential Global Positioning System) (see Table 2

Semi-automatic matching of target vehicle
In order to enable automated measurements of the position of the GNSS receiver in the aerial images, it is necessary to recognise the vehicle unambiguously and to be able to determine the exact position of the receiver on the car roof. For this purpose, a magnetic sticker with striking colors and high contrast was placed on the roof of the target vehicle. In addition, the pattern supported the pilot and helicopter crew in recognising the vehicle on the road in case of interruptions during the tracking.
An automatic tracking of the cross center in the aerial image sequences based on a NCC-matching was applied. For the tracking, sub-pixel accuracy and rotations must be taken into account. It is also advantageous to limit the search space by approximately projecting the vehicle into the original image based on the measured trajectory. After this process, all positions are

Accuracy of LIDAR DTM
An analysis of the LIDAR DTM height shows that its accuracy is much better than the specified 10 cm all over the test area. The statistical parameters of the height differences dZ between the LIDAR DTM and the stationary GNSS measured heights at 46 points are µ = −0.013 m, σ = 0.040 m, and RM SE = 0.041 m. This means that the position error induced by the height error of around 4 cm does not exceed 2 cm at image edges (based on a viewing angle of max. 23 • with 50 mm focal length). The influence of the DTM error in exact nadir direction is zero. Figure 8. Height differences between vehicle based GNSS measurements (mode A) and DTM heights plus relative vehicle heights at the calibration track.
Another indication of the high quality of the LIDAR DTM is a comparison of the measured heights at the vehicle with the heights of the DTM plus the relative vehicle height. In Figure 8 the differences are plotted for 360 positions along the calibration track. On average, an RMSE of 0.0294 m is calculated which again proofs the feasibility of the height concept of the proposed method in general.

Positional accuracy of aerial imagery
All aerial images are georeferenced by a bundle adjustment using the ground control points measured with a stationary GNSS device. The positional accuracy of the image sequences is then determined using 23 check points which are also measured with a stationary GNSS device. The separation between ground control and check points was done randomly. After bundle adjustment the RMSE at check points was calculated to 0.064 m for X, 0.056 m for Y, and 0.109 m for Z coordinates.
The image acquisition configuration is not ideal for highly accurate image orientation, as the helicopter stops, changes speed and there are mostly no parallel flight strips. In particular in the urban area, problems occurred with image orientation, as the vehicle had to deviate from the planned route due to road works (see Figure 9). As a result, the aerial images are not covering all measured ground control and check points along the originally planned route. Thus, the image block of the vehicle tracking campaign is extended by images from a regular flight grid. Based on the additional image acquisition, all ground control and check points are covered and the original image block is stabilized.

Results at the calibration track
In order to estimate the general accuracy of the presented method, the coordinate differences between aerial image vehicle position and GNSS measured position are analysed. These differences are caused by various error sources, e. g. coming from the image georeferencing, the pattern matching, the LIDAR DTM, and the post-processed GNSS trajectory. However, in this analysis it is not possible to estimate the influences of individual error sources, as it was done in Sections 3.1. and 3.2., because only the sum of all relevant individual differences is known.
The calibration track is particularly suitable for this investigation, as the GNSS receiving situation is assumed to be perfect Figure 9. Footprints of the urban image sequence (black) supplemented with an image block acquired in regular strips (cyan), planned route (green) and ground control and check points (red dots).
without occlusions. Thus, the error of the GNSS trajectory can be minimized considerably. Furthermore, the GNSS trajectory is post-processed using SAPOS (mode A), which provides accuracy at centimetre level. In Figure 10 the absolute coordinate differences between both solutions are shown for all images along the calibration track. The coordinate differences between the aerial image derived position and the GNSS position does no contain any systematics and the absolute differences are in most cases smaller than 10 cm. The RMSE values are the same for both calibration tracks 0.06 m for X and 0.06 m for Y. This error includes the noise caused by the vehicle based GNSS measurements whose RMSE is estimated by the postprocessing software to 0.02 m for X and Y. Given these values, it can be concluded that from a statistical point of view the threshold of 10 cm can be reached even if single differences are higher.
Statistical information about the coordinate differences at the calibration track for two days are provided in Tables 3 and 4. Column 2 of the tables lists the number of samples used for the analysis. The statistics include the RMSE of the X and Y differences, the mean and standard deviation of the horizontal distance D and the vertical distance dZ. Also provided are the percentiles of D for 63.8 %, 95.4 %, and 99.7 %. In particular the 95.4 % is required for the pass/fail criteria decision process for the validation of automotive GNSS receivers.

Validation of the sensor latency
It is worth mentioning that based on the calibration track the specified sensor latency could be validated in a separate step. real time of acquisition. It plays an important role if the positions of moving objects based on their time stamps are compared. For the DLR 4k camera system a sensor latency of 3 ms is specified. Figure 11 shows distances between GNSS positions (mode A) and aerial image positions in driving direction for four different latency times at the calibration track. For this, four equally spaced latency times in the range between 0 ms and 9 ms are chosen and the differences are calculated, whose sum reach a minimum for a latency time around 3 ms (qed). In Figure 11 it can also be seen that the differences are the same when the vehicle is stationary, while at higher speeds the differences become larger.

OUTLOOK
The results at the rural, motorway and urban scenario are not analysed in this paper and will be presented in future. This topic is also closely connected to the GNSS certification process. The following issues in this context will be investigated: • The pass/fail criteria decision process for the validation of automotive GNSS receivers based on the proposed methodology • A comparison between reference track and high-and automotive-grade GNSS tracks like in Figure 12 • A discussion about reasons for failures at the GNSS tracks, e. g. by using a 3D model of the environment • A comparison between GNSS parameters like DOP (dilution of precision) and number of satellites with deviations from the reference track

SUMMARY AND CONCLUSIONS
In this paper, the concept and results of a new method to generate reference vehicle trajectories in real-world situations using aerial imagery are presented. It was shown that the developed method provides a tool for creating reference trajectories that is independent of the GNSS reception at the vehicle. Moreover, it was demonstrated that the proposed method reaches an accuracy level of 10 cm which is defined as necessary for certification and validation of automotive GNSS receivers.
In particular, the influence of all relevant error sources were analysed, like the error from the LIDAR DTM, from the sensor latency, from the semi-automatic matching of the cross center, and from the image orientation. For example, the accuracy of the LIDAR DTM was validated with stationary GNSS measurements which resulted in a RMSE of 4 cm in height. This deviation may cause an error of 2 cm in X/Y position for the given configuration. The RMSE error caused by the image orientation is 6 cm resp. 5 cm, which was determined by check points. Another analysis demonstrated that for all points at the calibration track the RMSE error is 0.06 m for X and 0.06 m for Y coordinates. This error includes all relevant possible error sources.   Although the analyses performed cannot fully prove that every vehicle position derived from the aerial images is below the 10 cm threshold, no evidence of major deviations was found in the randomly selected samples. Of course, based on this result, it is also clear that the automotive industry's specification of 20 cm is far undercut.
The presented method thus not only fulfils the qualitative requirements of a certification of GNSS receivers, but also provides data for a high-precision analysis of the deviations of GNSS positions of different receivers. Thus, a valuable contribution to the safety assessment of autonomous driving is provided. In addition, the method is not limited to the test areas mentioned and can be applied to other areas and sensors if the required reference data is available.
For "daily use" applications, a helicopter might be too costly. But the certification of a GNSS receiver, e. g. for series use in a car, is a special case: Here it is critical to test in all relevant scenarios (urban canyons, trees, reflective house fronts, etc.). Generally, an ascent permit for a drone is not issued. There are also no known drones that can track a vehicle autonomously for a longer distance, such as 20 km. However, the higher flight altitude of the helicopter compared to a drone has little effect on the expected accuracy, which in turn diminishes a potential advantage of drones.