AUTOMATIC POLE DETECTION IN AERIAL AND SATELLITE IMAGERY FOR PRECISE IMAGE REGISTRATION WITH SAR GROUND CONTROL POINTS

: The world-wide absolute geographic positioning accuracy of optical Satellite imagery is mostly about a few pixels of the image resolution. So for example WorldView-3 images have a CE90 of about 4 m. Also the direct georeferencing without ground control information of aerial imagery is in the same range of one to a few metres. These inaccuracies originate predominantly in uncertainties of angular measurements for the sensor attitude. An angular error of only one arc-second at a satellite 750 km above ground results in an absolute error on ground of 3.6 metres. On the other hand radar satellites like TerraSAR-X or TanDEM-X do not measure angles but signal runtimes. So if we identify the same point in an optical image and in a radar image we can solve the problem of inaccurate angle-measurements in the optical sensor models and are able to georeference optical images world wide absolute to below one pixel. In this paper we present a method for identiﬁcation of point-objects which can be detected in both types of images: the footpoints of poles. If such a footpoint of a pole can be detected simultaneously in both types of images the geoposition of the optical image can be corrected to the accuracy of the point-measurement in the radar image. To achieve a high accuracy also a nearly perfect correction of all errors in signal propagation times of the radar signals has to be conducted. In this paper we describe how the footpoints of poles will be extracted in optical spaceborne or air-borne imagery and how these footpoints are correlated to the potential footpoints of poles detected in the radar imagery.


INTRODUCTION
Satellite imagery and also most aerial imagery have usually an absolute geographic accuracy without ground control information of about 4 m or 1 m respectively (CE90). But for many applications like the generation of high-definition maps (HDmaps) for automated driving a world-wide absolute accuracy of below 20 cm is required (EUSPA, 2021). The imprecise absolute position origins beneath errors in position and inertial measurements mainly in inaccurate angle-measurement of the attitude of the camera. In contrast to the acquisition of optical imagery a radar-sensor measures distances with accuracies to the wavelength of the signal. So, if we can identify an object both in an optical image and also the same object in a radar image we may correct the absolute georeferencing of the optical image to the accuracy of the radar image (Kurz et al., 2019). For typical X-band radar satellites like TerraSAR-X or TanDEM-X the absolute accuracy is in the range of only a few centimetres as shown in fig. 1 after applying all necessary corrections to the signal propagation time , Balss et al., 2018. Earlier work on correlation of SAR and optical imagery  led to a patent describing this method of registration of remote sensing imagery (Bamler et al., 2008).
If we have at least two perfectly corrected radar images with different acquisition angles we are able to calculate from these an absolute 3D coordinate of an object in the range of a few centimetres. If we are also able to transfer this information to the corresponding object in the optical image we can in turn correct the geopositioning of the optical image to the same accuracy. * corresponding author In this paper we present a method for automatic detection of poles in optical airborne and satellite imagery and the correlation of these with provided high accuracy SAR (Synthetic Aperture Radar) ground control points.

DATA
Input for the pole detection are original nadir or side-looking airborne images or orthorectified airborne images as shown in fig. 2 with an image resolution or ground sampling distance (GSD) of about 15 cm. For each image the acquisition time and also the sensor model has to be known to calculate the direction of the shadows and the directions of the projections of the vertical poles in the image. Also very high resolution (VHR) satellite images (e.g. WorldView-2/3/Legion or Pléiades/Pléiades-neo) with GSDs of 30 cm or better can be used either as original or as orthorectified images. A poor and a good example of a small section is shown in fig. 3. In the left -the poor -example the viewing direction is the same as the shadow direction and so the pole-footpoints are hidden by the poles themselves. In the right image the shadow is casted to NNW while the pole itself is projected to WNW and the footpoint can clearly be seen. Comparing the aerial imagery with a GSD of 15 cm to the satellite imagery with a GSD of 30 cm shows clearly that the extraction of pole-shadows from satellite imagery may not be as good as from aerial imagery since only large poles can be seen and the contrasts are still much worse than in aerial images.  In the first step for each type of the input imagery potential footpoints of poles will be extracted. For the radar imagery this is done in the "geodetic SAR-processor" which detects single strong corner-reflections within a dark surrounding without other reflections (see section Geodetic SAR processor). The poles in the optical image are detected as descibed in section Detection of poles and shadows.
So for each type of the input images a list of potential pole footpoints ("poles (SAR)" and "poles (optical)" in fig. 4) will be generated. Since the different acquisition methods result also in different sub-sets of poles to be detected and the methods both also generate different types of wrongly detected objects (false positives) in the next step the resulting lists of poles get correlated as described in section Correlation. The final result is a list of poles detected simultaneously in both types of images and the shifts between the footpoints of these poles. Using these derived shifts the absolute geographical position of optical images can be corrected to the accuracy of the positions delivered by the radar sensor.

Geodetic SAR processor
By correcting the signal propagation times of the radar pulses as they pass through the ionosphere and troposphere, an accuracy of a few centimeters can be achieved in the range measurement between the radar satellite and an object on the ground .
Using accurate orbit information and two images obtained with different incidence angles, a stereo method can be used to determine the 3D coordinates of the point . shows sections of SAR images of the two data sets, with the object to be measured located exactly in the center. The best results can be achieved with poles, since they form a so-called "corner reflector" together with the earth's surface as shown in fig. 1. This kind of reflector appear in the SAR images as sharply focused points. The coordinates from the base of the pole are determined.
Accuracies in the centimetre range can be achieved even from relatively small poles, such as road signs (see fig. 5). Furthermore, DLR's "SAR Geodesy Processor" estimates the achieved accuracy for each measured object (Balss et al., 2018).  For the detection of poles in optical imagery in a first step the direction of the shadows is calculated from the known geolocation and the acquisition time of the image. Second the direction(s) of the poles are calculated from the location and attitude of the camera system as shown in fig. 7. For satellite imagery only one pole direction can be assumed for the whole image. For aerial imagery the pole and shadow directions can only be used when processing original or single orthorectified images, but no more in ortho-mosaics where all viewing directions and acqusition times are mixed. Due to this our presented method is based only on detecting the shadows of poles. In fig. 8 a small section of the test area in Frankfurt am Main (Germany) is shown (cf. fig. 2). Left the true-ortho image of the aerial imagery, right the overlaid poles detected fully automatically by the method presented in this paper. In the first step narrow dark objects get filtered from the aerial image using morphological filtering as shown in fig. 9. Let the input image be I rgb and the mean (gray-value image) of the spectral bandsĪ = 1 3 r,g,b i

Detection of poles and shadows
Ii.
With the morphological gray erosion as and the morphological gray dilation as ⊕ the morphological gray closing (•) and the morphological gray opening (•) can be defined on a struc-turing element B as: The sensitivity of the method to our searched "narrow" objects is defined by the size of the structuring element B. So if only objects narrower than 5 pixels should be extracted B may be a box of 5 × 5 pixels. The narrow dark objects image L is calculated using the structuring element B as N.B.: If we calculate the viewing angle (which is strongly locally varying over the image for aerial images) from the sensor model we can also apply the narrow object detection for dark and bright objects (using L bright =Ī • B −Ī) and the appropriate gabor filter to the image to extract dark and bright poles themselves. Using both -the poles and the pole-shadows -will lead to a better extraction of the pole foot points. But as lined out above the acquisition geometry mostly not allows this and the poles are mostly also nearly invisible. So we concentrate in this paper only on the extraction of pole shadows.
On this gabor filtered image G a special clustering method "distmax" is applied which derives an image D containing the maximum length of continous segments in the same direction as shown on the right side of fig. 10.
Based on this clustered image D all objects with a certain minimum length get extracted and modeled as the searched shadowlines. The endpoint in direction towards the sun of each of these lines is the searched footpoint of the pole as show in fig. 11.

Correlation
After extraction of the pole-footpoints (green dots in fig. 12, left) these get correlated with the geodetic SAR ground control points ( fig. 12, right) which also represent mostly footpoints of poles. But both methods deliver different points and not a complete coverage of all poles. Also both methods generate false positives. In the case of the SAR-GCPs there are often corner reflectors by chance, e.g. on cars. In case of the optical pole detection also dark narrow features coincidentally oriented in the sun shadow direction like e.g. the road between zebra crossings may be detected as poles (cf. fig. 10, left top edge). For this we developed a special correlation of both results incorporating a sophisticated quality assurement and outlier detection. Fig. 13 shows an overview of this correlation process. Figure 13. Overview of the correlation process For the ng SAR-GCPs and np poles detected in the optical imagery first a ng × np-matrix is calculated containing the distances of GCP g to pole p. Assuming a maximum distance dmax based on the nominal accuracy of the sensor-systemfor aerial sensors with an estimated accuracy of about 1 m we take 5 m -only distances lower than dmax are kept. From these a list containing the distances in x-and y-direction and the involved pole-and GCP-IDs with length l is built as cl = [[dx, dy, p, g], .
..]. dx, dy) and the distances to this mean position as di = (dx,i − mx) 2 + (dy,i − my) 2 . In each iteration the list element i with the maximum di is dropped from the list. The iteration terminates until the minimum number of points is reached or the standard deviation of all (dx, dy) to (mx, my) drops below half the resolution.

Now iteratively the mean shift (mx, my) is calculated from all
The result after the pole-detection in the optical imagery and the correlation of the footpoints from these with the geodetic SAR ground control points finally gives the correction for the desired global absolute geo-referencing of the optical imagery.

Aerial imagery
The first dataset used for our experiments are aerial images acquired on 2017-04-20 over Frankfurt am Main (Germany) as shown in fig. 2. Calculating the sun azimuth angle for the center of the image at 8.597 • E, 50.129 • N and the acquisition time 13:50:42 UTC of the selected image (R1906, right looking camera of the DLR 3K camera system (Kurz et al., 2012)) gives an sun azimuth angle of 230.66 • and a sun zenith angle of 49 • .
First the pole shadow detection -the detection of narrow dark objects -as shown already in the description of the method in fig. 9 is done followed by the gabor-filtering and distmax operation as shown in fig. 10.
For the whole scene -the area shown in fig. 2, right -covering 450 × 750 m 2 (3000 × 5000 pixels) 2598 poles were detected and 9380 GCPs were delivered from the geodetic SAR processor. From the whole 2598 × 9380 correlation matrix only 271 pairs fulfill the condition dmax < 5 m. The mean shift of all these pairs was (mx, my) = (0.08, −0.32) (all in metres) with a standard deviation of (σx, σy) = (2.23, 2.26). The pair with the largest deviation of 5.27 m was the first pair to remove. After 260 iterations the minimum of 10 points remained with a final mean (mx, my) = (0.68, −0.99) and a standard deviation of (σx, σy) = (0.11, 0.08). So the final correction is a shift of (68 cm,−99 cm) with a GSD of 15 cm as shown in fig. 14 (red line at the lower left pole).

Satellite imagery
The second example is a section of 1800×900 m 2 (6000×3000 pixels at GSD of 30 cm) from a WorldView-3 image acquired over Munich (Germany) at 2019-07-04, 10:21:26 UTC as shown in fig. 15.  Due to the different acquisition methods of optical and SAR imagery different objects are detected. In the case of optical images the shadow must be clearly visible, this means it should be cast to a plain bright surface. In case of the radar measurement the ground before the pole in direction to the satellite has to be smooth, e.g. a road. If the ground is rugged or grass the signal is degraded. Furthermore the pole and the ground has to form a rectangular corner. If the pole or the road is slanted the signal get's also worse.
On the other hand many wrong objects will be detected both in the radar and in the optical imagery. The geodesic SAR processor returns all strong reflectors with no other similar reflectors in a certain surrounding as radar GCPs.
In case of optical imagery all narrow dark lines lying coincidentally in the sun shadow direction will be detected as pole shadows. These may be dark shadow corners at cars, rail tracks, etc. as shown in fig. 19.
Using only the shadows in the images has also an other big disadvantage: Often the footpoint of poles is covered with high grass, bushes or other obstacles. So the footpoint can not be clearly seen and the pole appears to be shifted in shadow direction. But using a large number of detected poles in the optical and also in the SAR images and the described sophisticated outlier detection and correlation algorithm the finally reached improvement of the global absolute accuracy of aerial and satellite imagery is better than one image-pixel.

CONCLUSION AND OUTLOOK
In this paper we presented a method for detection of foot points of poles in optical aerial or satellite imagery and correlating these with high accuracy geodetic SAR ground control points to achieve a world wide absolute geographic geolocation of the optical imagery better than one image pixel.
As shown the final accuracy of the correlation between radar points and the detected and correlated pole foot points in the optical image is about 10 cm for aerial imagery with a GSD of 15 cm and about 20 cm for satellite imagery with a GSD of 30 cm. The derived absolute corrections of the optical imagery are about 1.2 m for the aerial and 3.2 m for satellite images.
For the correction a coverage of the required region with geodetic SAR ground control points derived e.g. from radar images of TerraSAR-X or TanDEM-X in the Spotlight Mode (GSD about 1 m) is necessary. Using such reference points all optical imagery of the area -aerial or satellite -can be automatically georeferenced using the described method. It must be pointed out that there exist some restraints for the acquisition of the optical imagery. So the pole shadows must be clearly visible. This means for airborne images the flights have to be conducted at clear, non cloudy sky. This restriction is on the other hand always satisfied for satellite images. But in this case the acquisition geometry is essential. So only satellite images with a difference of at least 20 to 30 • between sun-and acqusition-azimuth angles can be used (cf. fig. 3) otherwise the pole shadow is hidden by the pole itself. Since the detections of poles in both -the radar and aerial data -is incomplete and suffers from many misdetections additional work will be needed to generate nearly complete pole inventories. In the evaluation we found 25 real poles in the radar and 35 real poles in the optical data with 6 poles detected in both. So only 6 poles were detected out of potentially 60. For a better coverage all points from radar and optical detections may be taken into account and a more detailled analysis may be conducted if there may be a pole candidate also in the other image which was rejected beforehand.