LIDAR AND PHOTOGRAMMETRIC DATASETS INTEGRATION USING SUB-BLOCK OF IMAGES

Imagery and Lidar datasets have been used frequently to extract geoinformation. Datasets in the same mapping or geodetic frame is a fundamental condition for this application. Nowadays, Direct Sensor Orientation (DSO) can be considered as a mandatory technology to be used in the aerial photogrammetric survey. Although the DSO provides a high degree of automation process due to the GNSS/INS technologies, the accuracies of the obtained results from the imagery and Lidar surveys are dependent on the quality of a group of parameters that models accurately the user conditions of the system at the moment the job is performed. This paper shows the study that was performed to improve the tridimensional accuracies of the aerial imagery and Lidar datasets integration using the 3D photogrammetric intersection of single models (pairs of images) with Exterior Orientation Parameters (EOP) estimated from DSO. A Bundle Adjustment with additional parameters (BBA) of a small sub-block of images is used to refine the Interior Orientation Parameters (IOP) and EOP in the job condition. In the 3D photogrammetric intersection experiments using the proposed approach, the horizontal and vertical accuracies, estimated by the Root Mean Square Error (RMSE) of the 3D discrepancies from the Lidar checkpoints, increased around of 25% and 75% respectively.


INTRODUCTION
The integration of aerial imagery and Lidar datasets has improved the autonomous approaches for geo-information extraction from imagery. However, to use this combination, accurate Interior (IOP) and Exterior Orientation Parameters (EOP) are required to have imagery and Lidar datasets in the same mapping or geodetic frame. Today, many photogrammetric aerial surveys are performed simultaneously to collect images and Lidar datasets in the same reference system using Direct Sensor Orientation technology. Although the Lidar and image sensors share the same Inertial Measurement Unit (IMU), frequently the datasets do not match accurately due to the variability of the Interior Orientation Parameters (IOP) and mounting parameters in-flight condition.
Usually, the image sensor system calibration is performed independently of the Lidar system calibration and Direct Georeferencing of images depends on the local flight conditions, for example, temperature and atmospheric pressure variability may modify the relative position and orientation of the camera to IMU (Kersting, 2011). Additionally, the atmospheric refraction may modify the collinearity condition of the light ray in its trajectory between object and image spaces (Andrade 1977). Then, a procedure not general, used to improve the accuracy of the Direct Georeferencing of images, is the insitu system calibration, including the interior orientation and mounting parameters (Yastikli and Jacobsen, 2005). The system calibration, in principle, can be performed before or after every mission to estimate the mounting and IOP parameters. However, due to technical requirements, time, and cost, the system calibration is not regularly applied. Overcoming the resources limitations to perform the system calibration, different solutions are proposed to improve the accuracy of the Direct Georeferencing, such as Integrated Sensor Orientation (Heipke et al., 2002) and refinement of a set of system calibration parameters via the in situ self-calibration (Cramer and Stallman, 2002). Mitishita et al., (2014) showed a study to increase the accuracies of Direct Georeferencing of images by in-situ camera calibration. Two sets of IOPs, estimated by in situ calibration process, are analyzed and used in photogrammetric experiments. The IOPs in flight condition improve significantly the accuracies of 3D coordinates of the point object extraction. Elsharkawy and Habib, (2016) performed a study to verify the tridimensional accuracies from Direct Georeferencing and Integrated Sensor Orientation considering small biases in direct measurements of the position and orientation of the INS body frame and similar errors in boresight angles and lever arm components.
Direct Sensor Orientation is a frequent technology used in simultaneous photogrammetric and Lidar surveys. This procedure can automatically acquire the Lidar and imagery datasets in the same mapping frame by the global navigation satellite system and inertial measurement unit (GNSS/IMU) systems. However, the accuracies of the integration of the photogrammetric and Lidar datasets are dependent on the quality of parameters that models accurately the systems at the same time as the survey. Gneeniss et al. 2015 proposed a methodology to perform camera self-calibration using Lidar Control Points (LCPs) in the Bundle-Block Adjustment (BBA). The approach provided an efficient and cost-effective alternative for in-flight camera calibration.
Integrated Sensor Orientation (ISO) has been used to improve the accuracies of the imagery and Lidar datasets integration. Using Lidar data as a ground control of position information, the ISO approach refines the direct EOP estimation when its values do not attain the required accuracies. However, to carry out the ISO with a minimum number of Lidar control points (LCPs) over the block of images area or without any use of ground control points, three basic conditions must be met. First, the block configuration should have enough forward and side overlap areas with a minimum number of tie points in these areas (Von Gruber positions); second, accurate IOP values and third, accurate values of EOPs´ standard deviations (Cramer and Stallman, 2001). Mitishita, et al., (2016) proposed an approach to improve the Integrated Sensor Orientation (ISO) using in-situ camera calibration and a small sub-block of images extracted from the main image block. Costa et al., (2018) performed studies to verify the viability of methodology to improve the accuracies of the integration of aerial imagery and Lidar datasets, using Lidar control points (LCPs) in Integrated Sensor Orientation (ISO). Improving the study performed by Costa et al., (2018), this paper shows the study that was performed to improve the tridimensional accuracies of integration of aerial images and Lidar datasets using the 3D photogrammetric intersection of single models and EOPs estimated from direct sensor orientation. In this study, the term "single model" is a pair of images with absolute orientation. The Lidar dataset is used as a ground control of position information for the imagery survey performed by Direct Sensor Orientation technology.
In this way, this study considers that the obtained results from a bundle adjustment with additional parameters can be used to compute the small variability of the IOP and EOP in-flight conditions.
To evaluate the horizontal and vertical accuracies of the Lidar and imagery integration, two sets of experiments of the 3D photogrammetric intersection are performed. The first uses nominal IOP and EOPs and the second uses EOPs and IOP inflight conditions; a set of LCPs is used as the checkpoints.
The following four sections contain information about the materials and methodology, results and discussions of the performed experiments, as well as the conclusion and recommendations for future work.

Imagery and Lidar surveys
Two independent aerial surveys were conducted in an area, approximately 74 km 2 in size, of the municipal district of the Pouso Alegre city (State of Minas Gerais -Brazil). The first acquired a photogrammetric imagery block in May 2015 using the UltraCam Xp (S/N UC-SXp-1-40813045) digital aerial camera, mounted with Linos Vexcel Apo-Sironar Digital HR 5.6/100mm. The image block has six strips, taken in opposite directions (approximately Northeast and Southwest) with around 30% of side overlap. Each strip has seventeen images, acquired with nearly 60% forward overlap. Figure 1 shows the layout of the image block. The flight height was approximately 1600 m. For this flight height, the ground sample distance (GSD) is close to 0.15 m. The camera CCD sensor has 195 million effective pixels (11310 × 17310 pixels), the dimension of 67.86 mm x 103.860 mm, the pixel size equal to 0.006 mm.
The second flight survey acquired the Lidar dataset in June 2015. The survey was conducted in the same area of the block of image (approximately 74 km 2 in size, of the municipal district of the Pouso Alegre city). The ALS60 Laser Scanner equipped with a Leica CUS6 IMU was used. It has an absolute accuracies (RMS) -Position < 0.1 m; Roll and Pitch < 0.0025 deg; Yaw < 0.005 deg. The individual Lidar strips were collected with a mean point density of 5 points/m2 (nearly 0.25 m point spacing). According to the sensor and flight specifications, 0.20 m horizontal and 0.10 m vertical accuracies are expected for the acquired Lidar data.
The imagery and Lidar datasets were achieved from independent aerial photogrammetric surveys. However, both datasets are provided by the photogrammetric company in the same mapping reference frame (Gauss conform Projection).

Lidar control points (LCPs) extraction
The approach, used in this study, focused on a semi-automatic point feature extraction by the intersection of three building roof planes. According to Costa et al., (2017) the used approach is performed in four steps: filtering Lidar points on the building roofs; roof building planes extraction; roof building planes modeling; three planes intersection (LCP Characterization).

IOP and EOP refinements
The bundle adjustment with additional parameters is used to refine the interior orientation parameters of the UltraCam Xp camera in the flight conditions. Using the collinearity equations and Least Squares Bundle Block Adjustment (BBA), the theoretical collinearity condition among the point image, camera exposition station and point object is in practice recovered by additional parameters related to lens distortions, coordinates of principal point. The physical model, as proposed by Brown, (1971) is used to compute the Interior Orientation Parameters in flight condition. In this study, the bundle adjustment uses Lidar Derived Control Points (LCPs) and a small sub-block of images extracted from the entire image block obtained in the aerial survey.
The sub-block has two strips taken in opposite directions (approximately northeast and southwest) with around 30% of lateral overlap. Each strip has three images, acquired with nearly 60% forward overlap. Three non-aligned LCPs (2 horizontal/vertical and 1 vertical) were used as ground control points to perform the process considering the sub-block dimension. This small configuration of the control point was used, only, to fix the mapping reference frame. Figure 1 also shows the spatial distribution of three LCPs and the position of the sub-block in the entire images block.
Approximately 34 tie points, close to Von Gruber regions, were measured by semi-automatic procedure, using the Leica Photogrammetry Suite (LPS) 2011. Additionally, the EOPs of the images from direct sensor orientation are included in the BBA as additional observations by weight constraint. The EOP values are weighted according to the nominal accuracies.
The roots mean square errors (RMSEs) of the residuals of orientation angles (ω, φ, χ), computed in the bundle adjustment, are used to refine the values of EOPs that were estimated by Direct Sensor Orientation.

3D photogrammetric intersection
The obtained results from the 3D photogrammetric intersection of single models (pairs of images) are used to verify the accuracies of the integration of imagery and Lidar datasets. The values of refined IOP, are used to perform the experiments. In addition, the model orientation angles, from the Direct Sensor Orientation, are also refined using the root mean square errors (RMSE) of orientation angles (ω, φ, χ) residuals computed in the bundle adjustment with additional parameters. Four models (pairs of images) inside of the entire image block are used to perform the 3D photogrammetric intersection.
Two experiments are conducted for each model. The first used only unrefined IOP and EOP values and the second used the IOP and EOP that were refined by the proposed approach.
In the experiments of the 3D photogrammetric intersection, the images EOPs (position and orientation), estimated from direct sensor orientation, are fixed absolutely by weight constraints. Using unrefined IOP and EOPs, the EOP values are weighted according to the nominal precisions of the IMU (GNSS/INS integration). On the other hand, when IOP and EOPs are refined by the proposed approach, the EOPs are weighted according to the refinement precisions estimated empirically in the BBA.
The obtained results from the experiments are compared and discussed. Two or three LCPs inside in each photogrammetric model (pair of images) are used as checkpoints to achieve the horizontal and vertical ground accuracies.

Lidar control points (LCPs) extraction
Twelve Lidar Control Points are computed to perform this study according Costa et al., (2017). Three LCPs are used, as the control points, to perform the BBA. The remained LCPs are used as check points to verify the horizontal and vertical accuracies of the 3D photogrammetric intersection (imagery and Lidar datasets integration). The spatial distribution of these LCPs in the image sub-block and the entire image block is shown in Figure 1. The photogrammetric models (pairs of images), within the entire image block, have two or three LCPs.

Bundle Adjustment with additional parameters
The bundle adjustment with additional parameters was performed using the following measurements precisions: 0.003 mm (half of the pixel) for x and y image coordinates; 15 cm for X, Y, and Z coordinates for the LCPs (Lidar 3D coordinates accuracies); for the EOPs, computed from Direct Sensor Orientation, nominal values of the Applanix POSTrack AV 310 IMU were adopted: 10 centimeters for the tridimensional position and 54 arc seconds for Omega (ω) and Phi (φ) and 126 arc seconds for Kappa (χ), considering the results of the trajectory accuracies in Post-Processing mode.
The UFPR Bundle Block Adjustment (calibration and photo triangulation) is used to perform the BBA experiment. Table 2 shows the values of additional parameters (IOP) that were considered significant. A parameter was considered significant when its standard deviation (parameter's precision estimation), obtained from the variance-covariance matrix, was at least ten times smaller than the parameter magnitude. The values of the nominal IOP from the camera certificate were also included in Table 2 to support the discussion and analyses.  EOPs measurements of the images taken in Northeast flight direction are, in almost all of them, smaller than the a-priori precisions adopted, except the residuals from kappa rotation angle, as can be seen in Figure 3. The RMSE computed from these residuals is nearly two times the value of nominal precision, which will be considered in this study as a systematic error in this measurement. Considering the images taken in Southwest flight direction, most of all the obtained residuals from EOPS measurements are smaller than a-priori precisions adopted. Only a small tendency, close to the nominal precision, is found again in kappa rotation angle measurements, as can be seen in Figure    The values of the measurements RMSEs, shown in Table 1 and the obtained precisions of the additional parameters (IOP), shown in Table 2, confirm that the BBA achieved acceptable precisions. Comparing the values of focal length and the principal point coordinates, from the camera manufacturer certificate to the BBA, it can be seen that their values were changed significantly. The variations might be attributed to instability in the camera geometry and/or environmental changes under operational conditions or due to the mathematical correlation among parameters (f -Zs; xo -Xs; yo -Ys) in BBA. These correlations may allow the parameters related to the focal length and coordinates of the principal point to absorb some inaccuracies connected to the direct estimation of the Xs, Ys, and Zs. These correlations do not invalidate this study since refined IOP will be used to experiment with the 3D photogrammetric intersection in-flight conditions.
Unlike the parameter k1 from the camera certificate (reported in Table 2), the radial lens distortion was not found in obtained results from the BBA. All radial lens distortion parameters were not significant in the variance and covariance matrix from the BBA process. This variation can be considered as an expected result due to the insignificant value of radial distortion reported in the camera certificate (less than 0.004 mm). 0.000 σ yo (mm) 0.000 k 1 (mm -2 ) -2.0361265x10 -08 σ k1 (mm -2 ) 1.1316235x10 -09 f = Focal length; (x o , y o ) = Coordinates of principal point; (k 1 ) = Radial lens distortion; (σ) = Standard deviation;

3D photogrammetric intersection
As mentioned before, the 3D photogrammetric intersection of single models (pairs of images) using EOP measurements from Direct Sensor Orientation is used to integrate the imagery and Lidar datasets. The study considers that the tridimensional accuracies of 3D photogrammetric intersection can be improved by the obtained results from BBA or they are capable to model Kappa the small variability in the IOP and EOP in-flight conditions. In this way, the EOPs, from the Direct Sensor Orientation, are refined using the root mean square errors (RMSE) of orientation angles (ω, φ, χ) computed in the BBA. Additionally, direct correlations among some IOP and EOP parameters can absorb some inaccuracies connected to the direct determination of image position Xs, Ys and Zs.
Two sets of orientation angles refinements were estimated from the BBA. For the images taken in the Northeast flight direction, the values of refinement are 18, 43 and 205 arc seconds, respectively for Omega, Phi and Kappa and for the images taken in the Southwest flight direction, the values of refinement are 25, 2 and 90 arc seconds, respectively for Omega, Phi and Kappa.
Four photogrammetric models (pairs of images) are used to derive 3D object space coordinates via a single photogrammetric intersection. The photogrammetric models are inside the entire image block with different distances from the image sub-block. The images from the photogrammetric models (M01 and M02) were taken in the Northeast direction. The photogrammetric models M02 and M03 are positioned in the same strips of the sub-block. The photogrammetric model M03 is the closest model to the image sub-block. The images from the photogrammetric models (M03 and M04) were taken in the Southwest direction. Figure 1 shows the spatial distribution of the photogrammetric models inside the entire image block.
The photogrammetric models have two or three LCPs as checkpoints. The small number of LCPs in each photogrammetric model is explained by the difficulty to have a building roof with three planes in rural areas. All photogrammetric models have a set of photogrammetric tie points well distributed in the model region (close to the "Von Gruber locations"). These points are used to evaluate the precision of the photogrammetric intersection in the image space (y-parallax). Two groups of experiments are performed; the first used nominal IOP and EOPs from the Direct Sensor Orientation. The second used refined IOP and the same EOPs, refined using the root mean square errors (RMSE) of orientation angles (ω, φ, χ) computed in the BBA. In both sets of experiments, the images EOPs (position and orientation) are fixed absolutely by weight constrains and half of the pixel (0.003 mm) is adopted for measurement precisions in image coordinates. The obtained results from two sets of experiments are shown in Tables 3 and 4.

Refined IOP and EOPs M01
RMSE_y 22  The precisions analyses of the obtained results from the experiments of photogrammetric intersection are performed through the inspection of the values of the root mean square errors of y-parallax, reported in Table 3. Considering these values, it can be concluded that, using refined IOP and EOPs, the y-parallaxes are reduced significantly in all photogrammetric models. Although only four models were used in the experiments, the obtained results indicate different precisions for two groups of photogrammetric models. The lowest y-parallax values were obtained in two photogrammetric models (M02 and M03) close to the images sub-block.
Although the values of y-parallax have increased in two photogrammetric models farthest from the image sub-block (M01 and M04), these values of y-parallax can be acceptable (close to 1.5 pixels) in most photogrammetric approaches.
The values of the RMSE of the horizontal and vertical discrepancies from the LCPs checkpoints within each photogrammetric model are used to evaluate the obtained accuracies of the 3D photogrammetric intersection experiments. From the inspection of these values, reported in Table 4, it can be concluded that the horizontal and vertical accuracies improved in all photogrammetric models when refined IOP and EOPs were used to perform the experiments. Due to the large difference in the focal length (nominal versus refined), the improvement in vertical accuracy is notable (close to 75%). A close look at the RMSEs of the horizontal discrepancies in Table 4, it can be verified a similar behavior to y-parallax values; better horizontal accuracies were obtained in two photogrammetric models close to image sub-block (M02 and M03).
In this study, the expected vertical accuracy for a single photogrammetric intersection on the object space is estimated based on the average flight height (1600 m), the average baseline (452 m), the image pixel size (0.006 mm), and the focal length (100.4 mm). Using the mathematic equation, reported by Alamús and Kornus (2008), the expected vertical accuracy is computed. Its value is close to 48 cm. For the expected horizontal accuracy, it was assumed the value of two times the Lidar horizontal accuracy (40 cm), considering what was mentioned by Vosselman and Maas, (2010); the horizontal accuracy of a control point, extracted from Lidar dataset, can vary from 0.1 to 1.0 meter. In the performed experiments, using refined IOP and EOPs, most of the vertical discrepancies, reported in Figure 7, and the mean RMSE value of vertical discrepancies is smaller than the expected vertical accuracy. Additionally, the mean RMSE value of horizontal discrepancies, reported in Table 4, is close to the value of expected horizontal accuracy. Considering the values of expected accuracies and the RMSEs values, reported in Table  4, it can conclude that the experiments achieved satisfactory horizontal and vertical accuracies. Figure 6. Shows a graphic of tridimensional discrepancies of the LCPs from the 3D photogrammetric intersection using unrefined IOP and EOPs. Figure 7. Shows a graphic of tridimensional discrepancies of the LCPs from the 3D photogrammetric intersection using refined IOP and EOPs.

Unrefined IOP and EOPS
A close look at Figures 6 and 7 it can be concluded that the better tridimensional accuracy was achieved when the IOP and EOPs were refined by the proposed approach. Although the horizontal discrepancies (DX and DY), in both experiments, have similar behavior, better horizontal accuracy was achieved when the refined IOP and EOPs were used. Due to the large difference in focal length in two sets of IOP, large variability in vertical discrepancies (DZ) is observed in the performed experiments. However, expected vertical accuracy was achieved when the refined IOP and EOPs were used.

CONCLUSION AND FUTURE WORK
The paper showed an empirical study that was performed to improve the tridimensional accuracies of the integration of aerial imagery and Lidar datasets by the 3D photogrammetric intersection of single models (pairs of images) using EOP measurements from Direct Sensor Orientation. The study proposed an approach to refine IOP and EOPs in the job condition. In this way, a Bundle Adjustment with additional parameters was performed using a small sub-block of images extracted from the main image block. The IOPS and EOPs are refined via the obtained results from the BBA.
Two groups of experiments of the 3D photogrammetric intersection were performed. In the first, unrefined IOP and EOPs were used and in the second, the IOP and EOP were refined by proposed approach. From the obtained results of the performed experiments, the main conclusions are drawn: (a) 3D photogrammetric intersection, using images' EOP (position and orientation), estimated from direct sensor orientation, was applied to integrate the imagery and Lidar datasets. Using unrefined IOP and EOPs, the vertical accuracy didn't reach an acceptable result. The mean vertical RMSE of vertical discrepancies from Lidar checkpoints is near to 1.5 meters and the mean values of yparallaxes in "Von Gruber location" is close to three image pixel. On the other hand, when the IOP and EOP are refined by the proposed approach, the horizontal and vertical accuracies were improved significantly.
Considering the obtained values of root mean square errors of the Lidar checkpoint discrepancies in the performed experiments, the horizontal and vertical accuracies increased around 25% and 75% the respectively; (b) Better horizontal accuracy was achieved in two photogrammetric models near to images sub-block used to perform the Bundle Adjustment with additional parameters. In these models, the mean Root Mean Square Error of y-parallaxes is close to half image pixel. In the other two photogrammetric models, far away from the images sub-block, the values of y-parallaxes in "Von Gruber location" are close to 1.5 image pixels; (c) In the 3D photogrammetric intersection of single models (pairs of images), using unrefined IOP and EOPs, the EOPs were estimated from direct sensor orientation and their values were weighted in the BBA according to the nominal precisions of the IMU (GNSS/INS integration); the IOP is extracted from the camera certificate; (d) The 3D photogrammetric intersection achieved better precisions (y-parallaxes) and 3D accuracies when IOP and EOPs are refined by the proposed approach and the refined EOPs are weighted according to the empirical precisions; (e) The Bundle Adjustment with additional parameters, using a small-sub block of images and three LCPs, achieved acceptable precisions; the obtained results from this BBA were used to get IOP and EOPs under flight conditions, required to increase the accuracies of the performed experiments of the 3D photogrammetric intersection.
Future works will continue to investigate whether the spatial position of photogrammetric models, inside of the entire block, can change the performance of the proposed approach to increase the 3D accuracies of the integration of imagery and Lidar datasets by 3D photogrammetric intersection.

ACKNOWLEDGMENTS
We would like to thank the two Brazilian governmental agencies CNPq (The National Council for Scientific and Technologic Development) and CAPES (The Coordinating Agency for Advanced Training of High-Level Personnel) for their financial support of this research and TOPOCART (Surveying Engineering and Aero Mapping Company in Brazil) to provide the necessaries material to develop this study.