EVALUATION OF THE POTENTIAL OF AERIAL THERMAL IMAGERY TO GENERATE 3D POINT CLOUDS

: This research evaluates the ability of thermal images obtained from aerial platforms to produce 3D point clouds. In this study, the thermal camera is first calibrated. Then, in order to avoid data redundancy, the key frames of the obtained thermal video are separated from other frames. Afterwards, the point clouds are generated and then the thermal ortho image is created from the key frames. The evaluation is done using visible orthophoto, ground control points and the linearity of the edges of buildings extracted from thermal images. The results of this study show that the thermal ortho image matches the visible ortho image with a good accuracy in the study area. Moreover, the standard deviation of the edges of the buildings has been calculated for a number of reconstructed buildings in thermal ortho with proper dispersion. 77% of the measurements taken from the edges of the buildings coincide with a straight line with an accuracy of better than two pixels, and about half of these values are extracted with an accuracy of better than a pixel.


INTRODUCTION
3D reconstruction refers to the process of obtaining the shape and structure of an object through the processing of photos or videos captured from that object.The reconstruction of various urban features such as buildings, or natural features in the form of 3D models has been studied in sciences such as computer vision and photogrammetry for a long time.Nowadays, the emergence of digital cameras with high resolution has leaded many researchers to conduct their studies in the field of automatic imaging systems.Visible cameras are known as standard imaging tools; however, they have limitations when utilized.The storage of colors and the ability to see objects and features requires an energy source such as the sun or an artificial light source.Therefore, the main problem when working with visible cameras is that the quality of the images taken depends on the brightness of the environment; the intensity of this brightness, its direction, the balance in colors and such factors can be challenging.Moreover, no image can be taken in complete darkness.To solve imaging problems with visible systems and also to add other information to the captured images, other sensors have been introduced.Thermal cameras are passive sensors that can capture the infrared rays emitted by all objects with a temperature above absolute zero and do not require any external energy source.This type of camera was initially introduced as a tool for surveillance and night vision and for military applications (Gade and Moeslund, 2014).Over time and with the commercialization of their technology, cheaper thermal cameras were introduced for civilian applications.Today, aerial thermal images obtained from unmanned platforms are used in many fields: from managing water resources and forest health monitoring to discovering and tracking objects on the surface of the oceans and discovering and tracking pedestrians and vehicles, building energy modeling and similar applications (Leira et al., 2015, Smigaj et al., 2015, Ma et al., 2016, Cho et al., 2015).Furthermore, fusing thermal and visible imageries has been investigated in order to precise 3D extraction of building roofs (Dahaghin et al., 2021).In many of these applications, thermal data have been obtained with the help of unmanned aerial platforms.For example, thermal and multispectral remote sensing using unmanned aerial platforms have been used to monitor and investigate vegetation in research (Berni et al., 2009).Pedestrian detection and tracking in low-resolution thermal images in (Ma et al., 2016) is another application of thermal data from unmanned platforms.In the German Space Agency (Pless et al., 2012), the production of orthophoto and 3D surface model has been done using thermal imaging.In this research, two thermal cameras with cooling detectors were used and the calibration of these cameras was taken into consideration.In the research (Pech et al., 2013) multi-temporal thermal orthophoto has been produced for spatial analysis of temperature distribution.Another research (Lagüela et al., 2014) has produced a 3D model of buildings in which geometric and thermal information are used together; in this research, oblique thermal images are used.In the research (Mouats et al., 2015), stereo thermal cameras were used for visual odometry.Moreover, 3D reconstruction with the help of these images has been investigated and the performance similar to the visible band has been achieved.Considering the importance and application of thermal data, as well as the importance of 3D modeling in photogrammetry, this research will evaluate the ability of thermal images obtained from aerial platforms to produce 3D point clouds.In fact, at first aerial thermal videos are taken and their key frames are processed to produce a 3D model.

DATA AND STUDY AREA
The study area of this research is Charmshahr in the suburbs of Tehran.This area was chosen in such a way that it includes vegetation, buildings, roads, cars and various features in order to evaluate the ability of thermal images to reconstruct each of these types.An image of the location of this area is shown in Fig. 1.

Figure 1. Study area located in Charmshahr
The sensor used in this research is a thermal camera that can be installed on the UAV system.In this research, thermal video was taken from the target area.The camera is installed transversely and perpendicularly to the flight path of the drone to provide the required transverse coverage.The important characteristics of this thermal camera are shown in Table 1.

Parameters Values
Thermal resolution 640*480 pixels Table 1.Main characteristics of the used thermal camera The data obtained from the study area is in the form of strips of consecutive frames.Some of these images are shown in Fig. 2.

METHODOLOGY
The proposed method for evaluating the ability of UAV-based thermal images to generate 3D point clouds is shown in the flowchart of Figure 3.

Calibration of Thermal Camera
The designed calibration pattern is rectangular in dimensions of 24 x 44 cm2, including 28 circles with 20 mm diameters that are separated by 50 mm intervals.Moreover, there are 4 circles in the form of coded targets in order to determine the justification of the calibration pattern.This screen is made of flex material so that, in addition to being light, it can retain heat for the duration of the image acquisition.To create circles in it, the laser cutter is also used.Figure 4 shows the image of the designed calibration pattern.Imaging with a thermal camera was done while the desired pattern was placed on a bright monitor.In order to achieve the features of calibration, the centers of the circles should be extracted.Since the image of circles is recorded as an ellipse due to lens distortions, in this section the ellipse extraction algorithm is implemented based on the duality property of ellipses in (Ouellet and Hébert, 2009).The results of the implementation of this method can be seen in Fig. 5.

Figure 5. An example of extracting calibration features (centers of circular targets) for a thermal camera
In this research, the mathematical basis of the calibration process is the use of the central aperture camera model, and the camera distortion parameters are also taken into account.This recommended calibration pattern works well for the calibration of this camera.

Keyframe extraction from thermal videos
In order to extract key frames, bluM criteria is first applied to all frames (Crete et al., 2007).The threshold for this criterion is 0.5.This means that frames with bluM greater than 0.5 are considered blurry and are removed from the candidate frames.
In the following, the criteria related to appropriate overlap between frames, appropriate baseline and GRIC criteria (Ahmed, 2009) are applied on the candidate frames.In this study, 2135 key frames are extracted.

Initial 3D reconstruction
In this part, the approximate external orientation elements are calculated for all frames that have cumulative errors and scaling between transitions.Furthermore, the presence of radial distortion in the camera causes errors in 3D reconstruction.
After calculating the initial elements, the process of calculating the final elements and extracting the ortho image is done on the data, the results of which are stated in the next section.

Generating dense point cloud from aerial thermal images
In order to perform dense matching, the semi-global matching algorithm is used (Hirschmuller, 2005).In this method, image matching is done on a pixel scale with good accuracy by optimizing the path based on the global cost function.Moreover, in this method, several baselines can be used and the information related to the point clouds of a region can be combined and as a result point cloud with more accuracy and better quality is prepared.The execution time of this method is lower than other methods and therefore it is more suitable for online applications.The evaluations performed on this method show that SGM is a stable method against radiometric changes that have been occured in the images for various reasons (Mohammadi and Samadzadegan, 2013).All the mentioned advantages make this algorithm useful for matching in this study.Stereo matching algorithms usually consist of four stages: calculation of matching cost, combination of matching costs, depth image calculation (solving of optimization problem) and depth image correction.The two-dimensional energy function used in the semi-global matching algorithm is shown in Equation ( 1): where p is the position of the pixel, D is its parallax and c(p.Dp) is the cost for that pixel P1 and P2 are respectively penalty terms for costs whose parallax value differs from the parallax of the central pixel by one unit or more than one unit.T is also a logical operator whose value is 1 for accuracy and zero otherwise.The ortho image generated with the help of this algorithm can be seen in Fig. 6.

Evaluation Method
In order to quantitatively evaluate the results, two different methods have been used.
• The first method: In this method, with the help of existing thermal and visible orthophotos, a number of sharp and specific points have been selected as control points and check points.Fig. 6 shows the position of control points and check points in the entire region.15 points were used as control points and 15 points were used as check points.
In order to create a connection between the control points in the thermal and visible ortho image, the coefficients of a transformation equation should be obtained with the help of the control points.In this study, the best equation that can establish this relationship is a quadratic rational equation.In our tests, projective equations, first degree polynomial and second-degree polynomial have also been tested and it was observed that the second-degree rational equation models the relationship between two ortho photos better than other equations.The equation is shown in relation (2).
where, X and Y correspond to ground coordinates or visible image coordinates and x and y correspond to thermal image coordinates.By placing the coordinates of the check points on the left side of these equations, the calculated X and Y values are obtained.These calculated coordinates are compared with the measured coordinates from the check points and the accuracy is estimated using the RMSE and STD statistical criteria: 2 1 ( ) In Equations ( 3) and ( 4), dri is the error related to the check point and μ is the average error of all check points.
• The second method: A number of the sides of the buildings in the thermal ortho photo have been selected and the accuracy of the points of these lines has been estimated.Doing this test, we can understand how well the lines in the thermal ortho photo are fitted.On each of the sides of the buildings five points are selected and a line is fitted to these points using the least squares method.Then, the distance of each of these points from the fitted line is calculated.Therefore, for each line, there will be five values of distances from the line.The standard deviation of these values is considered as the accuracy of line fitting.The smaller the standard deviation, it indicates that the side of the building was more similar to a straight line.

RESULTS
Table 2 shows the error values in the ground space for each of the check points.Table 3.The error values in the image space for each of the points According to the Table 3, RMSE and STD values are calculated as 3.9666 and 1.6206 pixels, respectively.In the second test, 35 lines are selected from 17 different buildings and the linearity of the edges is calculated.Fig. 7 shows a number of buildings selected for this test.Moreover, the value of the standard deviation for line number 1 shows that this line is well reconstructed (Figure 9).77% of the measurements taken from the edges of the buildings coincide with a straight line with an accuracy of better than 2 pixels, and about half of these values are extracted with an accuracy of better than a pixel.

CONCLUSION
Due to the application of thermal sensor in darkness and unfavorable weather conditions such as fog and dust, it is beneficial in photogrammetry processes in order to produce three-dimensional information.Furthermore, use of unmanned aerial platforms due to their lower cost, the ability to work in inaccessible or dangerous areas, along with control devices and the ability to install various sensors and cameras on them are suitable for many applications.The results of the investigations show that in this research, the thermal orthophoto with RMSE and STD values of 0.1608 and 0.0542 meters respectively in the ground space is consistent with the visible orthophoto.The values of RMSE and STD in the image space are also calculated and are equal to 3.9666 and 1.6206 pixels, respectively.Moreover, 77 percent of the measurements taken from the edges of the buildings coincide with a straight line with an accuracy of better than 2 pixels, and about half of this value was extracted with an accuracy of better than a pixel.Thermal data, despite its limitations related to the lack of texture information and the challenges it causes in various fields such as calibration, removing extra frames, dense matching and other steps, can provide information to obtain a relatively suitable threedimensional image of the areas that were photographed in unfavorable lighting or atmospheric conditions as well as being an information source in addition to visible data to help the photogrammetric process of 3D modeling.

Figure 2 .Figure 3 .
Figure 2. Some of the frames taken from UAV platforms

Figure 4 .
Figure 4.The imaging network for calibrating the thermal camera Several images were taken from different stations and a photo was taken at each station by rotating the camera 90 degrees.Imaging with a thermal camera was done while the desired pattern was placed on a bright monitor.In order to achieve the features of calibration, the centers of the circles should be extracted.Since the image of circles is recorded as an ellipse due to lens distortions, in this section the ellipse extraction algorithm is implemented based on the duality property of ellipses in(Ouellet and Hébert, 2009).The results of the implementation of this method can be seen in Fig.5.
a y a xy a x a y X c x c y c xy c x c y

Figure 6 .
Figure 6.Generated ortho thermal image and the position of control (red) and check (green) points in the study area

Figure 7 .
Figure 7.A number of selected buildings to check the accuracy of the extracted edges Figure 8 shows that, line number 23 is not reconstructed well according to the calculated standard deviation value.

Figure 8 .
Figure 8. Location of line number 23 (left) and standard deviation chart for line number 23 (right)

Figure 9 .
Figure 9. Location of line number 1 (left) and standard deviation chart for line number 1 (right)

Table 2 .
Error values in the ground space for each point According to Table2, RMSE and STD values are calculated as 0.1608 and 0.0542 meters, respectively.Table3shows the error values in the image space for each of the points.
Table 4 shows the standard deviation values obtained for each of the lines.

Table 4 .
Standard deviation values for the sides of buildings