EXTRACTION OF SOLAR CELLS FROM UAV-BASED THERMAL IMAGE SEQUENCES

This paper discusses the automatic generation of thermal infrared ortho image mosaics and the extraction of solar cells from these ortho image mosaics. Image sequences are recorded by a thermal infrared (TIR) camera mounted on a remotely piloted aerial system (RPAS). The image block is relatively oriented doing a bundle block adjustment and transferred to a local coordinate system using ground control points. The resulting ortho image mosaic is searched for solar cells. A library of templates of solar cells from thermal images is used to learn an implicit shape model. The extraction of the single solar cells is done by estimating corners and centre points of cells using these shape models in a Markov-Chain-Monte-Carlo algorithm by combining four corners and a centre point. As for the limited geometric resolution and radiometric contrast, most of the cells are not directly detected. An iterative process based on the knowledge of the regular grid structure of a solar cell installation is used to predict further cells and verify their existence by repeating the corner extraction and grammar combination. Results show that this work flow is able to detect most of the solar cells under the condition that the cells have a more or less common radiometric behaviour and no reflections i.e. from the sun occur. The cells need a rectangular shape and have the same orientation so that the model of the grammar is applicable to the solar cells.


INTRODUCTION
Today, automatic 3D reconstruction and texture extraction is focused on high resolution images and image sequences from the visual spectrum.These methods are mainly using homologous points to link the images in a relative orientation and extract 3D coordinates for the homologous points (Hartley and Zisserman, 2004).Some of these methods also include a self-calibration for the camera (Agarwal et al., 2009;Longuet-Higgins, 1981;Maybank, 1993;Mayer et al., 2012).The resulting camera orientations and 3D points have to be transferred from the model coordinate system to the global coordinate system either using external GPS/INS orientation information, a matching with given 3D building models, or a coregistration based on ground control points.
To find a good set of homologous points and remove outliers, the RANSAC algorithm (Fischler and Bolles, 1981) is used in most solutions, where random minimal sets of homologous points are used to estimate a relative orientation, that is checked against the other homologous points.
Most of the homologous points in solar parks are located either on flat roofs or on flat terrain.This has a significant influence on the quality of the relative orientation as the reconstruction of the interior orientation is not possible from a planar scene (Maybank, 1993).For planar scenes the homography is an alternative way of orienting image pairs (Hartley and Zisserman, 2004;Pollefeys et al., 2002).
Corrected relative orientations are used as input for dense matching techniques to generate a 3D point cloud not only for the tracked features used in the bundle adjustment, but for most of the pixels of the input images with correspondences in several images (Hirschmueller, 2008).Several works have used this two-step 3D point cloud extraction for image sets from RPAS mounted RGB * Corresponding author cameras (Mayer et al., 2012;Tuttas et al., 2015) and thermal cameras (Hoegner et al., 2016).
There is quite a limited number of works on transferring methods for 3D reconstruction to the thermal infrared domain.A geometric calibration including principal point, focal length, and radial distortion parameters has been investigated by some groups (Luhmann et al., 2010;Laguela et al., 2011).3D reconstruction and texture extraction in thermal infrared are applied for sets of images and ordered terrestrial image sequences (Hoegner and Stilla, 2015) or image sequences taken by a thermal camera mounted on a RPAS (Westfeld et al., 2015;Hoegner and Stilla, 2018).Both 3D reconstruction and texturing are influenced by various conditions as the thermal radiation of façades depends on temperature differences between inside and outside, weather conditions, and materials.To overcome limitations in the 3D accuracy of thermal infrared based 3D points, a combination of thermal infrared cameras and 3D recording systems like laserscanners (Borrmann et al., 2013) and time-of-flight cameras (Hoegner et al., 2014) or photogrammetric point clouds (Hoegner et al., 2016;Hoegner and Stilla, 2018) is possible.
Terrestrial images or images taken from a mobile mapping system (Hoegner et al., 2007;Reznik and Mayer, 2008) can be used for documentation of frontal faces visible from the street level to generate façade textures for a given 3D model from oriented fused image sequences.Whereas in (Reznik and Mayer, 2008) RGB images are used for a Markov-Chain-Monte-Carlo approach for window extraction, first attempts for window extraction from thermal infrared façade textures have been done using grammars like Gestalt system (Michaelsen et al., 2012).This papers is focussed on the automatic extraction of solar cells in solar parks from thermal infrared (TIR) images acquired by a remotely piloted aerial system (RPAS).In large-scale industrial applications like the observation of solar parks, RPAS-mounted thermal cameras are used to inspect the solar cells.The quite big areas need a recording as video streams to achieve a quite fast flight speed for covering large areas in short time.So, no external orientation is recorded for every image but images show a big overlap an thus can be quite easily relatively oriented in a bundle adjustment.Knowing the exterior orientation of a few images and having a couple of ground control points is enough for orientation in general.For solar parks, most of the images show a more or less repetitive pattern of solar cells ordered in rows and columns.This increases the number of mismatches in the automatic detection of homologous points.As the temperature of a solar cell depends on the transfer of solar energy to power, defects in solar panels can be seen in hot cells.Today's observation of these effects is done mainly manually by checking recorded images of solar parks and writing down the manually detected defects.A transfer to geo-referenced ortho images would allow the localisation of solar cells and the analysis of possible defects using automatic image processing techniques.

METHODOLOGY
The methodology is split into two parts (Fig. 1) At first, an ortho image has to be generated from the TIR image sequence (Chap.2.1).The proposed method is designed to work with a minimum set of pre-knowledge.It is assumed that the exterior orientation parameters of the recorded images are unknown but that the images are recorded in an image sequence with several frames per second and with a GPS reference every second as initial estimation.If so, we can assume that neighbouring images in the sequence only have slightly different exterior orientation parameters, corresponding homologous image feature points show almost the same radiometric profile and have only a small offset in the two images.It is also assumed that there is no geometric calibration of the camera and the interior orientation parameters and distortion parameters are only given as initial estimates for the bundle adjustment.It is further assumed that a small set of ground control points exists that is derived from either from direct measurements or given 3D geometry like i.e. 3D building information.
The second part deals with the extraction of the solar cells from the ortho image (Chap.2.2 and 2.3).The extraction of the solar cells is based on the assumption that cells have a specific shape and radiometry so that they can be found using a library of solar cells for matching.This library is used to train an implicit shape model of a cell consisting of four corners and a centre point.Possible cells are found by using the library in a Markov-Chain Monte-Carlo approach as proposed for window extraction by (Reznik and Mayer, 2008).It is assumed that radiometric difference lead to undetected cells and that cells are ordered in rows and columns so that the used grammar can guess positions of missing solar cells and adopt the extraction process.

Ortho image generation
The ortho image generation follows the standard work flow as proposed by Förstner and Wrobel (2016) and consists of a combined forward backward propagation including camera self calibration.Feature points are detected generating a gradient descriptor (Förstner and Gülch, 1987).If we assume the image sequence is recorded with several frames per second, the changes in the image content are small enough to use simple correlation to find corresponding homologous feature points in neighbouring images.The features are then tracked through the sequence.The ground control points are manually marked in a set of images.Ground control points do not necessarily have to be visible in the whole image sequence and not all ground control points have to be visible in one image.Image coordinates of the homologous points are used as observations in the bundle adjustment.Their object coordinates are unknowns as well as the interior and exterior orientations.For the interior orientation excluding the distortion coefficients, the camera specifications are used as initial estimates.The distortion is modelled with five parameters: three radial symmetric distortion coefficients and two tangential distortion coefficients.The ground control points are included as observations and virtual unknowns.The global coordinate system is defined by X pointing east, Y pointing north, and Z being the height.
The following dense matching ( (Hirschmueller, 2008)) generates a dense 3D point cloud that is used to interpolate the grid cells of the digital elevation model for the ortho image generation.Due to the low ground sampling distance of TIR camera images (2) it is assumed that the accuracy and density of the 3D points is sufficient for a surface plane estimation.Nevertheless, the limited ground sampling distance and resulting point density and accuracy will not allow the extract single solar cells from a 3D point cloud generated from TIR images.

Extraction of single solar cells
The solar cell extraction is based on the approach of Reznik and Mayer (2008) originally designed for window extraction from RGB façade textures.It is assumed that the coordinate systems of the ortho image and the templates are defined parallel to the axes of the solar cells.In that case vertical edges of the cells follow the y-axis and horizontal edges of solar cells follow the x-axis.This assumption is true if the local coordinate system is defined by ground control points where their coordinate system axes are parallel to the vertical and horizontal edges of the rectangular solar cells.At first, a training data set of solar cells has to be generated to learn the specific structures of the solar cells, in this case the four corners of the cell rectangle.For that, a number of representative solar cells is cut out including a local neighbourhood from the generated TIR ortho images.These template cells are used to extract the features for the implicit shape model.
In the templates, Förstner points (Förstner and Gülch, 1987) are detected and the points inside the cells are kept, points in the neighbourhood are rejected.For the template cells, the dimensions are known and so the centre point is easy to calculate.For all Förstner points inside the cell, the distances to the centre point are calculated.These distances are stored as description of cells and the mean distance of all corners is calculated for later centre point estimations.The process is visualised for one template cell in figure 2  In the next step, Förstner points are detected in the whole ortho image.For every Förstner point in the ortho image, a similarity using cross-correlation with the Förstner points of the templates is calculated.Förstner points are accepted as possible corner points of the correlation coefficient with at least one corner point of one template reaches 95%.After the correlation, it is estimated which Förstner point is an upper corner or a lower corner by calculating the gradient direction for the points.As it is assumed that solar cells appear colder than the surrounding, the gradient has to point away from the cell.That means that for upper corners the gradient points up and for lower corners the gradient points down.In fact, many more Förstner points are detected than real corners of solar cells.
So far, possible corner points have been detected and classified as upper or lower corner points.In the next step, corner have to be combined to possible rectangles.A Marcov-Chain.Monte-Carlo approach is used to combine upper and lower corners to rectangles and to calculate their centre points.This is done by estimating possible centre points for a set of four corner points.
It is assumed for an optimal solar cell that the centre point of four corners -two upper corners and two lower corners -is the intersection of the four circles around the corner points with the mean distance of corner points calculated from the templates.The standard deviation of the distances is used to define thresholds for the minimum and maximum distance of a corner point to its centre point.This leads to minimum and maximum radius for areas of interest around a corner point where possible centre points can be located.As upper corners must be above the centre point, their centre point is only searched for in the half circle below the points and because lower corners have to be below the centre point their centre point is only searched for in their upper half circle.The Markov-Chain-Monte-Carlo approach estimates the most appropriate centre points to fit in a rectangular grid and the estimated optimal distances of corner points from the templates.
Beginning with a first upper corner point that is assumed to be a left upper corner, possible candidates for right upper corner and the lower corners are searched.A corresponding upper right corner has to fulfil two conditions: first, the y value value must not differ more than a given threshold as it should be in a horizontal line with the upper left point; second, the half circles of both points have to intersect.For the lower left point, in addition to the intersecting half circles with both upper corners, the x value must not differ more than a given threshold as the upper and lower left corner should be on a vertical line.The lower right point at the end has to intersect all three other half circles with its half circle and must not differ more than the given threshold in y and x value from the upper right and lower left corner respectively.If four corner points fulfil all these conditions, there centre point is calculated from the mean half distances of the x and y values.

Iterative process to add further cells
With this basic concept, only rectangular solar cells are found where all four corners have been correctly detected and combined.This is only the case for a small sub group of all existing rectangular cells.Further cells are added in an iterative process.
Possible parameters for the distances of corner points are derived from the already extracted cells.Four different cases have to be distinguished: In case 1 (fig.3), a closing rectangle has to be found.It is assumed that from the previous iteration both neighbouring cells in the row are already detected.In that case, the four corners already exist, but a rectangle is not generated because the four half circle intersections for possible centre points (yellow) differ to much.In the figure, the possible centre point of the two left corners is visualised because these two corners are used to find the corrsponding right pair of corners.To detect such a rectangle, it is necessary to check for existing cells, if the right or left pair of corners could share a centre point with another pair of corners that already belongs to a cell.The calculation of the centre point is done as described in chapter 2.2.In case 2 (fig.4), two corner points are already assigned to a rectangle and a possible centre point is given from the intersection of the half circles (yellow).Within the conditions defined in case 1, there exist two other extracted Förstner points that have not been set as corners so far.If these two points fulfil the conditions of case one, they are added to the new rectangle.In case 3 (fig.5), two corner points are already assigned to a rectangle and a possible centre point is given from the intersection of the half circles (yellow) where one Förstner point exists that fulfils the conditions of case 1 and is no corner point so far.The fourth necessary point is missing.In that case, the fourth point is estimated from the x and y coordinates and added as virtual corner point.In case 4 (fig.6), only two corner points from an existing rectangle are given and a possible centre point is given from the intersection of the half circles (yellow) where there is no Förstner point that fulfils the conditions defined in case 1.In that case, it is assumed that the rectangle to detect has the same dimensions than the rectangle where the two existing corners belong to and there x and y values are estimated from the existing neighbouring rectangle.
To stop that iterative process at the end of a row of cells, for added new corner points that have not been detected as Förstner points before, a radiometric check is done.These points are compared using a cross-correlation to corner points of the template cells and are rejected if the value is below 95%.It has to be mentioned that this iterative process is not able to find solar cells if no cell has been detected in a row at all in the beginning.To add these cells, a grammar is necessary that estimates if a further row of cells exists.To do this, the distance of detected rows is used to predict the position of another row.Corner coordinates of an existing detected cell are used to calculate x and y values for a cell in the new row.For these virtual corners, the cross-correlation to the template corners is calculated.If the value is at least 95%, the points are kept as initial cell in that row and the iteration starts again in that cell.

Dataset
The image sequence was acquired with an Optris PI Light Weight thermal infrared camera using a bolometer sensor at 30 frames per second.The images have a size of 382 x 288 pixels with a radiometric resolution of 0.04 K, stored as radiometric intensities with 16 bit integer.The UAV recorded the image sequence manually controlled by a pilot.The mean flight height above ground was 50 meters.Recordings were done in an oblique forward view to minimized direct reflection from the sun.Thus, the ground sampling distance in one image differs from approximately 8 pixels per meters to 12 pixels per meter.The full sequence consists of 5200 images.The recorded scene is a solar facility on top of a roof of a gym.The scene was recorded by daytime on a sunny day.Trees in the east shadow small parts of the roof.The roof itself is not totally flat but slightly convex.The scene was recorded in an oblique view to have a nadir view on the tilted surface of the solar panels.Figure 7 shows a single image out of the sequence including the shadow.One can see the distortions especially at the roof edge.

Ortho image generation
Figure 8 shows the resulting dense point clouds for the TIR images.One can see the convex roof and the solar cells on top.On the right, there are points belonging to the trees and in the front there are points belonging to other roof parts of the gym and the ground.Table 1 and 2 show the results of the bundle adjustment and self calibration.One can see that the mean correction of the observed exterior orientation parameters is within 0.013 and 0.019 meter and the rotation correction between 0.025 and 0.039 degrees with a standard deviation of 0.003 meters and 0.004 degrees.The estimated parameters for the interior orientation that were set as unknown here show a standard deviation of 0.003 to 0.007 meters.Looking at the estimated 3D points in the bundle adjustment (Tab.3), it can be seen that the reprojection error is approximate 0.2 pixels.The mean reprojection error for the ground control points is 1.46 pixels and the RMS of the 3D coordinates of the ground control points is 0.268 meters in X, 0.529 meters in Y, and 0.969 meters in Z.This lack of accuracy of the ground control points is a result of the measurement with a low cost GNSS receiver as the points are mainly used to orient the ortho image according to the solar panels.

Solar cell detection
In the ortho image (fig.9) five representative cells are taken as templates (fig.2) and the statistics are calculated.In the given example, the template images are selected with a size of 46 x 34 pixels.
Next, Förstner points are detected in the ortho image (fig.10).The resulting Förstner points are grouped in upper (red) and lower (green) points that are similar to corner points of the templates (fig.11a).Possible centre points are all intersections of the half circles of at least two Förstner points.From these points, centre points are estimated by the mean value of all possible centre points between corresponding corner points.The Markov-Chain-Monte-Carlo optimazation reduces the amount of possible cells to the maximum set that fulfils the same conditions of width and height and fits to a regular grid in rows and columns.As a result an optimized set of upper and lower corner points and centre points is generated.Figure 11b shows a sub image of the ortho image mosaic with the extracted corner points and estimated centre points.It can be seen that not all chosen Förstner points are correct corner points and that not all corner points have been detected.Table 5 shows the completeness and correctness of upper and lower corners and centre points for the whole data set, where a corner that belongs to two neighbouring cells is counted only once.One can see that for the upper corners completeness and correctness are around 70% whereas for the lower corners correctness is up to 90% but only half of the corners are detected (57%).Nevertheless, the search strategy finds 80% of the centre points directly and has only a few false detection (correctness 93%).number of points correctness completeness upper corners 431 70% 73% lower corners 431 93% 57% centre points 375 92% 80% Table 5. Completeness and correctness for upper and lower corners and centre points in the qhole data set.
In the first cell search, only cells with four extracted corners and a centre point are detected.The result can be seen in figure 12a.One can see that only a small number of the centre points and corners visible in figure 10 have been detected as cell so far.After iterations with case 1 to case 4, the cells in figure 12b are detected.In total, after the iteration, around 78% of the solar cells are detected with corners and centre point.
Comparing the result of the cell extraction to a cross correlation approach (Fig. 13) one can see that the correlation of the template images with the ortho image is able to easily detect the solar panel rows.However, due to the low contrast changes, it is not possible to define a global threshold for the correlation coefficient to detect single panels.
Figure 13.Searching for the template cell in the ortho image.White pixels are marked as possible good correlation with a correlation coefficient > 0.5.

DISCUSSION AND OUTLOOK
The results show that the used method is suitable for solar panel detection.The main reasons for missed cells are inaccurate corner detections.A possible improvement could be to change the area of interest for the corners in the templates (fig.2).Using an area of interest that fits more to the shape of the cells, the number of Förstner points that are wrongly seen as corner points could be reduced.On the other hand, a smaller and more specific area of interest is more sensitive to changes in the dimensions of the cells or distortions if the cells have a strong slope against the surface plane.This could lead to missing corners for cells and reduce the number of detected cells.An ellipsoid area of interest seems to be a good solution for further refinement.In general, the limited geometric resolution of the thermal images and the resulting ortho image mosaic limits the accuracy and the possibility to find small radiometric features that belong to corners.Furthermore, the training cells do not cover all possible radiometric appearances of solar cells especially cells in the shadow of the trees are not found at all.
The pre-knowledge of the regular structure can be integrated in a more prominent way.If several cells in a row have been detected, it is possible to calculate the mean size of these cells and predict further cells candidates.These candidates can then be verified by adopting the Förstner point search for corners.The same possibility is given, if several rows are found to predict missing rows from the distance of the known rows.In contrast to estimating rows from found panels, a cross correlation can be added as a pre-processing step.The correlation shows good results in finding solar panel rows, but fails in the detection of single panels.Knowing the rows from the correlation can further refine the search space for corners of solar panels.
In the actual results, detected cells are not corrected to have the exactly same dimensions and be a perfect rectangle.If the grid structure of rows and columns is identified, all corners of all found cells can be adjusted to form rectangles in a regular grid.
Another possibility would be to integrate 3D knowledge in the extraction of the panels.Unfortunately, the 3D point density and accuracy of the TIR image based point cloud is insuffient for that task.It should be considered to additionally record high resolution RGB images to reconstruct a dense 3D point cloud and derive and refined digital elevation model.

Figure 1 .
Figure 1.Overview of the proposed method.
: Red crosses are detected Förstner points, green rectangle is the shape of the cell, yellow squares are the local neighbourhoods of the cell corners to keep detected Förstner points.Distances are calculated for the Förstner points inside the yellow squares and visualised as blue lines.

Figure 2 .
Figure 2. Template solar cell.Red crosses are detected Förstner points.Green rectangle is the shape of the cell.Yellow squares are the local neighborhoods of the cell corners to keep detected Förstner points.Distances are calculated for the Förstner points inside the yellow squares and visualised as blue lines.

Figure 3 .
Figure 3. Iterative process case 1: a) four corners have been detected but are not combined so far.Possible centre points marked in yellow.b) newly added rectangle for the four existing corners with verified centre point (blue)

Figure 4 .
Figure 4. Iterative process case 2: a) two corners have been detected and two Förstner points exist as possible corners.Possible centre points marked in yellow.b) newly added rectangle with two existing and two new corner points with verified centre point (blue).

Figure 5 .
Figure 5. Iterative process case 3: a) two corners have been detected and one Förstner point exists as possible corner.Possible centre points marked in yellow.b) newly added rectangle for the two existing corners, the Förstner point and the newly generated virtual corner point with verified centre point (blue).

Figure 6 .
Figure 6.Iterative process case 4: a) twocorners have been detected but no Förstner points fulfil the conditions for further corner points.Possible centre points marked in yellow.b) newly added rectangle for the two existing corners and two estimated corner points with verified centre point (blue).

Figure 7 .
Figure 7. Image of the recorded sequence.One can see the solar panels on the roof including the shadow of the trees and the distortion of the image.

Figure 8 .
Figure 8. Point cloud generated in the bundle adjustment.One can see the convex roof and the solar cells on top.On the right, there are points belonging to the trees and in the front there are points belonging to other roof parts of the gym and the ground.X [m] Y [m] Z [m] ω [deg] φ [deg] κ [deg] Mean 0.013 0.015 0.019 0.039 0.031 0.025 Sigma 0.003 0.003 0.003 0.006 0.004 0.004

Figure 9 .
Figure 9. Ortho image mosaic generated from the image sequence and the point cloud.

Figure 12 .
Figure 12.Iterative process of cell detection: a) Initial detection of cells where all four corners and the centre point have been detected.b) Final result of detected cells after the iteration process.

Table 1
Using the image sequence and the point cloud, the ortho image mosaic is generated (fig.9).One can see the rows of solar cells are now rectified.On the right side, the shadow of the trees cov-

Table 3 .
Bundle block adjustment: 2D homologous points, 3D object points and their reprojection error, 3D ground control point (GCP) accuracy (RMS) and reprojection error.ersparts of the roof.Table4shows the density of the generated surface model (DSM) and the ortho image mosaic.

Table 4 .
Density of 3D point cloud, derived digital surface model (DSM), and resulting ortho image mosaic.