RECONSTRUCTION OF RECTANGULAR WINDOWS IN MULTI-LOOKING OBLIQUE VIEW ALS DATA

With multi-looking oblique view airborne laser scanning (ALS) it is possible to create point clouds with a nearly complete 3D coverage of a larger area. This allows, in contrast to nadir view ALS, the extraction of façade information. This paper concentrates on the reconstruction of windows. Because of the limited point density, received from oblique view ALS, the approach aims at the reconstruction of rectangular windows from sparse point clouds (<10 points/m). In a pre-processing step window centres are determined. For that indoor points, which lie behind the façades planes, are detected. The following reconstruction process consists of two main steps. First the window centres are used to create a hypothesis for the window outline by searching for a rectangle with maximum size, which includes the window centre but no points of the point cloud. In the second step these outlines are represented by probability density functions to model the uncertainty of the edges. All edges of one type, i.e. left, right, upper or lower edge, are combined by multiplication of their functions. Subsequently these functions are used to allocate the final edge positions to each window. The windows can be reconstructed with a width and height error of a few decimetres, what corresponds to the typical point distance in the point cloud, as far as the window centres are provided in a sufficient quality. The approach performs better the more equal windows are arranged in a façade row or column.


INTRODUCTION 1.1 Motivation
Today most of the larger cities have got a 3D building model in LoD2, with modelled roof structures and textured façade faces.On the way to more realistic ground view representation of the buildings, façades details have to be reconstructed (LoD3).Beside a more realistic visualization for navigation or urban planning these façade elements play a role in applications like thermal inspection of buildings (Iwaszczuk et al. 2011), since the window area shows the temperature of a reflected area instead of the building temperature, or the analysis of Persistent Scatterers in radar images (Auer et al. 2010).Another large subject area, where detailed building models are used, is the field of construction site monitoring and building information models (Tang et al. 2010).Haala & Kada (2010) give an overview on works on automatic 3D building reconstruction from images as well as from laser scanning data.As shown in this paper, there are a lot of approaches for the automatic reconstruction of 3D city models from airborne nadir looking laser scanning (e.g.Verma et al. 2006, Dorninger & Pfeiffer 2008, Kada & McKinley 2009).In these cases only the roof structures are modelled since only few points of the building façades are available.Approaches for the reconstruction of façades from terrestrial laser scanning data by contrast have the disadvantage that they usually contain only small areas of a city (e.g. a single building) and/or the buildings are only seen from a certain direction (e.g. for mobile mapping systems).To overcome this lack of data there is the possibility to use multi-looking oblique view airborne laser scanning (ALS).A point cloud with nearly complete 3D coverage of a larger area, but with limited point density, can be created in that way.From this the question arises if this kind of data is sufficient to derive façade details, at least the most important ones, especially the windows.Addressing this question an approach is introduced in this paper for the automatic detection and reconstruction of rectangular windows for this class of data.

Related Work
Several works have been done on window detection or reconstruction from different kind of data.Approaches using images for window reconstruction have been proposed for example by Lee & Nevatia (2004) or Meixner & Leberl (2010).In these works edge detection is performed on the façade texture.In a first step the edge image is used to gain candidate windows by projecting the edges in horizontal and vertical direction.The resulting profiles show peaks at potential window positions.The final positions are determined by moving the hypothesized lines along their perpendicular direction and summing up the image supporting and negative evidence to find the best position.Reznik and Mayer (2008)  Because of the higher resolution used in the mentioned studies the holes at window positions are represented much better in the point cloud than in (oblique view) ALS, what motivates for the development of a window reconstruction approach for sparser data.

Overview
In Section 2 the detection of seed points for the window reconstruction is described.In the following Section 3 the reconstruction approach is introduced, which mainly consists of the generation of window hypothesis using a quadrant based search method and the subsequently refinement of the window edges.Section 4 shows some results from our test data set.Section 5 comprises conclusions and future work.

WINDOW DETECTION
The goal of the window detection is to get the window centres.The approach shortly described here is basically the same as in Tuttas & Stilla (2011): Indoor points which are lying inside the building are chosen as indicator for window locations since features used in other approaches may fail.For example long mesh triangles may also occur at a lot of other locations since there are wider point distances in the ALS point cloud.The approach is based on the assumption that laser pulses can pass through windows and are reflected inside the building.In a first step façade planes are searched by a region growing algorithm.
From these points the main plane of every façade is determined with a RANSAC algorithm.The points are then transformed into a façade own coordinate system, with x-and y-axis showing along the horizontal and vertical direction of the façade, respectively.Points lying behind the main façade plane are then determined by fitting a Gaussian function on a histogram which shows the number of points which are within a certain orthogonal distance to the main plane.The position - is chosen as threshold for the indoor points with a minimum threshold of 10 cm.Finally the indoor points are projected into the main façade plane regarding the incidence angle of the laser.
In the remaining points the windows should now be represented as holes.The indoor points are used to compute the window centres.For that a binary image is produced, which is having an entry at a certain resolution cell, if an indoor point exists at that position.The image is oversampled and smoothed.Then the entries of the image are summed up in vertical and horizontal direction what leads to two functions.The peaks in these functions are the window centre position in horizontal (x) and vertical (y) direction, respectively.Window positions are placed at every possible combination of x-and y-positions.

Quadrant Based Search for Window Edges
The goal of this procedure is the generation of window outline hypothesis in a sparse point cloud, with a known window centre.The situation is depicted in Figure 2. The crosses are the laser points, the black point in the middle is the detected seed point.Starting from the seed points all points within a vertical or horizontal distance d s are allocated to their quadrant.Each quadrant is again split into two parts dividing it by 45°, resulting in the pairs III, IIIIV, VVI and VIIVIII.The points within the parts I, IV, V and VIII determine the boundary of the window in x direction.The x coordinate from the point closest to the seed point (in x direction) defines the position of the edge in the respective quadrant.The same applies to the part II, III, VI and VII but for the y coordinate.Finally the coordinates are taken as edges which define the largest rectangle within the edges of all quadrants (grey area in Figure 2).The division by an angle of 45° is not ideal if the windows are not square.Because of this the procedure is performed iteratively.The corner points of iteration i (x j , y j ) define the direction of the line which divides the quadrant in the next step and the edges are determined again as described above.If the corner point coordinates do not change anymore the iteration stops.This process is sketched in Figure 3 for the parts VII and VIII from Figure 2.  The PDF shall regard that it is more likely that the boundary is closer to the seed points, because of possibly missing points below, above, right or left to the window.Based on these requirements the Rayleigh distribution (Eq. 1) was chosen.

Determination of the Window Outlines
The PDF is calculated for every edge.Then the functions have to be combined for all windows.This is done always for the right, the left, the upper and the lower edges by summing up the squared functions as shown in Eq. 2.
The local maxima in that function are used as the best window edges.For every window that peak defines the edge position which has the smallest distance to the edge position from the quadrant based search.The weighting of each function is performed by the factor b. As weight the difference of the edges in adjacent quadrant parts are used, since there will be larger differences if only few points are available around the window.That means the weights for the upper edges is the difference of y 1 and y 2 , for the edge on the left the difference of x 2 and x 3 and in the same way for the lower and the right edge.

Data
We use a dataset of the test area around TU München (TUM) recorded by four overflights with a helicopter (Figure 5).The area was scanned in 45° oblique view.The co-registration of these four different point clouds can be done with homologous planes or an adapted ICP algorithm (Hebel & Stilla 2009, Hebel & Stilla 2007).The total points are around 2.5 million.

Window Reconstruction
In the following the different steps are shown for a single façade which can be seen in Figure 6.In Figure 7     Figure 10.PDFs for all upper edges (blue) shown in Figure 9 and the combined function (black) with the final edge positions (black x).

Results and Discussion
In Figure 10 the final windows for the façade described in Section 4.2 is shown.
Figure 11.Final windows for the façade shown in Figure 6.
Since there is no real ground truth data available, the following measures are used to evaluate the results of the approach.
For the façade shown in Figure 11 the size of the windows in the upper and in the middle row has been measured and compared to the size of the reconstructed windows.The results are shown in Table 1.1.06 m (1 window) Table 1.Evaluation of window size (rectangular windows) As another measure the horizontal distance between the window centres of 4.15 m is used.The deviation from this values ranges (for all rows) from -19 cm to +20 cm, with a mean of 6 mm and a standard deviation of 19 cm.The values from Table 1 indicate that the accuracy of the reconstructed window size is in the range of the point distance of 20-50 cm.It can be seen that the benefit of the use of the PDFs is mainly for the window height, since in this case there are 15 windows in a row with the same height, so the redundancy takes effect.If the outline hypothesis is too far from the other outlines the impact of the PDFs is too small and large outliers occur (3 windows with height error larger than 1 m).The width error is larger than the height error because of two aspects: On the one hand there is only a sample of three windows per column and on the other hand there are windows with different width.The PDF approach does not work in all cases for the distinction between the larger and the smaller windows.The window centres are calculated as the midposition between the right and the left edge.The horizontal distance derived from the window centres with a maximum error of 20 cm is clearly better than the width, what indicates correct window positions but a reconstruction which is too wide because of the weak representation of the outlines.2. As in the results above the window size fits in the range of a few [dm].This holds if the height of the window without the arch height is considered.If the total height is used the error is around 1 m.Since the quadrant base search relies on the maximum rectangle which covers no points, it is probable that the arch will be cut at a certain height.This is because some points of the arch will fall into the respective rectangular area.Except from that, the assumption from the beginning that windows tend to be reconstructed rather to large than to small can be confirmed by the results of Table 1 and 2   The quality of the seed points plays an important role for the outline hypothesis and so for the whole reconstruction process.The highest demand on improvements lies in this task.If the seed points lie to close to the window edge the hypothetic windows are rather too small than to large and the assumption that the window hypothesis are generally too large does not hold anymore.In Figure 14 a comparison of the window outlines is depicted.On the left side (a) the outline from the automatically created seed points can be seen, while on the right side (b) the seed points were set manually.This also means, that a single point, e.g. from a cross bar can deteriorate the outline creation process.

CONCLUSIONS AND FUTURE WORK
This paper has shown a possible approach for the reconstruction of rectangular windows based on two main steps: First a hypothesis for the outline is created with an iterative quadrant based search algorithm, which functionality depends on the quality of the provided window centres.The second step consists of the merging of the outlines, describing their uncertainty with a Rayleigh-PDF which is weighted by the deviation of edges after the outline generation.If the initial position of the window centres has a sufficient precision the window height and width can be determined with an accuracy of a few [dm], what is also the range of typical point distances in our data.The approach works the better the more identical windows can be found in a single row or column of the façade, because this can mitigate the lack of information because of the sparseness of the point cloud.Furthermore there is the restriction that the approach is based on the assumption of rectangular windows, so that windows with other shapes also appear rectangular.As it was shown the point density does not allow to distinguish between arch and rectangle windows, what means that arch windows will normally reconstructed to small (in height direction).Regarding that topic it has to be remarked that is also difficult or even impossible for the user to see smaller window or to distinguish between different window shapes, what makes the limitation on rectangular windows acceptable.There are also cases, where the approach mainly fails, which can often be traced back to the fact of wrong window centres because of irregular arrangement of the windows.Since the ground floor has in many cases a special appearance the processing of the ground and the other floors shall be separated in future.A more accurate approach for the generation of the windows centre can make the approach more robust.As it was mentioned above it is also difficult for the user to determine the best window outline in the used data set, thus no ground truth could be manually created out of the input data.Since there is currently no existing building model, which includes windows, the generation of ground truth data is an important task for providing better evaluation results.Then it will also be possible to check the absolute window positions.
As further future works other PDFs can be tested and further weighting properties can be defined to better adapt the PDF on the actual quality of a certain outline.

Figure 2 .
Figure 2. Principle of the quadrant based search for the window outline.

Figure 3 .
Figure 3.Effect of the iterative procedure during the quadrant based search (referring to parts VII and VIII of Figure 2) 3.2 PDF-Function Each window outline determined in the previous step represents possible edges.Because of the low point density this edge has an uncertainty, which shall be expressed with a probability density function (PDF).Since windows are usually arranged in a regular way the combination of the density functions of windows in one row or column shall give the most probable line in that direction.The PDF used in this special case shall have the following properties: -The function shall have an adjustable parameter to express the uncertainty of the edge.-The PDF shall regard that it is more likely that the boundary is closer to the seed points, because of possibly missing points below, above, right or left to the window.Based on these requirements the Rayleigh distribution (Eq. 1) was chosen.
be seen in Figure4.The position of the hypothetic edges is places at the peak position which is equal to b.The cumulated distribution function at b has the value 0.4, which means that it is more likely that the edge is right to the peak.

Figure 5 .
Figure 5. Point cloud of TUM area; façades shown in Section 4.2 and 4.3 are marked with yellow rectangles.
to 9 the seed points as a result from the window detection are shown with blue dots.The chosen distance threshold to derive the candidate points is d s = 8 m.The window edges derived from the quadrant based search are shown in Figure7.All left edges are drawn as blue lines and all right edges as red lines in Figure8.The red lines in Figure9are the upper edges, the blue lines the lower edges.

Figure 6 .Figure 7 .
Figure 6.Image of the façade used as example in Section 4.2

Figure 8 .
Figure 8. Vertical edges for all window outlines shown in Figure 7, left edges are painted in blue, right edges in red.

Figure 9 .
Figure 9. Horizontal edges for all window outlines shown in Figure 7, lower edges are painted in blue, upper edges in red.The distances of the edges from the quadrant parts which are used for the weighting are multiplied with the factor 2 before they are used as factor b for the calculation of the PDFs.The minimum b is set to 0.5 (can be interpreted as meter) what reflects the typical point distance.In Figure 10 an example for Evaluation of window size (arch windows) The window sizes are validated against measured values for two further façades.At these façades two different kinds of arch windows can be found.The results are shown in Table . In the following Figures 12 and 13 two more façades are shown, for which only a visual check can be made.

Figure 12 .
Figure 12.Reconstructed windows and image of façade

Figure 13 .
Figure 13.Reconstructed windows and image of façade Window outline hypothesis for a façade part.a) Automatically determined seed points, b) Seed points set manually.