AUTOMATED DETECTION OF REPEATED STRUCTURES IN BUILD ING FACADES

Automatic identification of high-level repeated str uctures in 3D point clouds of building façades is c rucial for applications like digitalization and building modelling. Indeed, in m any architectural styles building façades are gover ned by arrangements of objects into repeated patterns. In particular, façades are generally designed as the repetition of some few ba sic objects organized into interlaced and\or concatenated grid structures. Sta rting from this key observation, this paper present an algorithm for Repeated Structure Detection (RSD) in 3D point clouds of buil ding façades. The presented methodology consists of three main phases. First, in the point cloud segmentation stage (i) the build ing façade is decomposed into planar patches which are classified by means of some weak prior knowledge of urban buildings formul ated in a classification tree. Secondly (ii), in th e element clustering phase detected patches are grouped together by means of a imil rity function and pairwise transformations b etween patches are computed. Eventually (iii), in the structure regularity estim ation step the parameters of repeated grid patterns are calculated by using a LeastSquares optimization. Workability of the presented approach is tested using some real data from urban sce es.


INTRODUCTION
In recent years the application of terrestrial laser scanning (TLS) for the reconstruction of highly detailed urban scenes has been continuously growing.Indeed, while airborne laser scanning can provide the global shape of buildings (roofs, footprints, volumes), ground-based data TLS can be exploited for modelling their façades.In this direction also mobile laser scanning and photogrammetric systems have been used for data collection because of their higher productivity.It is out of the scope of this paper the comparison between different techniques, but broadly speaking laser scanning is a simpler and highly automated approach, resulting in a shorter time for data acquisition and registration.However, the captured point clouds are generally difficult to handle because they may include a large number of unwanted objects (like cars, pedestrians, vegetation) and are affected by noise and outliers.For these grounds, raw data are processed to generate vector models.The complexity and the cost of manual modelling of building façades fostered the development of several methods for their automatic or semi-automatic segmentation.However, the stateof-art automatic approaches cannot generally reach the quality and the completeness of manual models.This is mainly due to the fact that point clouds may typically exhibit significant missing data due to occlusions.Moreover, the extraction of façade models requires the semantic interpretation of a very large variety of possible architectural elements, depending on the building typology, construction time, and style.Whether these tasks may be accomplished with ease (although time consuming) by trained human operators, they are a critical issue for machine learning systems.On the other hand, building façades usually feature large selfsimilarities and relies on a regular repetition of some basic patterns.The highly regular nature of building façades is widely demonstrated and arises from manufacturing, aesthetic and functional reasons.In addition, the presence of some repetitive patterns and symmetries are typical in almost all architectonical styles.These properties can be exploited in the modelling process and the presence of regular patterns can be used to recover missing information in correspondence of occlusions as well as to enhance automatic façade modelling.State-of-art methods using structure repetition in urban façades can be classified into interactive and automatic.In the former works (Zheng et al., 2010;Nan et al., 2010) the user firstly defines some basic façade objects which are then snapped to similar elements in the point cloud.The latter methods are more challenging, since there is no a priori knowledge of size, shape, and location of elements describing the pattern.In addition, façade elements can be incomplete or corrupted by imperfections and noise.For this reason automatic solutions (Müller et al., 2007;Xiao et al., 2009) rely on the assumption that the façade can be split into blocks forming a single orthogonal grid of elements.Even if there are a certain number of façades satisfying this assumption, in many cases real façades present a more complex structure.This work presents a more flexible strategy aimed at detecting in a fully automatic way concatenated and\or interlaced grids of repetitive elements.

RELATED WORK
Given the large volume of work on building and façade modelling, we refer the reader to the recent survey by Musialski et al. (2012) for a comprehensive review.Here we will only focus on previous works addressing detection of repeated patterns in building façades.Detection of symmetries, repetitions and regular structures in building has been extensively studied in the field of image analysis while point clouds received significant attention only lately.Image-based methods for façade analysis can be based either on single image analysis (Müller et al., 2007;Musialski et ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W2, 2013ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey al., 2010) or multi-view images jointly with the extracted point clouds (Xiao et al., 2009).In Lee and Nevatia (2010)

OVERVIEW OF THE RSD METHOD
In this work we present a completely automated algorithm to look for, detect and make use of repeated structures in point clouds of building façades acquired by TLS.The presented RSD method consists of three main phases: (i) point cloud segmentation, (ii) element clustering, and (iii) structure regularity estimation (see the workflow in Fig. 1).Stage (i) starts with the whole point cloud as input, then detects planar façade clusters and classifies them into façade objects.The segmentation strategy is based on a modified RANSAC (Fischer and Boles, 1981) implementation specifically aimed at segmenting building façades.In particular, some constraints based on buildings geometry are included into the process to guarantee that each detected planar clusters will correspond to a different façade element.Then, the detected clusters are automatically classified into façade objects (e.g.windows, walls, doors, etc.) on the basis of some a priori knowledge of façade geometry, which is organized into a classification tree.
Once façade elements are detected the aim of stage (ii) is grouping the set of façade objects by means of a similarity measure.During this phase, similarity is evaluated between pairs of patches and an initial estimation of the similarity transformation between objects is given.The grouping is achieved by a geometric registration followed by an iterative bottom-up clustering of façade objects.
In the final stage (iii), the parameters of the repeated façade patterns are automatically estimated.The way this task is afforded is inspired by the generalized Hough transform (Ballard, 1981) and the lattice voting scheme of Pauly et al. (2008).The information derived from the previous step is used to carry out a global optimization towards optimal alignment of façade elements.The final output is a series of lattices representing the local periodicity in the vertical and horizontal direction of façade objects.Since to some extent the result of the regularity estimation is dependent on the element clustering, these steps are iterated until convergence.
Figure 1.Workflow of the presented RSD iterative approach.

FACADE SEGMENTATION
The input to the processing framework is an unorganized point cloud coming from TLS survey of an urban building façade.We assume that the façade has a dominant planar structure, characterized by a flat dominant surface and with other façade's components having off-plane depth variations with respect to this plane, either positive (outwards) and negative (onwards).
This assumption is generally valid for modern building architecture styles.The modelling of historical premises with more involved geometry and decoration is not addressed in this work (see e.g., Grussenmeyer et al., 1999).In the case multiple scans of the building façade have been gathered, these need to be first aligned into a local Cartesian coordinate system having the Z-axis aligned to the ground up-vector.Then planar clusters in the original unorganized façade point cloud are detected by using the automated approach presented in Previtali et al. (2013).Here only some aspects mainly related to RSD are briefly described.In particular, the segmentation methodology can be divided in two main steps: (i) planar cluster detection and (ii) façade object classification.

Planar cluster detection
A first step towards the segmentation of the building façade is to detect all its planar clusters.Detection of façade objects is accomplished by using a specifically designed modified RANSAC algorithm.This implementation is aimed at reducing spurious results obtained by the standard sequential RANSAC.

Façade object detection
Planar clusters are then further automatically classified into façade object classes (e.g.wall, window, door, roof, etc.).This detection phase is important for further regularity detection.Indeed, during the object grouping phase similarities between objects are searched for only into each façade class (e.g.no similarity is sought between a door and a window).This reduces the computational time of the next phase and prevents wrong structure detection.Classification of planar clusters into façade objects is performed by using some recognition rules derived from the knowledge of urban scenes and façade priors.In particular, the following characteristics of each object are evaluated: size, position and topology, orientation.
The above object characteristics and priors are summarized in a classification tree which is used to guide the entire automatic classification phase (Fig. 2).
Figure 2. Classification tree for façade objects detection.

ELEMENT CLUSTERING
Once the identified planar clusters are subdivided into façade classes, the goal of element grouping is to gather similar objects together so as to identify repetitive elements.The grouping is achieved by a two-step clustering algorithm.First, elements classified in the same façade feature class are clustered according to their shape (base and height of the bounding box).Indeed, façade objects belonging to the same feature may present significant geometric differences and follow different repetitive patterns.In Figure 3, a façade with two different types of windows is shown.This first step is performed to have a rough clustering of similar objects and to reduce the number of misclassifications.Indeed, it is unlikely that objects having a significant difference in the bounding box shape present a high similarity between them.Once two objects S i and S j belonging to the same façade feature are clustered together, they are aligned to measure similarity between them.This task is performed by computing a rigid-body transformation T ij using a standard Iterative Closest Point (ICP) implementation (Besl and McKay, 1992).In particular, the transformation T ij is restricted to be a translational one.Indeed, in our work we are focusing only on determining repeated structures that can be obtained by translating a base object.Similarity transformations involving scaling and rotations are not addressed in this paper.
Once S i and S j are aligned, to tolerate poor quality input data we quantize the space of the overlapping region of the aligned slices and we compute the similarity between S i and S j in this quantized space.Specifically, the aligned slices S i and S j are embedded into a volumetric grid whose size is determined by the bounding box of the overlapping region B between the two slices (see Fig. 4b).
a. b. c.The grid resolution is fixed a little bit larger than the mean sampling distance in the point cloud.For each resulting voxel we define two functions v i and v j , indicating the number of points contained in the voxel from S i and S j , respectively.At the end, the original point clouds are quantized into a tensor representation (see Fig. 4c).Then the similarity measure (SM) between slices S i and S j is defined as (1): ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W2, 2013 ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey where: The adopted SM is a generalization of the normalized linear correlation coefficient (ρ).This means that SM represents a local similarity measure of point distributions in the region B between the two considered objects S i and S j .Like ρ, also SM may range from -1 (full inverse correlation) to 1 (full direct correlation).For this reason we can assume that SM values close to 1 indicate high similarity between S i and S j while in the case SM is close to zero or negative, these are assumed to be different.Once the similarity is measured for each pair of slices, slices having the maximum similarity are automatically clustered by using a bottom-up method as fare as no more clusters can be created.The clustering process is stopped until SM is lower than a user defined threshold (0.7 has been used in our tests).In this way, elements with low similarity with respect to the others in the cluster are discarded, improving the robustness of the method.

ESTIMATION OF STRUCTURE REGULARITY
The next step estimates the structure regularity for each group of previously detected elements.In particular we can observe that repeated elements in buildings generally form regular lattice structures.A lattice structure can be defined as set of 2D points arranged in a regular grid where columns and rows spacing may change.For this reason, a regular structure of size MxN is a tuple (S 0 , G MN ) where S 0 is the basic repeated element in the structure and G MN is a transformation group acting on S 0 .The transformation group G MN has a lattice structure consisting of M rows and N columns, where each element g ij , represented by coordinates Xg i and Yg j , is a node of the lattice.In order to estimate the transformation group G MN a generalized Hough transform voting scheme has been used.Indeed, a transformation space Ω, containing the identity transformation, representing all possible pairwise transformations is generated.Then, once two similar objects S i and S j are detected (Fig. 5a-d), they vote in Ω for the transformation T ij between them.The resulting voting scheme (Fig. 5e-f However, the set of pairwise transformations T ij between objects in a cluster defined in the previous step presents a statistical variation in the distribution of the estimated transformations due to noise in the model, local variations of sample position, and non-perfect alignment within the ICP registration phase.All these contributions lead to some inaccuracies in the transformation estimation.In addition, some transformations may miss due to holes in the input data.For these reasons, the reliably detection of regular structures calls for a grid fitting approach that is robust against noise and holes.To this end, a global optimization method based on the work of Pauly et al. (2008) has been applied.However, while in that work the estimation procedure is aimed at detecting only rectangular lattices that are represented as the composition of two base vectors (one for row and the other for column description), our methodology takes into account a more general configuration considering also the possibility that columns and rows spacing can vary within the lattice.The unknowns grid position for a lattice structure of M rows and N columns are represented by the row coordinates Xg 1, 2,…, i, …, M and column coordinates Yg 1, 2,…, i, …, N .The input data are the centres c ij (Xc i , Yc j ) of the clusters extracted from the set of transformations T ij mapped in the transformation space (Fig. 6a).To find the unknown grid positions g ij , we applied an optimization scheme combining two energy terms.The first one takes into account the distance between the grid location g ij to the transformation c ij (Xc i , Yc j ): where c ij is the cluster centre closest to the grid location g ij .The continuous variable α ij is a weight measuring how reliably a grid location is mapped to a cluster centre and vice versa.They are included as additional unknowns in the optimization process accounting for holes and outliers.Indeed, values of α ij close to zero indicate a hole or an outlier, while values close to 1 represent a reliable matching between transformation cluster and grid location.
The second energy term is aimed at maximizing the number of valid correspondences between grid location and cluster centres: The final objective function to be minimized is defined as: where γ balances the two energy terms.In the tests carried out, γ was fixed to 0.8.In order to find repeated similarity in the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W2, 2013 ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey lattice structure, the spacing between consecutive columns and rows are calculated and clustered.In the case some grids present a similar spacing, in the minimization process these additional constraints will be enforced.For example, in the case the spacing between columns Yc j -Yc j+1 and Yc j+1 -Yc j+2 are clustered together, the following constraint equation is added to the minimization process: To minimize the above objective function we used the Gauss-Newton Least Squares technique (Triggs et al., 2000).As can be seen, the minimization process is non-linear and a set of initial estimates of the size of the lattice in terms of rows and columns is necessary as well as the initial values for the grid locations g ij.
In Once the lattice is estimated, the algorithm operates iteratively by performing a new element grouping in correspondence of each node of the lattice.The process halts when no changes in the lattice nodes are observed between two successive iterations.In Figure 7 an example of lattice estimation is shown.
In particular, the presented algorithm can efficiently detect repeated features with non-rectangular lattice patterns (Fig. 7b) and may also deal with missing elements (Fig. 7a).In the second case it is possible to observe that the weight to the bottom-left element is close to zero.

RESULTS AND PERFORMANCE EVALUATION
In this section some results of RSD application to 3D urban facades are reported.All datasets were acquired by using a Faro Focus 3D laser scanner (www.faro.com),although the application of the presented approach is likely to be extended to point clouds with lower data quality, for example from time-offlight or mobile laser scanners, and from dense image matching.Datasets presented in this paper refers to facades of different complexity, style and scanning resolution.In particular, the scanning resolution, in terms of mean ground sampling distance ranges from 5 mm to 2 cm.Although several parameters have to be setup in our procedure, only three of them need a proper selection according to the specific dataset.The first one concerns the RANSAC threshold for plane detection which involves mainly the data quality and in particular noise.In experiments here a threshold value of 1 cm has been used.The second one is the clustering threshold and it is more influenced by the façade geometry and by the size of tiles.In tests a threshold of 20 cm has been selected.However, a smaller value may be more adequate in the case of façades with features very close to each other.The last critical threshold is SM.The default value of 0.7 generally provides good results, but in the case of noisy input data this value should be decreased to foster grouping. a. b.
c. Figure 7. Regularity structure superimposed to the point cloud (left) and lattice estimation (right).Lattices centres are coloured according to the associated weight (white for 0 and black for 1).
In Figure 8, some examples for the presented approach are reported.Figure 8b shows an example of a simple building where the repetition structure cannot be represented by a regular grid.The flexibility of this method in considering more complex lattice structures gives the chance to easily represent also this kind of regularity.Figure 8a shows an example with complex façade having multiple interlaced groups of repeated elements.This RSD approach is also applicable to façades with only a few repetitions even along with one dimension (Fig. 8c), since it is largely insensitive to the number of repetitions.
a.In order to evaluate the performance, a manual segmentation of the automatically processed facades has been accomplished.In particular, the fractions of commission errors (wrong detections) and omission errors (missing detections) in the detection of lattice nodes have been evaluated with respect of manually detected nodes, which have been assumed as benchmarking values.In

CONCLUSION AND FUTURE WORK
This paper investigates the problem of finding repeated structures in urban building facades by using a fully automatic procedure termed as repeated structure detection (RSD).The key observation is that facades with concatenated and/or interlaced repeated elements are ubiquitous in urban scenes.For this reason, an effective solution for detecting complex lattice structures is described, which is based on an iterative grouping and lattice estimation steps.Due to the flexibility of this approach, urban facades with various complexity and modern architectural can be handled.However, the RSD method is not applicable to facades which cannot be detected by a rectilinear model, for example in the case of completely irregular patterns like in many historical buildings.
occupancy raster map where occupied regions are represented by pixel value 1.To achieve this, points belonging to the same plane are projected orthogonally to a bitmap.All pixels in the bitmap containing at least one projected point are assigned the value 1, while others are given value 0. This raster map allows finding connected regions of pixels featuring value 1.Then all points whose projection belongs to the same connected component are grouped into the same cluster.Once all planar elements are detected, the extracted planes are clustered together to reduce over-segmentation problem.Object clustering is performed by evaluating three parameters: (i) similarity of normal vectors; (ii) perpendicular distance between planes; and (iii) intersection between clusters.

Figure 3 .
Figure 3. Example of a façade with two different window types (red and green) on the left and clustering results (right) using base and height of the bounding box of detected objects.

Figure 4 .
Figure 4. Example of a volumetric representation of a window: the original point cloud (a), the voxel splitting (b) and the final tensor representation v i (c).
) has peaks in correspondence of locations that are used as nodes for the estimation of the lattice structure G MN .Example of a simple 1D regular structure with all possible pairwise transformations (a-d) which form a characteristic cumulative pattern in the transformation space (e-f).
the first iteration, approximate values are determined by clustering in Ω the pairwise transformations T ij along the two dominant directions of the lattice (Fig. 6).Detected cluster centres are used as approximations of the lattice row (Xg 1, 2,…,M ) and column (Yg 1, 2,…,N ) coordinates.Determination of first iteration approximate values.The pairwise transformations initially estimated (a) are then clustered along the two dominant directions of the lattice (b-c) giving the initial estimates of the lattice structure.
Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume II-5/W2, 2013 ISPRS Workshop Laser Scanning 2013, 11 -13 November 2013, Antalya, Turkey b. c. Figure 8. Results of RSD algorithm.Notice the reconstruction of fully occluded windows (a) and the satisfactory performance in low resolution regions (b-c).
the unique characteristics of façade structures such as regularity and orthogonality are exploited in a statistical model to detect translational symmetries and repetitive window structures.

Table 1 .
Table 1, some statistics on both kinds of errors are presented, showing quite good results.Commission and omission errors for repeated structure detection.