INCREMENTAL REFINEMENT OF FAC¸ADE MODELS WITH ATTRIBUTE GRAMMAR FROM 3D POINT CLOUDS

: Data acquisition using unmanned aerial vehicles (UAVs) has gotten more and more attention over the last years. Especially in the ﬁeld of building reconstruction the incremental interpretation of such data is a demanding task. In this context formal grammars play an important role for the top-down identiﬁcation and reconstruction of building objects. Up to now, the available approaches expect ofﬂine data in order to parse an a-priori known grammar. For mapping on demand an on the ﬂy reconstruction based on UAV data is required. An incremental interpretation of the data stream is inevitable. This paper presents an incremental parser of grammar rules for an automatic 3D building reconstruction. The parser enables a model reﬁnement based on new observations with respect to a weighted attribute context-free grammar (WACFG). The falsiﬁcation or rejection of hypotheses is supported as well. The parser can deal with and adapt available parse trees acquired from previous interpretations or predictions. Parse trees derived so far are updated in an iterative way using transformation rules. A diagnostic step searches for mismatches between current and new nodes. Prior knowledge on fac¸ades is incorporated. It is given by probability densities as well as architectural patterns. Since we cannot always assume normal distributions, the derivation of location and shape parameters of building objects is based on a kernel density estimation (KDE). While the level of detail is continuously improved, the geometrical, semantic and topological consistency is ensured.


MOTIVATION AND CONTEXT
3D building models are nowadays essential for several tasks such as urban and telecommunication planning (Köninger and Bartel, 1998;Knapp and Coors, 2008) and disaster and rescue management (Kolbe et al., 2008) in particular flooding simulations (Schulte and Coors, 2008).Further, the data model CityGML (Gröger et al., 2012) supporting several levels of detail (LoD) is widely used in order to address semantics and exchange city data models.For many applications such as disaster management and visualization in particular in the navigation context, fac ¸ades (LoD3 models) are important.In order to achieve an interpretation of the observations with good quality models providing prior knowledge are needed.In this context, formal grammars receive increasing attention (Musialski et al., 2012).For the reconstruction of fac ¸ades Becker (2009) introduced a hybrid approach by the integration of inferred grammar rules into a nonparametric reconstruction process.Ripperda and Brenner (2009) used a probabilistic grammar for the description of fac ¸ades.The production rules are expanded by the relative frequency of a single rule extracted from a manually collected rule database.Martinović and Van Gool (2013) proposed an approach to learn stochastic attributed grammar rules for two-dimensional fac ¸ade generation and reconstruction from imagery data.Dehbi et al. (2016) presented a statistical relational based approach for automatic learning of an attribute grammar for building reconstruction.
Formal grammars demand a mechanism that fits data to underlying grammar rules.For this issue, several parse algorithms have been introduced.Han and Zhu (2009) used a two-staged approach in order to parse images based on a graph grammar.They assume that man-made objects are modelled by sub-objects using the grammar rules.Teboul et al. (2011) presented an approach for 2D shape grammar parsing of fac ¸ades using reinforcement learning.Toshev et al. (2010) introduced a method for the detection and parsing of buildings from 3D laserscan data.Martinović et al. (2012) proposed a three-layered approach for fac ¸ade parsing.Schmittwilken (2012) presented an estimator and parser for the identification and interpretation of fac ¸ades from LIDAR point clouds based on attribute grammars and a-priori probabilities.All these approaches expect offline data in order to parse the underlying grammar rules.
The presented work is part of a project which aims to build 3D building models during the acquisition of data by a lightweight autonomously flying UAV.It uses the visual information from cameras for navigation, obstacle detection, exploration and object acquisition leading to a high resolution dense 3D point cloud.Based on a semantically specified user inquiry the UAV delivers a point cloud stream improving the level of detail.Hence, an incremental model refinement is needed in order to deal with the continuous observation stream and to falsify or verify new model hypotheses.
The main contribution of this paper is the incremental parsing of weighted attribute context-free grammar (WACFG) for the automatic 3D building reconstruction.The parser refines models based on new observations.If a refinement is not possible, model hypotheses are falsified and rejected.In this way an adaptation of the available parse tree that has been acquired from previous interpretation steps is enabled.Parse trees are iteratively updated drawn upon ideas of graph transformation systems (Löwe, 1993;Heckel, 2006).Prior knowledge on fac ¸ade parts is given by probability densities.In order to avoid distribution assumptions, the location and shape parameters of building parts are identified with a non-parametric kernel density estimation (KDE).During the interpretation, geometrical, semantic and topological consistency is ensured Gröger and Plümer (2012) using transaction rules.
The remainder of the article is structured as follows: An introduction of formal grammars is presented in section 2. Our method will be explained in detail in section 3. Hereby, the identification of fac ¸ade parts is explained in subsection 3.1 while the algorithms for the incorporation of these identified parts based on the incre- mental parsing of the attribute grammar rules are presented in subsection 3.2.The paper is summarized and concluded in section 4.

FORMAL GRAMMARS
Ever since formal grammars were introduced by Chomsky (1956Chomsky ( , 1959) ) for reconstructing sentences of natural language, they have also been used to generate formal languages.The so-called Chomsky hierarchy distinguishes between four levels (type 0 to type 3) of formal grammars.A formal grammar G can be defined as the quadruple {S, N, T, P } of a start symbol S, a set of nonterminals N represented by capitalised initials, a set of terminals T denoted by lower case initials and a set of production rules P .
A special case of formal grammars are context-free grammars which correspond to type 2 according to Chomsky's hierarchy.Production rules appear in the form A → a where A is a nonterminal, and a is a sequence of terminals and non-terminals.This rule implies that each occurrence of the symbol A can be replaced by the string a.In order to express the likelihood of the application of rules, each rule is augmented by a weight or a probability defining a weighted or a probabilistic context-free grammar respectively (Geman and Johnson, 2002).
Motivated by requirements of early programming languages such as ALGOL and in order to cope with a restricted form of context sensitivity Knuth (1968Knuth ( , 1971) extended context-free grammars and introduced attribute grammars.Hereby, terminals and nonterminals are expanded by attributes, whereas production rules are extended by "semantic rules".The latter specify the constraints among the attributes.In our example these enable us to state that all steps within the same staircase have the same dimensions which cannot be expressed by context-free grammars.An extract of an attribute grammar of stairs is shown in listing 1.
The grammar symbols at the top are augmented by the attributes height, depth and width, which are used in the semantic rules R2(P2) to R4(P2) at the bottom in order to specify the identity between the shape parameters of risers and treads within the same stairs.Additionally, R1(P1) ensures that the new stairs consist of exactly one more step than the one before.The superscript indices n and n-1 are used to differentiate between multiple occurrences of the same symbol.
Listing 1: Excerpt of an attribute grammar for stairs P1 : S t a i r s n−1 → S t e p S t a i r s n P2 : S t a i r s → S t e p P3 : S t e p → r i s e r t r e a d R1 ( P1 ) : S t a i r s n .n u m b e r O f S t e p s = S t a i r s n−1 .n u m b e r O f S t e p s +1 R2 ( P2 ) : S t e p .w i d t h = S t a i r s .w i d t h R3 ( P2 ) : S t e p .h e i g h t = S t a i r s .h e i g h t R4 ( P2 ) : S t e p .d e p t h = S t a i r s .d e p t h . . .

INCREMENTAL PARSING OF 3D POINT CLOUD
As yet, the parsing of formal grammars expect an offline data processing, especially for 3D point clouds.Based on a given grammar, the parser searches for a most likely parse tree with successively grammar rule application that matches the underlying data well (see figure 1).In this section we introduce a novel approach that enables to deal with changing data, e.g.provided by an UAV.The parser interprets the building parts within the measured point cloud and determines their descriptive location and shape parameters.Based on the detected parts a derivation tree is built that reflects the 3D model of the building.As soon as a new point cloud segment is provided, it is interpreted and the derivation tree of the building reconstruction is updated.This update of the tree is the main discriminative aspect of the incremental parser in comparison to an offline parser.The whole algorithm of the incremental parser (see algorithm 1) is subdivided into the two parts building part detectors and tree transformation which are executed consecutively.An overview of our approach is shown in figure 3. Beforehand, the main fac ¸ade plane P lane is estimated and consequently an orthogonal rectification of the point cloud is performed.
The first part of the algorithm is responsible for the detection of the building parts within the measured point cloud P ts.Herewith, the descriptive location and shape parameters of each building part is determined.These parts build a set of nodes N odes.This identification step is the core of section 3.1.The results of the interpreted dataset from the first part are employed to incorporate the observed building part in the model derived or predicted so far.A predicted tree is deduced based on a reasoning process as described in Loch- Dehbi and Plümer (2015).If there is no tree available, a new model is constructed otherwise.In both cases, the model consists in a parse tree.It describes the taxonomy and partonomy of a given fac ¸ade reflecting semantics of each building part with respect to an a-priori learned weighted attribute context-free grammar (W ACF G) (Dehbi et al., 2016).The interpretation is performed as long as new observations (point cloud stream) are received.This incremental interpretation is discussed in section 3.2.

Identification of fac ¸ade parts
This section describes the process of the determination of the fac ¸ade parts.This is based on building part detectors tailored for each building part of interest.At first, the main fac ¸ade is detected using a fac ¸ade detector.This is performed with the use of a RANSAC estimation (Fischler and Bolles, 1981) based on the observed and geo-referenced points P ts.RANSAC delivers the estimated plane P lane together with the consensus set that supports the plane model.We assume that the input point cloud is taken from a fronto-parallel view of the world plane.Otherwise, an orthogonal rectification of the point cloud is performed according to the transformation in equation 1 leading to plane points P P ts (Förstner and Wrobel, 2016).Herewith, a rotation R(Np, Nxz) using the vector pair (Np, Nxz) along the axis Np × Nxz takes place.Np represents the normal of P lane, whereas Nxz is the normal of the xz plane of the world coordinate system.I, Ñp and Ñxz denote the identity matrix, the normalization of Np and Nxz respectively.
In this manner an accurate detection of fac ¸ade parts is enabled.
To this end, further building part detectors are required.Openings such as windows or doors in the fac ¸ades are interpreted as holes in the point cloud.This corresponds to areas with a low point density.Therefore, a kernel density estimation (KDE) (Wand and Jones, 1994;Wang and Suter, 2004) is appropriate for the determination of the location and shape parameters of windows and doors.KDE enables a non-parametric estimation of a probability density function avoiding to make data distribution assumptions.The bandwidth of the used Gaussian kernel is adapted and optimized for each estimation depending on the underlying data (Benoudjit et al., 2002).The identification of the fac ¸ade openings in algorithm 2 starts with a kernel density estimation of the x-and z-coordinates of the points P ts.Consequently the extrema along both directions are identified.The openings in the fac ¸ade correspond to the intersection of the horizontal and vertical orange lines in figure 4.These denote the orthogonal projection of single minima mix and miz on the according axis leading to an approximative centroid and hence a location of an opening.Furthermore, in order to determine the shape parameters (width and height) a bounding box BB is defined over each centroid.Each bounding box BB is expanded in four directions (left, right, up, until a minimal point number is reached.Besides, the maxima max and maz in the according direction are used for the robust determination of the shape parameters such as shown in figure 5. Further background knowledge -probability density functions -about shape parameters of building parts derived from a relational building database (RBDB) is also used in order to acquire a good size of the bounding box BB.The latter corresponds to the shape parameters of a single fac ¸ade part and hence a leaf of a parse tree.These nodes are collected in a list T empN odes.As can be stated for the windows of the ground floor in figure 4, due to structured walls the derivation of the shape parameters is in general not straightforward.In this case, a local floor-wise reestimation of the parameters is needed.To this end, a segmentation of the point cloud is performed based on the maxima values of the coordinates.This enables a recursive application of the previous steps for each floor based on a new bandwidth leading to a list N odes and their improved parameters.In order to discriminate between windows and doors, common architectural patterns as well as probability density functions derived from RBDB are used.

Incremental refinement of parse trees
This section deals with parse tree transformations based on the building parts detected and acquired from the first stage described in section 3.1.These parts are integrated in a parse tree with respect to the grammar rules of the WACFG.If there is no tree already available, e.g. at the beginning of a UAV mission, a parse tree T ree is predicted using a stochastic and geometric reasoning based on few observations such as the fac ¸ade width (Loch- Dehbi et al., 2013).If a parse tree already exists, the resulting building parts N odes from the building part detectors are regarded as leaves of the current parse tree.They represent together with the WACFG and the parse tree T ree the input of algorithm 3 for the parse tree transformation.Since the parse tree represents a directed acyclic graph, our approach is inspired by the ideas of Graph transformation (Löwe, 1993;Heckel, 2006) in order to adapt the tree structure based on the concept of transformation rules.The latter consist of a start and a target graph that are mapped using transitional rules.
The first step in order to integrate the new observations in the model T ree is a diagnostic step.In this stage, nodes of the current tree T ree that deviate from the new observed fac ¸ade parts N odes are identified and labelled.To this end, the discrepancy is measured based on a Hausdorff distance based analysis (Edgar, 2007).In this step, only nodes in the new observed region are considered.The remaining nodes out of this scope remain unchanged and are maintained in the new tree.During the node comparison, it is differentiated between four states of nodes: error-free, error-prone, erroneous and new.Error-free nodes are those still having the same location and shape parameters taking the observations uncertainty into account.In this case old nodes are confirmed by new observations and retained in the so far constructed tree.Error-prone nodes are characterised with nearby location values and one or more significantly different shape parameters.Nodes that are no longer supported by the new observations, are denoted as erroneous.Furthermore, new nodes are those without a correspondence in the tree.
The diagnostic step leads to a list of corrupted nodes CN odes and their attributes that will be corrected in a subsequently step leading to a list CorrN odes together with the new nodes.The corrected attributes that affect other parse tree nodes are considered, the according changes are then propagated based on the semantic rules of WACFG.The WACFG describes the taxonomic and partonomic structure of fac ¸ades by a weighted context-free grammar (WCFG) and the substantial topological as well as geometric constraints which are described using Statistical Relational Learning (SRL) methods namely Markov Logic Networks (MLNs) (Dehbi et al., 2016).
In the next step, erroneous nodes from CN odes and possibly belonging parse tree parts are deleted.That would lead to conflicts between the current tree structure and the underlying grammar rules.In order to deal with such cases, a rule selection is performed based on the rule weights as well as the floor information derived during the identification stage in section 3.1 leading to a most likely rule.If the conflict cannot be resolved, the according parse tree part is deleted from T ree.The belonging nodes are retained in a separate node list giving the possibility for a later consideration.Geometric and topological errors caused by deletion of nodes or tree parts are again considered and corrected using the attributed part of the WACFG.Constraints such as alignments of windows in floors or attributes identity are ensured using MLNs.
The list CorrN odes is now considered.Based on the floorwise segmentation and the rule's likelihood, a rule selection is performed.Hereby, rules with higher node support are preferred.Based on this deduced parse tree structure, constraints are ensured in a same manner as in the deletion case using the semantic rules of the WACFG.

CONCLUSION
This paper introduced a novel approach for the automatic incremental refinement of fac ¸ade models.The refinement process is based on a weighted attribute context-free grammar.In contrast to other approaches that expect an offline data interpretation, the presented method deals with continuous observations coming from an unmanned aerial vehicle (UAV).Prior knowledge on fac ¸ade parts (e.g.windows) is given by non-parametric distributions using a kernel density estimation (KDE).These distributions are iteratively adapted using new observations.Accordingly, available parse trees acquired from previous interpretation steps are adapted with respect to the rules of the grammar.The falsification or rejection of model hypotheses is supported as well.As long as the level of detail is continuously improved, the geometrical, semantic and topological consistency is ensured.
Up to now, the presented approach paid attention to fac ¸ades that can be split into floors.The underlying W ACF G supports several splitting schemes, e.g.column-wise.In order to take more sophisticated fac ¸ades into consideration an extension of the rule selection strategy is required.

Figure 1 :
Figure 1: Offline parsing of 3D point clouds using formal grammars.

Figure 2 :
Figure 2: A possible derivation tree of a context-free gable roof building grammar.As illustrated in figure2, a possible context-free derivation which describes the structure of a gable roof building can be modelled

Figure 3 :
Figure 3: An overview of the incremental attribute grammar parsing of a 3D point cloud stream.

Figure 4 :
Figure 4: KDE of the consensus set of the fac ¸ade main plane of the Frankenforst manor in Vinxel/ Germany.The extrema are incorporated for the floor-wise and column-wise segmentation of the fac ¸ade and for the identification of openings.

Figure 5 :
Figure 5: Local KDE in the neighbourhood of an opening in the main plane of the fac ¸ade from figure 4. The maxima are incorporated for the identification of shape parameters of a window.

Figure 6 :
Figure 6: An incremental interpretation of the fac ¸ade (Fa) of the Frankenforst manor in Vinxel/ Germany.On the left: Parse tree and the belonging resulted interpretation of two observed floors (Fl) with windows (w) and doors (d).On the right: An updated parse tree after the observation of a further floor.Algorithm 3: TRANSFORM PARSE TREE -Construct or update a parse tree Input: W ACF G . . .Weighted attribute context-free grammar Input: N odes . . .Observed and interpreted building parts Input: T ree . . .Parse tree (Can be empty) Output: T ree . . .most likely parse tree 1 [CN odes] = FIND CORRUPTED NODES(T ree, N odes); 2 [CorrN odes] = CORRECT ATTRIBUTES(T ree, CN odes); 3 DELETE NOT SUPPORTED NODES(T ree, CN odes); 4 [T ree] = PROPAGATE ERROR ON DELETE(AG, T ree); 5 [T ree] = ADD NODES(T ree, CorrN odes); 6 [T ree] = PROPAGATE ERROR ON ADD(W ACF G, T ree);

Figure 6
Figure6illustrates a scenario of an incremental interpretation of the main fac ¸ade of the Frankenforst manor in Vinxel / Germany.At first, two floors have been observed.The according interpretation so far is shown as a parse tree on the left side of the figure.The incorporation of a new observed floor triggers an adaptation of the tree structure as shown in the parse tree on the right.The shape parameters are also adequately updated.