A REQUIREMENT ANALYSIS ON EXTENDING SEMANTIC 3D CITY MODELS FOR SUPPORTING TIME-DEPENDENT PROPERTIES

: Semantic 3D City Models are used worldwide for different application domains ranging from Smart Cities, Simulations, Planning to History and Archeology. Well-deﬁned data models like CityGML, IFC and INSPIRE Data Themes allow describing spatial, graphical and semantic information of physical objects. However, cities and their properties are not static and change with respect to time. Hence, it is important that such semantic data models handle different types of changes that take place in cities and their attributes over time. This paper provides a systematic analysis and recommendations for extensions of Semantic 3D City Models in order to support time-dependent properties. This paper reviews different application domains in order to identify key requirements for temporal and dynamic extensions and proposes ways to incorporate these extensions. Over the last couple of years, different extensions have been proposed for these standards to deal with temporal attributes. This paper also presents an analysis to which degree these extensions cover the requirements for dynamic city models.


INTRODUCTION AND MOTIVATION
Semantic 3D City Models describe not only spatial and graphical aspects of the city objects, but also provide ontological structure including thematic classes, attributes, and their interrelationships.Hence, such information models make it possible for applications and simulation tools to distinguish urban objects (like buildings and streets) and use their rich thematic and geometric information for queries, statistical computations, simulations, and visualizations.For these reasons, many cities worldwide such as Berlin, Singapore, and Helsinki are creating and using Semantic 3D City Models.
Several organizations and standards provide semantic data models for modeling city objects which are being used for different applications.CityGML (Gröger et al., 2012) issued by the Open Geospatial Consortium (OGC) is one of the very popular standards for modeling and exchanging Semantic 3D City Models.This standard facilitates the integration of heterogeneous data from multiple sources and allows for the representation of geometrical and semantic attributes of the city level objects (such as buildings, streets, water bodies, vegetation etc.) along with their interrelationship to the other objects.INSPIRE (Infrastructure for Spatial Information in the European Community) (INSPIRE, 2013) is an initiative of the European Commission for developing a European Spatial Data Infrastructure.The INSPIRE Directive addresses 34 spatial data themes (such as Administrative units, Buildings, Transport networks, Land use, Geology, etc.) which are discoverable and interoperable through the implementation of a common set of standards, data models and web services.Industry Foundation Classes (IFC, 2016) is another very popular standard for modeling and exchanging Building Information Models (BIM).This standard has been developed by buildingSMART International and provides open properties may change w.r.t time.For example, a construction event leads to the change in geometry of a building (i.e.addition of a new building floor or demolition of an existing door).The geometry of an object can be further classified according to its shape, location, and extent, which can also be changed w.r.t.time.A moving car object involves changing only the location of the car object, however, a flood incident involves variations in location and shape of water.There might be other properties which change w.r.t.semantics of city objects, e.g., hourly variations in energy or gas consumption of a building or changing the building's type from residential to commercial.Some properties involve changes in appearances over the period of time, such as building textures changing over years or traffic cameras recording videos of moving traffic over definite intervals.3D City Models comprise relevant real world entities and also represent interrelationships between objects.Such interrelationships may also change over time.Hence, it is important to consider that the representation of time-varying data is required to be associated with these different properties.
Temporal variations involve time points mapped onto the specific attributes (i.e.spatial, thematic, topology, or appearance).Such mappings are often realized by discrete recordings or by using interpolation functions and involve quantitative changes which can be defined as a function of time.For example, varying energy consumption values of a building can be determined for specific points of time (i) in the past by querying a database for historic data, (ii) in the present by querying a real-time sensor, and (iii) in the future by a simulation software.However, there are other scenarios where features begin or cease to exist over different time intervals, for example, addition of a new building or demolition of an old building.Such scenarios involve qualitative changes and are fundamentally different from the quantitative changes.Such changes can not be defined as a function of time on a feature's property as the state of features changes.Hence, it is also essential to consider that Semantic 3D City Models handle both quantitative and qualitative changes within cities/city objects.Since both quantitative and qualitative changes involve variations w.r.t.time, it is also important to determine if they can be handled by the same mechanisms.

Time-dependent properties in Application Domains
Semantic 3D City Models (i) can give the spatial context to all information that is related to the physical entities in cities, and (ii) provide a means for interactive and spatio-semantic queries and aggregations for numerous application domains.(Yin et al., 2015) is an emerging field towards improving operational efficiency and quality of life of cities in the light of rapidly increasing urban population.This concept allows managing city resources such as energy and water with the help of advanced information and communication technologies such as sensors and the Internet of Things (IoT), Big Data, Cloud Computing, and also geospatial technologies.Another term "Digital Twins", originated from Manufacturing Industry, is also widely gaining attention.A Digital Twin is a digital counterpart of a physical asset, which collects information via sensors and IoT devices and applies advanced analytics and artificial intelligence to gain real-time insights about the physical asset's performance, operation or profitability (Grieves, Vickers, 2017).Digital Twins are key for Cyber-Physical Systems comprising smart machines and storage systems capable of autonomously exchanging information, triggering actions and controlling each other independently (Frontoni et al., 2018).Bridging the virtual and physical worlds together in this way can also help Smart City applications to improve decision-making and reduce risks by predicting issues before occurrence (Mohammadi, Taylor, 2017).

Smart Cities and Digital Twins Smart Cities
Several Smart City projects such as Helsinki 3D+1 , Virtual Singapore2 , and Future City Pilot Phase 13 highlight the importance of developing Digital Twins to improve operational efficiency by integrating various city objects with real-time sensor data streams.These sensors can be stationary such as Smart Meters and weather stations.Some of the sensors can also be non-stationary such as air-quality sensors mounted on a car measuring air pollution over different parts of a city at different time intervals or pedestrian flow analysis involving pedestrians moving into or out of a stadium (e.g.before or after a scheduled football match).Since all of the mentioned Smart City initiatives consider Semantic 3D City Models as an integral component of their infrastructures, it is highly important that the next generation Semantic 3D City Models support seamless integration with sensors and IoT devices.It will allow measuring real-time observations associated with city objects, which can be be used for performing advanced analytics.Requirement 1: Linking Sensors and IoT with city objects.There are several possibilities for establishing links between city objects and sensors.As shown in figure 1, one option (represented as 'A') is to link them by defining additional semantics such as Linked Data4 .Linked Data provides a structured way to define semantic information for interlinking different types of data.
It can be built upon standard Web technologies such as HTTP, RDF, and URIs.It can also be based on Ontology representation languages such as OWL and UML.However, this approach would add another representation to the already existing representations of Sensors as well as City Models.Furthermore, this would require additional management for the semantics of Linked Data.The other possibility (represented as 'B' in figure 1) is to define explicit links to city objects within the representations of sensors.This approach would allow associating time-dynamic observations directly to the city objects.For example, a sensor interface measuring real-time gas consumption of a building may comprise of direct links to the building object or building's room where the Smart Meter is installed.However, current sensor modeling standards do not allow defining the sensor's relation with the city object property.In this way, we can only name the feature of interest, which typically is an object and not the property of an object.For instance, if a building object has an attribute "gas consumption", the issue is that the sensor operator would have to maintain the link(s) to different geospatial datasets (e.g.INSPIRE, CityGML, and IFC), that are not under his/her control.The third option (represented as 'C') is to define explicit links to real-time observations within the city object.In this way, city object attributes can also be associated with time-dynamic sensor observations.For example, the above-mentioned attribute "gas consumption" will become dynamic by having defined links to the interface for the Smart Meter observations.Similarly, in an application related to Civil Engineering, Bridge Deformation Monitoring (Davila Delgado et al., 2017) is an example of predictive maintenance which involves a systematic measurement and tracking of the alteration in the shape or dimensions of the bridge as a result of stresses induced by applied loads over a period of time.In such cases, an explicit linking of involved sensors installed at the bridge would allow overriding the static attribute values of the Digital Twin of the Bridge object.
Requirement 2: Events and Alerts.In the applications related to disaster management or early warning systems, synchronous communication with sensors may be insufficient as the applications depend on asynchronous events, such as external communications, status changes, natural phenomena, or data updates.In such cases, automatic notification by using a publish/subscribe scheme are required where clients are notified with the desired information.For example, in the case of water management systems, the notifications of water gauges exceeding a threshold may prevent critical flood scenarios.Similarly, real-time processing of air quality data may alert citizens and administrators for critical air pollution situations.The explicit linking of city objects with real-time sensors and IoT enable users and applications to subscribe to data streams and be alerted in case of occurrence of any particular phenomenon.

Urban Mobility Requirement 3: Moving Objects.
Applications related to traffic management, navigation, aviation or maritime monitoring also require the overlay and integration of moving objects.For example, the project 'AERO-TRAM' (Hagemann et al., 2014) involve a mobile measurement system mounted on the roof of a tram and examine spatial distribution of pollutants (such as particle number concentrations and nitrogen oxides).Such cases require mapping locations and sensor observations w.r.t.time.The other example could be to access data on moving objects that passed through a certain area at a certain time after a disaster, therefore gathering information on density and flow of people and vehicles in a more timely manner.Hence, it is important to extend city objects in such a way that they can be mapped with moving objects.This approach requires modeling a data structure using which timeseries data can be exchanged with appropriate metadata to allow correct machine interpretation and thus correct use for further analysis.

Urban Simulations
Requirement 5: Supporting Complex Patterns and Schedules.
In many applications, it is not sufficient just to provide a means for the tabulation of time-value pairs.The applications may require patterns to represent dynamic variations of properties based on statistics and general rules.For example, energy applications can be used to study variations and patterns in the energy consumption of a building for weekdays, weekends, public holidays, or even customized time period (e.g. between 6 pm and 10 pm).In different simulations, such timeseries can also be used as a basis for defining schedules.For example, schedules in the energy simulations may be required for specifying set point values for the heating and cooling systems, or for defining the operational schedules of energy systems, ventilation, lighting, and electrical appliances.Similarly, in the cases of traffic analysis, a public bus line following a schedule can also have a repeating trajectory.
2.2.4 Urban Planning Semantic 3D City Models are considered as an important source of information in Urban Planning.As cities constantly evolve, it is necessary to record their changes over time and that each city object may be represented with its own life-cycle.For example, for a time sequence, a building may be constructed, modified, destructed and replaced by other ones.Similar needs can also be identified in serious gaming projects where objects of a scene may evolve according to a given scenario.For instance, a building may be represented in different states like "destroyed", "burned" or "partially destroyed" that can be called by the application in coordination with user actions (Chambelland et al., 2011).Some applications may also address to work with planning alternatives of buildings or other structures, e.g. for comparison by a reviewing body.The planning alternatives are not different versions of actual structures at different times but different structures that might be substituted for one another.In urban planning scenarios, different planning authorities can also work with alternative planned versions at the same time to insert a newly generated object or delete or update any existing object (Samuel et al., 2018).
Requirement 6: Managing Alternative Versions.Hence, it is essential for Semantic 3D City Models to extend in such a way that their successive versions represent the state of all features of the entire city at specific points in time.In addition, the authorities can work, in parallel, with different workspaces or branches to insert, delete or modify the objects.Such additions can be merged with the earlier versions of city objects to form the final versions leading to multiple possible futures.However, for cross-domain applications, it is important to exchange the city models along with their parallel versions to perform, e.g."what-if" analysis.This requires (i) managing all the parallel versions within one dataset and (ii) interoperable data exchange format in order to be interpreted by different applications and software systems.

History and Archeology
Semantic 3D City Models also play a role in the documentation and reconstruction of both historical and contemporary events.Examples include crime scene and accident reconstructions, representation of battles and other historical events, archival descriptions of historical structures before demolition, documentation of construction and demolition of buildings (Pfeiffer et al., 2013).Each of these involves a sequence of versions of a "reality" at a certain time or in a certain time sequence.In such cases, events are often reconstructed from conflicting and incomplete evidence.
Requirement 7: Managing Historic Versions.Such scenarios require enabling backward compatibility for Semantic 3D City Models to handle multiple representations of the past of a city.A given date may be a starting point to imagine the past and constructing several scenarios.Hence, it is also required to manage and exchange all the historic versions within one dataset.Such backward compatibility is also beneficial for Archeology, for instance, for urban reconstruction of ancient cities, modeling of archaeological 3D objects and their attributes, managing excavations, testing reconstruction hypotheses, and analyzing development of sites over time.

EVALUATION OF 3D CITY MODELS
This section reviews the current state of three popular standards for representing Semantic 3D City Models and evaluates them against the key requirements identified in the previous sections.

Direct links to Sensor and IoT Devices
Different applications and use cases require linking city objects with real-time sensor and IoT observations.An approach proposed by (Zhu et al., 2016) allows integrating city models based on the CityGML standard and spatio-temporal air quality data retrieved using the international standard Sensor Observation Service (SOS) (Bröring et al., 2012) issued by the OGC.The demonstration allows visualizing dynamic variations of pollutants together with static city objects.Similarly, another architecture developed by (Santhanavanich et al., 2018) supports integration of heterogeneous sensor data with CityGML based models.The sensor data included pedelec usage from a Smart Electric Bike, user fitness level from a Smart Watch, and weather data from an Open Weather Portal and integrated and visualized with city objects using the OGC SensorThings API standard (Liang et al., 2015).Although both studies successfully demonstrate coupling of real-time observations with city objects using open and interoperable standards, the main focus is on visualization of time-dependent properties.Another concept called CityGML Dynamizers (Chaturvedi, Kolbe, 2016) not only allows establishing explicit links between sensor data and the respective city object attribute, but also allows overriding those static attributes by dynamic sensor observations.For example, if a building has a Smart Meter installed for measuring real-time gas consumption and the Smart Meter readings are accessible via standardized web services (such as OGC SOS or SensorThings API), the Dynamizer can be defined for the specific building or room having explicit links to the sensor based services, and the respective gas consumption attribute of the building can be overridden by the real-time sensor observations.CityGML Dynamizers have been successfully implemented as an Application Domain Extension (ADE)5 and are planned to become a part of CityGML 3.0.
Although IFC does not provide any in-built module to deal with Sensor and IoT data, many researchers have made developments in this direction.A web based system called "Otaniemi3D" (Dave et al., 2018) provides information about energy usage, occupancy and user comfort by integrating BIMs and IoT devices through open messaging standards (O-MI and O-DF) and IFC models.Another conceptual web information service framework (Wang et al., 2013) demonstrates the idea for Smart Building by combining live sensor data based on the OGC standards with IFC models.By knowing the building status at any given time and location, which is largely unseen to most users, it is possible to change occupant behavior, improve building safety, avoid unnecessary energy consumption and facilitate better working environments.
Considering the popularity of RESTful architectures, an extension has been proposed for the OGC SensorThings API standard to be considered as a solution that meets the legal obligations stemming from the INSPIRE Directive, thus simplifying the process for extending existing spatial data infrastructures to the IoT (Kotsev et al., 2018).The proposal shares perspective on what should be done with regards to: (i) data encoding; and (ii) the use of SensorThings API as a download service for INSPIRE.

Managing Events and Alerts
Although CityGML does not explicitly support managing alerts and notifications, the integration with modern interfaces may allow achieving Publish/Subscribe capabilities.The new OGC Publish/Subscribe 1.0 standard (also known as PubSub)6 provides asynchronous communication across OGC service interfaces and data types, including coverages, features, and observations and enables users to subscribe to the real-time data streams.MQTT7 is also a lightweight messaging protocol designed specifically for IoT devices.It attempts to minimize network bandwidth and device resource requirements.Major interfaces such as SensorThings API already support MQTT extensions allowing users to subscribe to specific events.Furthermore, in order to perform real-time processing for gesospatial data, an extension of MQTT protocol called GeoMQTT (Herle et al., 2016) has been developed.CityGML Dynamizers and INSPIRE allow integrating with modern sensor interfaces such as SensorThings API and MQTT and, thus, making it possible to subscribe to specific events and receive alert notifications.However, no implementation currently demonstrates the Publish/Subscribe functionalities so far.

Moving Objects
The architecture proposed by (Santhanavanich et al., 2018) allows modeling varying locations of the bike using OGC SensorThings API, which can further be visualized with CityGML based models.The master's thesis (Ruhdorfer, 2017) propose an architecture to integrate and visualize traffic simulation results from a simulation tool VISSIM with city models based on the CityGML standard.CityGML Dynamizers also support variations in locations with respect to time using  in-line timeseries as well as by linking with external timeseries.However, no implementation for moving objects is provided yet.
IFC does not provide support for moving objects and trajectories.The integration with other standards such as SensorThings API might create possibilities for supporting moving objects, however, to the best of our knowledge, no such implementations exist.The specialized observation from the INSPIRE guideline TrajectoryObservation allows representing a series of measurements along a trajectory, for example, along a ship's track.Each measurement is made at a separate point along the trajectory and at a separate time.

In-line Support of Timeseries
The CityGML Energy ADE Working Group extended the CityGML 2.0 classes to timeseries explicitly (Agugiaro et al., 2018).The data model distinguishes between regular and irregular timeseries.In regular timeseries, the values have a defined start and end time and a constant time increment.In irregular timeseries, each value has an individual timestamp.
The timeseries values itself may either be stored directly in-line within the CityGML document or in a separate file with table structure.CityGML Dynamizers also provide a way to model such dynamic variations with explicit timeseries representations.Dynamizers utilize standardized encodings such as the OGC TimeseriesML8 standard.Utilizing TimeseriesML, the timeseries can be represented as interleaved time/value pairs or by a domain range encoding with the metadata of timeseries and timepoints.For example, interpolation and aggregation types for each point can be defined determining missing values or multiple values to specific time points.
IFC provides the support of timeseries using the in-built IfcTimeseries module.It allows a natural association of data collected over intervals of time.Within the specification, timeseries can be regular or irregular.In regular timeseries, data arrive predictably at predefined intervals.In irregular timeseries, some or all time stamps do not follow a repetitive pattern and unpredictable bursts of data may arrive at unspecified points in time.The modeling of buildings and their performance involves data that are generated and recorded over a period of time.Such data cover a large spectrum, from weather data to schedules of all kinds to status measurements to reporting to everything else that has a time related aspect.Their correct placement in time is essential for their proper understanding and use, and the IfcTimeseries subtypes provide the appropriate data structures to accommodate these types of data.However, IfcTimeseries cannot be used to make arbitrary buildings or object properties dynamic.
The INSPIRE data models provide a 'Specialized Observation' package which define ten specializations of observations of the O&M specification.One of specializations is the PointTimeSeries Observation, which represents a series of measurements at the same point, e.g.timeseries having regular measurements by a fixed station.Similarly, the MultiPointObservation is a specific type of Point-based observation.It is intended for cases in which measurements are made at a set of discrete points at the same time.For example, a sensor network reporting temperature at 10am.The points themselves are not on a grid but may be distributed in any manner, for example unevenly spaced around a coastline.

Complex Patterns and Schedules
The CityGML Energy ADE extends CityGML 2.0 classes by modeling four different types of schedules: (i) ConstantValueSchedule, specifying only one value for the complete time interval regarded, (ii) DualValueSchedule defining two different values, one for operating times and one for idle times, (iii) DailyPatternSchedule specifying different time periods within a year where each period is related with schedules for specific days of the week (e.g.week day, weekend, or a specific day of the week), and (iv) TimeSeriesSchedule, where any (regular or irregular) time series may be used.On the other hand, Dynamizers support complex patterns by introducing the concepts of (i) Atomic and (ii) Composite timeseries.Atomic timeseries represent dynamic values having time points that can be based on local/relative time reference systems.This allows mapping customized time values such as weekdays, weekends, seasons etc.Furthermore, Composite timeseries generally consist of timeseries that can be nested to an arbitrary depth.Such timeseries can be repeated several times to obtain specific patterns.For instance, in energy applications, an Atomic timeseries may be defined for a working day, a Saturday, and a Sunday (represented by A, B, and C respectively).Now, in order to reflect a pattern of energy consumption of the entire week, a Composite timeseries may contain five repetitions of Atomic timeseries A followed by single representations of timeseries B and C (represented as AAAAABC).The advantage in using such Composite timeseries is that we can express time-varying data for an arbitrary long time period by combining and repeating various patterns.
IFC and INSPIRE do not provide data structures to modern complex patterns and schedules based on available timeseries.

History and Version Management
A modification has been proposed to the CityGML schema by adding temporal information on buildings (Pfeiffer et al., 2013).However, this method allows registering only definite states.CityGML schema modification and possible standardized exports are not discussed.A new Versioning module (Chaturvedi et al., 2017) has also been proposed for the CityGML standard.It extends the CityGML data model and allows exchanging different versions and version transitions within one dataset.In this way, a complete history or evolution of the city model is supported by version transitions having bi-temporal attributes.Furthermore, the concept also allows managing parallel alternative designs of the objects at the same time.Like Dynamizers, the Versioning Module is also planned to become official part of CityGML 3.0.
The concept of temporal version is not currently implemented in IFC, but some extensions have been proposed.For example, the proposal by (Zada et al., 2014) suggests six existing entities from the IFC standard to be modified to represent as new entities within the IFC schema to support the idea of object versioning that holds the history of changes to objects of the BIM model.(Nour, Beucke, 2010) also propose an approach in which both object versioning and IFC model are integrated together in an open multidisciplinary collaborative environment.Object versioning gives the possibility to have several versions of the content (attributes' values) of an object.The development of design in terms of addition of new objects, deletion of objects or modifications of attributes' values of pre-existing objects can be captured in a graph structure.INSPIRE defines several requirements and recommendations for modeling life-cycle information of spatial objects which include UML stereotypes and properties allowing for bitemporal modeling of geospatial objects.Furthermore, a separate property exists for denoting a specific version of a geospatial object.However, currently only exchanging the last version of spatial objects is supported by INSPIRE; historic versions cannot be provided yet (and especially not within one data file).

CONCLUSIONS
This paper reviews major application domains for Semantic 3D City Models ranging from Smart Cities, Simulations, Planning to History and Archeology.These domains include many applications where time plays an important role.Such time-dependent properties may be qualitative or quantitative in nature.For example, a Digital Twin of a city object requires seamless integration with IoT devices for real-time analytics.On the other hand, an urban planning application require handling different versions of city model and objects.This paper recommends key requirements to be considered for temporal extensions of Semantic 3D City Models.The requirements include direct linking with Sensors and IoT devices with eventing and alerting mechanisms, explicit in-line support of timeseries, schedules, and complex patterns, support of moving features, and history and version management.Further, the paper evaluates three standards OGC CityGML, buildingSMART IFC, and European Commission INSPIRE for the identified requirements.In the future, other standards such as IndoorGML and LandInfraGML can also be evaluated in similar ways.
CityGML 2.0, IFC v4, and INSPIRE specifications provide limited built-in support for handling with the listed timedependent properties.For example, INSPIRE and IFC provide modules to model timeseries, however, there are no data structures in order to define schedules and patterns based on those timeseries.Recognizing the importance of time-dynamic properties in various applications, several independent research work such as Dynamizer ADE, Versioning ADE, and Energy ADE have been carried out intending to resolve many of the named requirements.For example, it was attempted to integrate real-time sensor and IoT devices with objects.Similarly, the data models were extended to support versions of city objects.Dynamizer ADE and Versioning ADE are planned to be a part of CityGML 3.0.

Figure 1 .
Figure 1.Possibile ways for linking Sensors with City Objects.

Table 2 .
Evaluation of Semantic 3D City Model standards for the listed requirements.'+' denotes complete support and 'o' denotes limited support of the requirement by the standard specification or the related extension.