3D URBAN GEOVISUALIZATION: IN SITU AUGMENTED AND MIXED REALITY EXPERIMENTS

In this paper, we assume that augmented reality (AR) and mixed reality (MR) are relevant contexts for 3D urban geovisualization, especially in order to support the design of the urban spaces. We propose to design an in situ MR application, that could be helpful for urban designers, providing tools to interactively remove or replace buildings in situ. This use case requires advances regarding existing geovisualization methods. We highlight the need to adapt and extend existing 3D geovisualization pipelines, in order to adjust the specific requirements for AR/MR applications, in particular for data rendering and interaction. In order to reach this goal, we focus on and implement four elementary in situ and ex situ AR/MR experiments: each type of these AR/MR experiments helps to consider and specify a specific subproblem, i.e. scale modification, pose estimation, matching between scene and urban project realism, and the mix of real and virtual elements through portals, while proposing occlusion handling, rendering and interaction techniques to solve them.


INTRODUCTION
Urban design and planning require knowledge and tools coming from various technical domains implying many users (citizens, architects, urbanists, stakeholders, etc.) to imagine and build urban spaces.From a geographic information sciences perspective, addressing the geovisualization of complex urban spaces is essential to favor visuospatial thinking and to support the cognitive processes of understanding and decision-making.3D geovisualization is widely used to visualize and interact with urban geospatial data (Döllner et al., 2006, Drettakis et al., 2007): it requires to manipulate heterogeneous data representing spaces and related thematic data (social, historical, environmental, etc.), in a cross-domain approach between various populations of 'urban designers', such as architects, urbanists, planners and citizens.
The concept of Augmented Reality (AR) is widely defined to refer to "an otherwise real environment "augmented" by means of virtual (computer graphic) objects" (Milgram and Kishino, 1994).AR is considered as "a radically new user interface paradigm, which aims to amplify a user's sensory perception directly by supplementing computer generated, mostly visual information" (Schmalstieg and Reitmayr, 2007).Three main characteristics are highlighted in order to design an AR application: it should combine real and virtual elements, be interactive in realtime and provide a 3D model of the world (Azuma, 1997).Beyond tablets and smartphones, AR devices on glasses make hand free and ease interactions with the virtual content.Going even further, when local environment can be analyzed in real time while adapting the rendering of the scene, AR becomes Mixed Reality (MR) enabling the blending of the physical world and virtual elements (Milgram and Kishino, 1994).AR/MR are thus meant to enrich not only visuospatial thinking and mental projections in space and time, but also a full sensory experience by facilitating data interaction and immersion.
We assume that AR and MR are now sufficiently technically advanced to qualify as a relevant medium for operational uses of geovisualization, in particular for urban design.In line of major scientific issues in geovisualization, we face difficulties to bring close together physical rendering and usability aspects in such a context.Therefore, in this paper, we aim at addressing and adapting geovisualization methods to AR/MR for the manipulation of 3D urban scenes.We propose to design a use case of an in situ MR application, aiming at facilitating the design of the city.We highlight that this use case design requires to re-visit the existing 3D geovisualization pipeline and to control some adaptations of data preprocessing, occlusion handling, rendering, and interaction processes.In order to demonstrate these potentialities, we present four AR/MR experiments, enhancing occlusion handling and interaction issues.At this stage, we assume to demonstrate our capacity to transfer and make more generic knowledge and methods from geovisualization to AR/MR and to propose relevant techniques to further manage an in situ mixed reality application.

RELATED WORK & REQUIREMENTS FOR AR/MR
There is an extensive literature on Augmented Reality (AR) in computer graphics, human-computer interaction and geographic information sciences, amongst others.Here, we aim at highlighting the existing contributions of such scientific and technical domains, in order to identify existing complexity when transferring knowledge and methods from 3D geovisualization for classical uses to AR/MR contexts.
Expected AR applications could benefit from 3D geovisualization knowledge and methods, while main issues of 3D geovisualization, especially regarding data rendering and interaction, are also revisited when visualizing on AR/MR devices.
2.2 Specific 3D rendering issues on AR devices 2.2.1 Pose estimation: A precise tracking of the viewer's pose (position and rotation) with no perceptible lag is the basis of an AR system.In (Zhou et al., 2009), based on an initial guess from the GPS and gyroscope, a model-based silhouette tracking and sensor fusion approach is used to refine the viewer's pose to provide a stable augmentation.More recently, (Armagan et al., 2017) refine in real time the position and orientation of a monocular system by aligning the semantic segmentation of its images with simple 2.5D building maps of building footprints.(Piasco et al., 2018) present a survey on Visual-Based Localization methods that are able to cope with the unavailability of direct positioning sensors.Furthermore, the AR device may be equipped with more or less sensors, such as a GPS, gyroscopes, a compass, an Inertial Measurement Unit (IMU), color cameras, depth cameras, LiDARs, etc., that may be fused to estimate the viewer's pose.

Lighting and occlusion:
Lighting and occlusion are the two principal tasks related to computer graphics when dealing with AR.They are critical as they are responsible for the visual coherence of the added virtual objects in the real world, especially if the augmentation is sought to be photorealistic.(Pessoa et al., 2010) present a solution providing realistic lighting of added object using an Image Based Lighting approach.(Hebborn et al., 2017) recently present some interesting results regarding occlusion handling at a relatively small computational cost.Even more complex than the addition of a new virtual object, the seamless removal of an existing real object demands to synthesize the pixel values behind this object.(Said et al., 2018) propose such a diminished reality application which rely on inpainting the image behind the object and the physical simulation of the specular effects.

Requirements on the 3D geovisualization pipeline for AR/MR
Three aspects of AR/MR contexts have to be taken into account, according if we visualize a 3D model of an object, at its position and 1:1 scale, in reality (in situ) or at any other place (ex situ), or at any other scales, outside (outdoor) or inside (indoor) buildings, and finally if the AR application is manipulated by one user (mono-user) or several ones (multi-user).These contexts provide different constraints and will imply technical adaptations.
2.3.1 In situ VS ex situ: In situ or ex situ AR/MR applications combine different proportions of real and virtual elements.
When visualizing an urban project in situ, there is no need to represent the surroundings of the project, because we visualize augmented data inside the real urban place.When discussing the same project ex situ, it can be required to represent also the geographical context of the project, in order to facilitate the recognition, the realism or the understanding of the project.Therefore, the data and 3D city models may be different, implying different rendering methods.This aspect of the context of use can be linked to the dichotomy introduced by (Hugues et al., 2011) regarding the differences between augmented maps and augmented territories.In situ AR/MR applications will experience 'augmented territories' when ex situ AR/MR applications will provide 'augmented maps or features' out of their real environment.

Indoor vs. outdoor:
The indoor or outdoor localization of the augmented reality experience will mainly affect rendering techniques.Indeed, lighting conditions are very different when being indoor or outdoor.The choice of the AR/MR device will be determinant regarding this, because of the different visualization capabilities of devices in an outdoor context.Outdoor applications ideally should include re-lighting techniques aiming at enhancing augmented data and integrating them realistically into the visualization of the reality (Lensing and Broll, 2012).

Mono-User VS Multi-User:
The main difference with multi-user AR/MR applications is to provide collaborative interactions between the different users, in addition to the individual interactions available for each user, provided in any cases.Multiuser interactions enable users to work on the same AR features (such as a 3D model or data coming from real time digitization, etc.) or to allow distant users to exchange informations (contextual data, augmented data, etc.), which will imply to combine in situ and ex situ contexts in the same application.A strong benefit when using AR/MR devices in a multi-user context would be that collaborators can see precisely where the others are looking, having their vision axis intersecting the model, as a visible dot for example.All these collaborative interactions imply to foresee how users will collaborate among themselves and how augmented data should be modified and updated regarding these interactions (Ismail and Sunar, 2009).

Use context: urban design
Main steps in the urban design are related to an imagined, simulated or in progress territory.Indeed, the whole process of urban design is dedicated to plan and preview a future version of the urban spaces.This process aims at building up new neighborhoods, at designing new houses, buildings, schools, factories, etc., at renovating old or aging constructions, or at planning interior designs.It implies to enable city planners and architects to preview and visually analyze their projects for the targeted spaces.Connecting AR/MR devices and geovisualization knowledge and methods should then be relevant, in order to help them to simultaneously visualize the targeted spaces and their indoor or outdoor projects, such as to interact with urban and thematic data.AR/MR visualizations would be useful in communication or participation steps in the urban design process: citizens or stakeholders would be able to visualize the new project, to look up several versions of the project, and would be involved in the design process by participating to some aesthetic or structural choices, such as the selection of color schemes or construction materials, or the composition of shapes, volumes or perspectives (Petit, 2015, Brasebin et al., 2016).In the specific case of urban renovation, AR/MR will also be useful to support the analysis of the visual impact of the destruction or the change of a part of the city (building, monument, sculpture, urban furniture, facades, etc.).

Augmented data interaction
The global challenge behind the expectations of the final users is to be involved in the process of re-drawing and re-thinking the urban spaces.As a citizen, a stakeholder or a city planner, the users should be able to experiment various designs, at various scales of the urban space (city, urban blocks, buildings).As a consequence, combining geovisualization and AR/MR are meant to be immersive and realistic enough to make users understand differences between the reality and the mixed visualization they experience without being confused by representation or interaction artifacts.

Augmented
Data interaction: Geovisualization through AR/MR devices should provide easy to handle augmented data in order to enhance users' comprehension of their 3D geometry.Moreover, the navigation into the resulting mixed environment should be intuitive and immersive, enabling the users to manipulate scale and time dimensions of the phenomenon they visualize.That entails to provide intuitive interactions to the users, such as the drag&drop like displacement of 3D models, the rotation or resizing of such features, etc.The understanding of complex 3D volumes within the urban space is at stake here.

Augmented Data personalization:
AR/MR devices should be exploited to enable users to graphically customize the mixed environment they experience.The users should be able to choose which scenarios they are visualizing, between several urban projects, several stages of urban plans, etc.Therefore, geovisualizations should be associated to interactions allowing to choose between different augmented datasets, to browse and customize them, to render them in real time or resize them easily.As a consequence, a diversified catalog of styles, stylization methods and rendering techniques should be available.Involving the users in aesthetic, functional or structural choices require to provide them intuitive interaction to do so.

Towards an in situ MR application: the targeted user experience
We introduce an in situ, mono-user and outdoor use case, in order to offer the possibility to design the city.This experiment combines computer vision, computer graphics, geovisualization and MR to offer a tangible fusion of real and virtual reality for outdoor purposes.The targeted users may be citizens, urban designers, architects, urbanists but also scientists.The scenario would take place into the city and would support the decision-making step of an urban design project, when re-designing buildings, while removing and replacing buildings by another ones.
The sequence of the immersive scenario will be the following.
The design of the city would work as illustrated by Figure 2, with the scenario of the removal and change of a building.When walking through the city, the users can decide to modify a specific area or a complete building (Fig. 2, a).They can select the area of an existing building by taping on it to create a removal polygon (Fig. 2, b), or select a building within the related GIS database.
Then the selected part or the whole building disappears (Fig. 2,  c), enabling the users to see the world as if this part never existed: the view part of the removed real building will be replaced by a synthesized view, from their position simulating the removal of this building, thanks to the integration of previously digitalized cities.Finally, the users can choose to replace the resulting empty spot by some other 3D models, for instance a detailed BIM model.
Our targeted in situ MR application requires to manage and adapt the usual existing 3D geovisualization pipeline, as well as to implement some indoor experiments to validate our proposition.

COMBINING GEOVISUALIZATION AND AR PIPELINES: A REQUIRED ADAPTATION
The implementation of the use case presented in the previous section requires to potentially adapt each following module of the 3D geovisualization pipeline presented below (Fig. 1). Figure 3 presents the processes that will be especially, revisited and extended (1,2), such as and adapted in order to take into account specific input informations coming from the AR/MR devices (3, 4), for AR/MR contexts: 1. 6D  AR/MR geovisualizations may require the same kind of datasets as regular geovisualizations such as 2D and 3D datasets, possibly with dynamic content and with metadata information: vector datasets, 3D models with or without textures, pointclouds, etc.We identify two main requirements for datasets to be used in AR/MR: • a lag-free AR visualization demands datasets that are relatively lightweight to load (either locally or streamed from a server at a suitable level of detail) and efficient to render (considering the hardware currently present on AR devices).• In situ AR of photorealistic datasets is very sensitive to the color and geometric details of the virtual elements.Suitable levels of detail are nowadays achievable with LiDAR and photogrammetric techniques.

Positioning the user in 6D
When combined with AR, the classical mouse or touchpad controllers of a geovisualization are replaced by the user movement, tracked by the AR device.This positioning is however typically relative to a local frame, whereas geovisualizations generally require an absolute positioning relative to the global Earth frame.
For instance, MR devices such as the Hololens and the recently announced Magic Leap One, keep track of their own relative positions and orientations using integrated sensors, whereas absolute positioning may only be provided by less precise sensors (GPS, compass), typically with an accuracy of about 5 meters, but may not always be available (e.g.indoor).This imprecise localization is generally sufficient to trigger the loading of datasets neighboring the user, but is far too imprecise for AR.These neighboring datasets may however prove to be useful for further automatic algorithms for the online pose refinement.Such algorithms are however out of the scope of this paper and we rely here on user interactions to initialize the absolute pose of the AR/MR device (Section 4.4).

Occlusion and visibility handling
While 3D geovisualization helps to understand volumes and morphologies by getting closer to the perception of the real world, it may also suffer from visibility issues not facilitating perception and cognition processes.Our in situ experimentation implies to provide a representation of how the reality would be if some features had been removed: 3D visibility calculations should be done to identify the part of the reference database that was hidden by the building the user wants to remove.The real time position of the users has to be taken into account to compute what they are seeing and what they would see when removing parts of the reality.That entails to calculate which part of previously hidden buildings should be added as augmented data.
Figure 4 shows the different fields of view for the users between the real world and what's added to fake the removal of objects.The benefits of using masks is that it is not necessary to compute the visibility of the virtual elements in real-time, natively done by the GPU: therefore, we do not rely on the local mesh acquired by the depth sensors of the AR/MR device, which are often prone to noise, errors and over-simplifications and have a range that is too limited for street-scale AR.
Figure 4. Removing reality elements using background 3D realistic models for substitution.
Conversely to removing real objects, adding new data such as building models is a much easier process.To handle the occlusion with the real world, we use the local mesh provided by the AR/MR device for depth testing, or may use portals and masks if the local mesh is of insufficient quality.

Rendering challenges through AR/MR devices
The first challenge is that the dedicated hardware on AR/MR devices are limited in terms of storage space, data bandwidth and graphics performance relative to desktop PCs for cost and power concerns.For instance, rendering pipelines are then usually implemented in OpenGL ES/WebGL rather than using the featurerich OpenGL.
In situ MR requires photorealistic rendering for many cases to cope with reality environment in a natural way.Not only does it need high resolution textured models but also close to Physically Based Rendering where we adapt the light on the virtual model to adapt to reality real time illumination.This is a complex task in MR because the model is to be visualized within the real world with its own shadows and global illumination.We aim at facilitating this task as the solar time is available as well as a quite precise 3D geometry surrounding the users: we can mathematically get the position of the sun and estimate shadows (with the caveat that see-through AR/MR devices are currently additive only so that it is not directly possible to cast shadows on real objects).Another lead consists in taking into account the video stream of the glasses to analyze the illumination in real-time.To seamlessly insert a digitized copy of real building in the case of see-through devices, we also use a smooth blending on the border so the transition between real and virtual is less perceptible.
Stylization of augmented data.AR/MR applications should provide the possibility to choose among styles with several levels of realism and abstraction.Catalogs of styles should be integrated in the rendering engine of further AR/MR devices, in order to enable designers to choose suitable stylizations or to get several possible stylizations.In particular, we aim at considering: • photo-realistic styles using projective and detailed textures for example, to visually integrate augmented data into the reality; • abstract styles to emphasize simulated or foreseeable urban shapes or to soften some shapes and area in the 3D scene; • sketchy styles to enhance in-progress urban projects.
In our AR/MR experimentations, augmented data contain mainly 3D models, coming from a 3D reference database or from an urban project, requiring to be rendered in a 3D environment.Therefore, we call for an extension of existing stylization specifications from the Open Geospatial Consortium (OGC), such as Style Layer Descriptor (SLD) and Symbology Encoding (SE) specifications (Lupp, 2007, Müller, 2006, Bocher and Ertz, 2018), to 3D standards formalizing 3D styles, as started in (Christophe et al., 2016, Brasebin et al., 2016), such as illumination properties and possible expressive stylizations.

User interaction with data
A crucial element in geovisualization concerns the user interaction with data and they are certainly not less important in AR/MR with the extended possibilities offered by the use of natural gestures.Four types of interaction are considered here: 1. Scale modification.Scale has a very special meaning in AR/MR as we can finally interact with elements at their original scale.Hence we ideally need an intuitive scale manipulation tool using both hands of the user: a first interaction consists in scaling uniformly on the three axis, a second interaction is to reach specific scales enabling visualizing the entire object sitting on a desk as well as the immersive visualization while walking inside it.2. Position.Positioning objects in AR/MR is pretty satisfactory since the 3D cognition of the environment is rather natural and thanks to the fact that hand gestures are in 3D as opposed to the usual two degrees of freedom computer mouse.
Hence, as we mentioned just above with scale modification, we propose to use directly hands movement to control the positioning, but this time with only one hand using an exponential motion allowing for precise small centimetric motions as for much larger motions.3. Real and virtual edition.In order to let the users choose where they need to visualize augmented data, we can provide the possibility to edit masks that hide virtual elements or, on the contrary, portals that hide the reality.Such a functionality will be useful for several applications.For an indoor use, we can imagine to enable the users to select windows or doors of the room in which they are manipulating the AR/MR device.These masks will serve as portals, through which the users will be able to visualize an alternative reality.Another interesting aspect of those masks is their abilities to change the rendering on a specific part of the scene, such as adding transparency to a wall, showing it in wireframe, etc.Also, it is worth mentioning the possibility to not only use fixed positioned masks but also masks more like lenses that could move according to the viewer glance.4. Style Interaction.AR could therefore be useful to access several upstream style configurations such as those recommended by the authors, while enabling the users to browse several parameterization scenarios for a given project: the users would be thus able to choose between colors, material, textures, etc. by visualizing them integrated into the reality.

AR/MR EXPERIMENTS
We implemented four types of AR/MR experiments to address the different requirements identified as subproblems to target our use case of an in situ MR application, designed in section 3: • Ex situ experiments, based on the visualization of an underground network (5.1) and the morphing of a street into a corridor (5.2), • In situ experiments, based on the implantation of a new 3D building (5.3) and on "see-through" effects (5.4).
These four experiments required addressing specific complexities of the geovisualization pipeline, such as the use of scale modification (5.1), the adaptation of virtual to real, i.e. the pose estimation (5.2), the matching between the realism of the scene and the urban project, i.e. the occlusion handling (5.3) and the mix of real and virtual elements through portals and lenses (5.4).All these experimentations where based on the Hololens device and the HoloJS framework.Illustrative videos are available in the supplementary materials1 .

Ex situ experiment: wireframe underground network
We experimented the ex situ visualization of a 3D model of the underground network of a sewer system.Figure 5 shows the visualization of these digitalized underground networks: at the size of a table enabling different collaborators to turn around it and visualize the full model instantly (Fig. 5, left) and at the 1:1 scale where they can then walk inside the model and have a totally different user experience, immersing themselves into it (Fig. 5, right).This experience at 1:1 scale is very effective to grasp the real volumes, such as the ceiling height.The tool provides to change smoothly from one scale to another one.More details may be found in (Devaux et al., 2018).This experiment validates the technical developments on the Hololens device and the ability to render simple wireframe meshes, with scale changes.A basic interaction has been developed to switch from reduced scale to real scale triggered by a pinch gesture.The reduced scale model is positioned in front of the user at initialization time and anchored to the room so that he can move freely around it.Its scale is computed to fit a given bounding box, so as to give an overview.Reduced scale and real scale models are manipulable using simple hand gestures for translation and rotation.A promising follow-up research would be the ability to automatically place or warp the real scale model according to the geometry of the immediate neighborhood of the user, as sensed by the device.This is closely related to recent ReDirected Walking (RDW) techniques, that warp large virtual worlds to enable real walking in small real environments, such as (Sun et al., 2018), except that the visibility of the real environment in AR/MR raises new questions and opportunities to adapt the virtual model to the real world.

Ex situ experiment: morphing a textured street model into an indoor corridor
We experimented the ex situ visualization of a street into an indoor corridor.The input augmented data used for this experiment is a 3D textured mesh of a street, derived from a mobile-mapping acquisition.This street is visualized by morphing its geometry to the volume of a local corridor using the user interactions, designed for absolute georeferencing plus a scaling interaction to match the corridor and street widths.Once the 3D model geometry matches the corridor, the user is able to walk into the street at a scale close to the original.In this experiment, using urban space analogies by morphing augmented data to the reality would help the user to feel inside the augmented data and to be immersed in the resulting mixed environment (Fig. 6).
Figure 6.Morphing a street into an indoor corridor.
As opposed to wireframe views, textured views mask more the reality.This is thus a slightly different context for the adaptation of the RDW techniques discussed in the previous experiment.Apart from trying to make the warping unnoticeable, different tradeoffs could be explored as well and evaluated with users, such as a noticeable warping that matches more precisely the real environment.

In situ experiment: implantation of a 3D building model
This experiment aims at helping the users to apprehend an alternative reality by adding a 3D model as augmented data into its real environment, using precise occlusion masking as before.The stake is to provide a convincing visualization of a future building, in order to involve the users into the urban design process.In this experiment, we load a BIM 3D model (Fig. 7, top-left) and enable the users to place it into the reality (Fig. 7, top-right), at its future position by drag & drop like interaction.The users are also able to resize the building 3D model.Figure 7 (bottom) illustrates the resulting mixed visualization when the building is implanted into the reality.

In situ experiment: "see-through" effects
We experimented a series of see-through experiments that can be considered as a derivation of our in situ MR scenario, in an indoor context.These experiments aim at enabling the users to see augmented reality through real objects, such as walls using virtual portals.These see-through experiments highlight the need for immersive visualization, even in a small part of the mixed environment.Moreover, they are an interesting example of mixed in situ and ex situ visualization, when the augmented data, visualized through previously captured virtual portals, come from remote, simulated or imagined places (Fig. 8).
Figure 8. See-through experiments: better weather simulation through 3 portals placed at the windows (left), desert landscape through a door (right) 5.4.1 Creating rectangular portals: 3D vertical rectangles, defined as 3D rectangles which edges are all either vertical or horizontal, may be determined geometrically using two opposite 3D points along one of their diagonals, as illustrated by Figure 9.They are geometrically simple but still very useful to model many indoor features that could be turned into portals to see augmented content: doors, windows, walls, etc.We implemented the creation of 3D points in AR by intersecting the ray from the gaze of the user with the local mesh provided by the AR device.The selection proved to be more precise when the user could continuously see the 3D point to be created as a 3D cursor superimposed on the AR view, that could be moved and adjusted by moving the head position and orientation and selected by a simple click/pinch interaction.The main caveat of this method is its dependence to the quality of the local mesh, which could be mitigated if priors were to be introduced such as the planar vertical geometry of the supporting walls.
5.4.2Selecting 3D geometric models: Alternatively, when 3D models of the reality are available, such as a 3D city model in the outdoor context or a 3D building model in the indoor context, some parts may be selected with a single click/pinch user interaction that retrieves the selected object with raycasting the user gaze against the 3D database of objects.Then, the geometry of the selected object may be used as a portal to render the augmented content, thereby replacing the selected object with the augmented content.

CONCLUSION
We presented four types of experimentations investigated with AR/MR in a 3D urban geovisualization context, in order to next be able to implement our more complex scenario of designing the city, in or ex situ.Each experimentation aimed at succeeding into one important design and implementation stand point, regarding: 1. the scale modification, for the ex situ underground visualization (5.1), 2. the adaptation of virtual to real, i.e. the pose estimation, for the morphing of the street into a corridor (5.2), 3. the matching between the realism of the scene and the urban project, for the implantation of a 3D building (5.3), 4. the mix of real and virtual elements through portals and lenses, for the "see-through" effects (5.4).
These experimental steps were required to identify the complexity underlying 3D urban geovisualization with AR/MR devices, as well as the needed adaptation of existing geovisualization pipelines.This exciting idea where the users can potentially manipulate the city in situ, i.e. removing real objects and adding new ones, in order to apprehend the impacts of urban projects, is still at stake.Stylization issues have not been specifically addressed here, but could be relevant to handle thereafter.Besides, supplementary tools of spatial analysis may be added in order to help the users to handle such a scenario, with other thematic data in order to consider issues of visibility, urban volumes and morphologies, noise propagation, respect of urban regulation, etc.
Experimental implementations must be continued based on intermediate evaluations and experimentations with final users.Adjusting significant and useful urban and thematic data with the rendering capabilities and interaction modalities of the system, is necessary in order to assess the assumed efficiency of AR/MR devices to improve the user experience for urban design.
Our purpose is to enhance and extend the capability of the system to be generic, extensible and interoperable with other geospatial data, styles and use cases.AR/MR devices may also be relevant to optimize data acquisition processes, while facilitating the annotation, the measuring and the digitizing of 3D geospatial data, and to propose a relevant environment for real-time 3D acquisition.
The future of geovisualization is inspiring thanks to the new possibilities offered by new interaction and visualization abilities, most of all, natural hand gestures and holographic rendering.MR devices may also transform users into mappers, continuously digitizing their local environment.It opens the way to new possibilities such as "teleportation" where one can evolve in beforehand digitized city or even unknown places, mapped in real-time with the recent advances in Simultaneous Localization and Mapping (SLAM) techniques (Durrant-Whyte and Bailey, 2006, Bailey andDurrant-Whyte, 2006) and light acquisition platforms like drones.

Figure 2 .
Figure 2. Use case design: (a) Original view in situ, (b) The user selects a part or an entire building to remove, (c) After removal, the real building is no more visible, (d) The user adds a part of a 3D Building Model.

Figure 7 .
Figure 7. 3D building model (top-left); real view of the construction site through an office window (top-right); implantation of the building based on augmented views placed on the construction site (bottom).

Figure 9 .
Figure 9. Capturing a vertical rectangle mask by 2-point picking on a white-board (left, then middle): the augmented data seen through the portal is indeed the photogrammetric model of the neighboring room, acquired by a DSLR camera and placed at its own real location (right).