OPPORTUNITIES AND CHALLENGES FOR AUGMENTED REALITY SITUATED GEOGRAPHICAL VISUALIZATION

Augmented reality (AR) enables to display situated geographical visualizations, i.e visualizations that use virtual elements that are displayed in a geographical location. The place where the data is displayed complements the visualization. Many applications that take advantage of AR and situated visualizations exist, but they differ in the visualizations they present, their relationship to the geographic locations and goals. To better understand why and how AR based situated geovisualization is used, we review 45 papers coming from Human Computer Interaction, Visualization and Geographical Information Science venues that present such applications. Inspired by existing classifications, we characterize these papers according to the data they visualize and the geographical distance between the visualization and the data the visualization represents. This analysis reveals existing opportunities for situated geovisualization applications using AR.


INTRODUCTION
Situated visualizations, defined as "a visualization that is located and overlaid in a place where it is meaningful and where the place adds to the visualization" (White et al., 2006), can improve the viewers' understanding of the visualized data (White et al., 2006, ElSayed et al., 2015. Examples range from botanical visualization (White et al., 2006) to smart cities and tourism. Geographical information is especially suited to be visualized with such means, as it is inherently tied to an existing geographical position or object, for example to display navigation aids or support users on the exploration of archaeological sites (Gleue, Dähne, 2001).
Because augmented reality (AR) enables enriching the real world with virtual elements, it has a long tradition in multiple research areas of being used for situated visualizations and more specifically, for situated geographical visualizations. Furthermore, progresses in GPS positioning and computer vision techniques facilitate locating the user and its point of view to align virtual and real content. The development of off-the-shelf devices supporting augmented reality, such as the Hololens and mobile devices, facilitates creating such mixed visualizations. In this paper, we are interested in applications that present situated geovisualizations using augmented reality i.e. applications that virtually present data that contains geographical information overlaid on an environment that adds meaning to the visualized data.
There is a wide range of applications that present such augmented reality situated visualizations, and even if they use the same metaphor to visualize data, their objectives and means vary widely. Furthermore, existing classifications of augmented reality and map-based applications, only consider the applications they target, or their function. To our knowledge, no existing classification focuses on the visualized data, and the relationship between the visualization and the environment. * Corresponding author Nevertheless, we argue that understanding why and how augmented reality is being used for situated geographical visualizations could be helpful to define novel research perspectives. To achieve this, we review 45 papers coming from different disciplines that describe situated geovisualizations using AR. We are interested on the relationship between the data visualization, the data that is being visualized and the visualization geographical position. Therefore, we draw from the existing definitions of situated and embedded visualizations (Willett et al., 2017) and the distinction between physical data and abstract data to classify the papers. This enables us to extract current challenges and propose further opportunities for AR situated geographical visualization.
The remaining of the paper is organized as follows. First, we describe our methodology to select the papers to review and the categories we use to classify them. Second, we review the papers according to these categories. Finally, we extract current challenges and propose further opportunities for AR situated geographical visualization.

SELECTING AND CLASSIFYING THE APPLICATIONS
Because we are only interested in augmented reality applications displaying geographical information, we used a keyword based search using the words "augmented reality" and "geographical", and "situated visualization" to select the papers to review, in google scholar and the websites of the top journals and conferences of visualization, human computer interaction, and geographical information sciences namely the Conference on Human Factors in Computing Systems (CHI) and the journals Transactions on Visualization and Computer Graphics, International Journal of Geographic Information Science and Cartography and Geographical Information Sciences. We completed the selection with forward and backward referencing. We kept only papers presenting an application designed for a specific domain, that used augmented reality to visualize data in its context, visualize geographical information (i.e. the visualized  Figure 1. Schema representing the geographical referent (a) corresponding to a geographical position, the physical referent (b-c), and the physical representation (d). The geographical indirection represents the distance between the geographical referent and the physical referent. They can be the same (a) or a scaled referent such as a map can be used (c). The spatial indirection represents the distance between the physical referent and the physical representation.
data is tied to a geographical position) and published after 2000. For example, we do not consider applications that enable users to overlay fantasy characters on the environment (Tarumi et al., 2005), as these characters do not contain geographical information and they do not depend on the environment where they are being visualized. This resulted in 45 papers.
We draw from existing classifications to characterize existing geovisualization applications using augmented reality. Applications combining GIS and augmented reality can be classified according to their function; augmented maps or augmented territories (Hugues et al., 2011). Augmented maps support data exploration, providing information about the environment, augmented territories help users explore the environment, such as wayfinding applications. More specifically geovisualization AR applications can be characterized by three requirements: indoor vs outdoor, insitu vs exsitu, and mono user vs multi user (Devaux et al., 2018).
Situated visualizations are defined through logical world and physical world components (Willett et al., 2017). Logical world components include the raw data and the visualization pipeline. The raw data refers to a physical referent, and through the visualization pipeline it is encoded into a physical representation. Embedded representations are a subset of situated visualization, that are "deeply integrated with their physical environment" (Thomas et al., 2018). These data representations present higher or lower spatial indirection and temporal indirection. Spatial indirection refers to the spatial distance between the physical referent and the physical representation. Temporal indirection refers to the temporal distance between the moment the data refers to and the moment the visualization is displayed. Because we are interested on the relationship between the visualizations, the data they represent, and the environment where the visualization is displayed, we decided to classify the papers using the notions of spatial indirection and temporal indirection.
The definition of situated and embedded data representations encompass all raw data that is tied to a physical referent. If the raw data contains geographical data, the physical referent will not only be the object the data refers to, but also the geographical position of the object. For example, when visualizing the sells of products on a shelf by encoding the sells using the color of the product, (Willett et al., 2017) the physical referent is the product, and the spatial indirection will depend only on the distance between the physical representation and the product. It will remain the same if the shelf is positioned elsewhere. However, if we are interested on visualizing data that contains geographical information such as the solar radiation on all the apartments in a building, the physical referent is the building and its specific geographical location; if the building was located elsewhere, the data would not be the same.
Thus, we propose to introduce an additional indirection to the existing ones (Willett et al., 2017), the geographical indirection and a geographical referent. This is especially useful when the physical referent of the visualization is a facsimile i.e. a scaled model of an existing object (Willett et al., 2017). Geovisualization often relies on this metaphor to visualize geographical data, by overlaying data on a background map. Summarizing, for each situated geographical visualization, we have the previously introduced physical referent that can be a facsimile, the geographical referent that corresponds to the geographical position of the data, the spatial indirection and the geographical indirection, the distance between the physical referent and the geographical referent, as displayed in Figure 1. Adding those two concepts enable us to distinguish between situated visualizations that display geographical data using a referent from situated visualization where the physical representation is located on the geographical position of the data. Furthermore it takes into consideration the distance between the physical referent and the geographical referent.
The visualization also depends on the data that is being visualized. To take this into account, we distinguish between physical data, where the data can be mapped to a geometry and abstract data that does not have a straightforward physical representation (Gershon, Eick, 1997). For example, we classify the 3D model of a museum as physical data and the number of visits the museum receives as abstract data. Because we are interested in studying the existing applications according to the relationship between the visualizations they present, the data that is being visualized and the geographical position of the data and the visualization, we classify the papers according to the kind of data they present, the geographical indirection and the temporal indirection. The classification of all reviewed papers is depicted in Figure 2 for physical data and in Figure 3 for abstract data.

PHYSICAL DATA
We understand as physical data, data that has a straightforward visual physical representation, such as geographical objects (buildings, lakes, roads). Thus, the resulting visualizations represent some aspects of the physical representation of the object, more or less realistically. We separate geographically situated from geographically distant applications which display physical data.

Geographically situated
Geographically situated applications present no geographical indirection i.e the physical representation is in the same geographical position that the geographical referent. To classify the geographically situated realistic visualizations we use the different levels of temporal indirection. First, historical visualizations present a high temporal indirection because they present old data. Present data encompasses data that has a low temporal indirection, it is used to complement the environment with data that is not visible, such as additional points of view. Finally, augmented reality also enables to plan information, thus the temporal indirection is unknown. Planned data represents objects that do not exist but might exist on the future.
3.1.1 Historical data Situated geographical visualization enables visualizing the reality as it was in the past (Tarumi et al., 2009, Lee et al., 2012, Gleue, Dähne, 2001, Schnädelbach et al., 2002. CityViewAR (Lee et al., 2012) is based on a mobile augmented reality system to visualize historical 3D models of the city of Christchurch, previous to the 2010 and 2011 earthquakes. The models are visualized in situ, i.e, where they were before being damaged. The 3D models are accompanied by additional information. The arecheoguide project (Gleue, Dähne, 2001) also depicts historical 3D models in situ through a guided archaeological tour. Instead of using mobile technologies, it relies on a custom system based on a computer and a camera. The Augurscope (Schnädelbach et al., 2002) depicts a historical 3D scene of a medieval castle. A second display depicts the position of the augurscope on the simulated 3D scene, to depict the viewpoint. The visualization of historical data is also useful for construction monitoring, as it enables visualizing previous states of the construction site .

Present data
Augmented reality enables the depiction of data that is hidden to the user because of the point of view, or because it is not easily accessible such as underground structures (Schall et al., 2009, Schall et al., 2013 or occluded objects (Livingston et al., 2002). For example, the Vesp'R system (Schall et al., 2009) was designed to reveal underground structures. These structures can be overlaid on the video, or depicted through an excavation tool, that simulates a hole in the ground and improves users' depth perception. The SmartVidente application builds on this previous application and provides also tools for on-site surveying and inspecting. Thus, users can edit existing models from the AR application using a laser range finder calibrated to the camera view (Schall et al., 2013). A similar approach has been used for riverbanks maintenance (Pierdicca et al., 2016).
Augmented reality can also be used to provide other points of view to the user (Veas et al., 2012, Nurminen et al., 2011. The techniques MultiViewAR and Variable Perspective View adress this (Veas et al., 2012). MultiViewAR enables switching the camera so as to have a different point of view of the scene. The Variable Perspective View includes two views coming from different cameras through scale-preserving deformation.

Planned data
Augmented reality does not only supports enriching the environment with additional data, but also adding non existing objects or modifying the existing ones, to visualize potential changes (Devaux et al., 2018, Skov et al., 2013. Mobile devices, for example, enable architects to see 3D models of possible buildings where they would be located on the environment (Skov et al., 2013). Other applications support not only adding objects but also modifying them, for example through virtual duplication of existing objects, users can have a preview of how existing buildings would look elsewhere (Langlotz et al., 2012). Existing buildings can also be virtually removed and replaced with alternative objects (Devaux et al., 2018).
Visualizing simulated data can also be useful for risk prevention. For example, the Tsunamulator (Lonergan, Hedley, 2015) is based on augmented reality to visualize a tsunami realistic simulation that takes into account the geography of an existing coastal area.

Geographically distant
Geographically distant applications display the visualization using a physical referent that is distant from the geographical referent. In these cases, the physical referent is a scaled model of the geographical referent, such as a paper map. To classify the geographically distant applications that present physical data we consider the temporal indirection as well. We find again historical visualization that consider old data. The second category presents lower temporal indirection, and augments 2D maps with 3D information.

Historical Visualization
Historical data can be displayed using a scaled referent, such as a paper map (Asai et al., 2008). This approach can provide access to locations unreachable otherwise. For example, a paper map of the moon can be used as referent and be augmented with real images and additional information (Asai et al., 2008). The images are displayed through a virtual lens, that acts as a metaphor for the zoomed in view. Augmented paper maps can also be used for archaeological visualization in museums, as 3D models representing ancient buildings can be overlaid on them (Eggert et al., 2014).

Present data
Paper maps can also be used as referent to depict realistic representations that present a low temporal indirection. For example, markers enable overlaying 3D models on top of a 2D map (Liarokapis et al., 2005). The Foldable Augmented Maps (Martedi et al., 2012) also overlay 3D models, but this technique supports foldable maps as well. This map is intended to help users in navigation tasks by supporting comparison of the 3D models to the buildings in the city. In this case, even if there is a geographical indirection, it is intended to be used in the proximity of the geographical referent, as depicted in Figure 2.
Papers maps have also been used to depict flood levels (Reitmayr et al., 2005) as support for crisis management. The current river extension is projected on top of a paper map, in addition to the location of emergency units. The view can also be complemented with images from the site displayed in a secondary display called the image browser.
Furthermore, applications are not restricted to a paper referent, it is also possible to use real distant objects as physical referents, such as a wall to visualize images of a street (Devaux et al., 2018). These three examples vary widely in the scale and the fidelity of the physical referent. The scale of the wall is closer to the real scale than the paper map scale, and the geometry of the wall is also closer to the real object than the paper map to the represented city.

ABSTRACT DATA
We understand as abstract data, data that does not has a physical representation, such as statistics or thematic and qualitative data . We consider also here data coming from sensors that does not have a visible physical representation, and thus have to be encoded otherwise, such as Co2 levels and temperature data We describe existing applications that visualize abstract geographical data according to their geographical indirection.

Geographically Situated
As mentioned before, geographically situated visualizations present no geographical indirection nor spatial indirection. However, it is worth mentioning, that these measures are more difficult to calculate in this context, as the relationship between the geographical referent and the data is not as straightforward as it was for visualizations depicting physical data. In this case, the data and the geographical referent are not of the same nature.
To classify the applications, we consider again the data temporal indirection through historical, present and planned data.

Present data
Augmented reality can be used to enrich the environment with abstract data that is not physically visible. Even if the data is not always real time, we consider that it presents a low temporal indirection because it represents the current state of the environment. For example, the environment can be enriched with additional information coming from sensors (Nurminen et al., 2011, Gliet et al., 2008, Resch et al., 2015, such as radar data. Overlaying glyphs on top of the environment can provide a weather overview, for example using an artificial rendering of rain or clouds (Resch et al., 2015) or including a glyph to symbolize wind direction and speed (Gliet et al., 2008). Displaying labels and annotations overlayed on the corresponding objects (Zollmann et al., 2016, Lee et al., 2015, Ioannidi et al., 2017 can help users navigate a place. Abstract information about geographical objects can be displayed with augmented reality so that it is embedded in the ob-jects the data references. This data can come from sensors, and enrich the environment, such as dynamic point clouds (Santana et al., 2017), but also depict statistics related to a geographical object or position. One of the most straightforward approaches is to use color on the floor to represent an attribute of the geographical area such as soil data (Zheng, Campbell, 2019) or yield data (King et al., 2005). The use of 3D visualization has also been explored. For example, (Whitlock et al., 2020) introduce a system to visualize scorch percentage per area using the color of a 3D point glyph, positioned in the corresponding area. This approach has also been used to visualize Co2 levels (White, Feiner, 2009) through different 3D visualizations (spheres, bars, and fog rendering), and wind direction and speed (Ritterbusch et al., 2013). Existing geographical objects can also be used to display data, such as depicting statistics about a building on its walls. Some recent work explore displaying solar radiation data (Beatriz Carmo et al., 2014) and a summary about the construction progress of the object . This approach has also been used to visualize the health of bee hives and depict some behavior attributes of the hives, such as the hive drift using a O/D graph (Engelke et al., 2016) and in mineralogy exploration (Engelke et al., 2019) by depicting minerals types on top of rocks.
Most of these examples add information to the environment through abstract visualization or overlaying information on top of existing geometry. However, it is also possible to use the existing objects to encode data, by changing their geometry in order emphasize an object or to reflect one of its attributes (Takeuchi, Perlin, 2012). For example, a user can make a building artificially higher to emphasize it, or change its visual appearance so it reflects its function such as making it look like a french restaurant if it is a french restaurant.

Planned data
In the case of abstract data, planned data can represent the output of some planning process, for example a possible route. Using augmented reality, the route is displayed on the environment to help them explore a location, alone (Jang, Hudson-Smith, 2012, Huang et al., 2012 or collaboratively (Reitmayr, Schmalstieg, 2004). These applications have also been applied to other vehicles such as maritime navigation (Hugues et al., 2010) and micro aerial vehicle navigation (Zollmann et al., 2014). The way of displaying the routes to be followed vary across the applications, from cylinders representing different points on the route (Reitmayr, Schmalstieg, 2004), to lines (Huang et al., 2012) or direction encodings (Jang, Hudson-Smith, 2012).

Geographically Distant
A scaled referent has been used to display non realistic visualizations as well, by using papers maps (de Almeida Pereira et al., 2017, Morrison et al., 2009, Wiehr et al., 2017 or tangible models (Mitasova et al., 2006). In this case, the physical referent is the scaled model, and the geographical referent the place the scale model represents.

Present data
Paper maps can be augmented with 3D data such as an elevation model and population data (de Almeida Pereira et al., 2017) or 2D data such as points of interest (Morrison et al., 2009). In the latter case, the map is used as navigation aid in proximity to the geographical location, thus the geographical indirection is smaller than in the other examples, as depicted in Figure 3.

Planned data
The illuminated clay system (Mitasova et al., 2006) represents a land 3D model using clay. Additional data can be displayed by using a projector. When the user interacts with the clay model, the overlay is modified to reflect the changes in the land. For example, the system can be used to test different scenarios of sediment control and flooding prevention by overlaying the slope of the land and the water flow direction.

Combining situated and distant visualizations
Situated and non situated visualizations can be combined to improve collaboration between users in different locations. Usually a first user or group of users, the "overseer" is indoors and examines the location remotely and a second user, "the explorer" is outdoors exploring the area in situ (Nittala et al., 2015, Li et al., 2014. To improve coordination, the overseer has access to a scaled referent of the geographical area, where the position of the explorer is displayed. The explorer has access to additional in formation in situ, for example route data (Li et al., 2014), or domain-specific data such as petrol production well locations (Nittala et al., 2015).

CHALLENGES AND OPPORTUNITIES
The paper classification reveals that most of the applications are geographically situated, use only one visualization category, present either abstract visualizations or the depiction of realistic 3D geographical objects. Drawing from this, we propose some research challenges related to geographically distant visualizations and their physical referent and combining them with geographically situated visualizations. We then present some research challenges related to visualizing new aspects of the data such as 3D dense data and uncertainty data.

Referent Fidelity
Most of the reviewed applications are geographically situated, so the referent is the real geographic object. However, when this is not the case, the physical referents represent more or less accurately the geographical object. For example, a paper map will represent the relative position and distances of the geographical objects. A printed 3D map represents, additionally to that, the elevation of the area. A wall used as support to present the texture of an older wall, located elsewhere (Devaux et al., 2018), is closer in scale and geometry than using a scaled referent or no referent at all.
To our knowledge, it is unknown how these different levels of referent similarity and fidelity might affect augmented reality applications. We might guess that more representative referent will support better users on their tasks, but then it will probably depend on the task and what data attributes it considers. Empirical studies could assess the efficacity of different scaled referents for different tasks, and how to provide adaptable referents.

Scales
The considered geographically situated visualizations mostly present the objects at their real scale. However, in most of the cases, the geographical referents are big, such as buildings (Lee et al., 2012 and terrain areas (Zheng, Campbell, 2019, King et al., 2005, and thus the visualization is on an environmental-scale (Barba, Marroquin, 2017).
This means that the space is non-manipulable and requires locomotion to explore it. Therefore, the user has not access to an overview of the space, and this can be problematic for some tasks such as planning operations. Multiview Ar and Variable Perspective View (Veas et al., 2012) address this problem by enabling to combine different views at the same scale. This has also been addressed through collaborative means to support petroleum-well planning (Nurminen et al., 2011). Nevertheless, users on the terrain could also benefit from a representation at a different scale such as the Figural Scale, where the space is manipulable with no need of locomotion.
Multiple techniques exist in HCI and visualization to combine views at different scales such as using overview and focus + context (Cockburn et al., 2009) and some of them have been used in AR settings (Mulloni et al., 2010, Kalkofen et al., 2007. However, these applications combine two different scales in the same view. Questions regarding as to how to include and mix visualizations using different scales and physical referents remain unanswered for situated visualization. For example, how can we combine different referents so as to provide the user with overview and detailed visualizations? What kinds of scales are more complementary? What information should be displayed in each visualization?

Visualizing 3D data
Most of the examples depicting physical data analyzed here visualize 3D objects, using their corresponding 3D representations. Visualizations depicting abstract data rely sometimes on 3D visualization, but these are used to represent mostly one dimensional data tied to a geographical positions such as soil scorch percentage (Zheng, Campbell, 2019) or O/D graphs (Engelke et al., 2016). The visualization of 3D to visualize 3D physical data such as wind direction and speed, or other kind of flows seems under explored for now, with only one exception (Ritterbusch et al., 2013). Nevertheless visualizing these information in context could provide insights about the relationship between such flows and the rest of the environment. Besides, there are 3D flow visualization techniques that could be added to augmented reality environments, such as particle systems (Kruger et al., 2005) and texture-based techniques (Telea, van Wijk, 2003). However, visualizing such data could potentially bring a lot of occlusion, so it could be interesting to study novel techniques to compose the environment with these kind of 3D dense visualizations.

Uncertainty
Even with recent advances in GPS systems and computer vision algorithms, it is still difficult to accurately position virtual content on the environment (Santana et al., 2017). Furthermore, when it is displayed it has inherently a position uncertainty but it is never displayed in the reviewed applications, even if visualizing uncertainty is necessary for diverse tasks, such as data based decision making (MacEachren et al., 2005). Because of the importance of the geographical indirection and spatial indirection in the correct understanding of the visualizations studied, adding a representation of the uncertainty related to the location of the visualizations could give users a bigger understanding of the data. Thus, another possible research direction, is to study how to represent uncertainty related to the position of the visualization in its physical referent, and if there is a geographical indirection, how to represent the uncertainty of the physical referent as well.

CONCLUSION
We presented challenges and opportunities for AR situated geovisualizations. To achieve this, we categorized 45 papers describing these kind of applications according to the data they represent, physical or abstract, their geographical indirection and goal. This classification helped us draw opportunities related to mixing physical referents of different scales, studying the effects of the physical referent fidelity, adding an uncertainty representation and visualizing dense 3D data.
However, our approach presents some limitations. First, our analysis is based on a subset of papers. We tried to control biases by selecting papers coming from different disciplines, but it is possible that not all relevant papers are included. As future work, we plan to extend our review and also consider the interaction techniques used in these applications, and how the geographical indirection and spatial indirection can be leveraged to facilitate interacting with the visualizations.