Enhancing the Reuse of Digital Resources for Integrated Systems to Represent, Understand and Dynamize Complex Interactions in Architectural Cultural Heritage Environments

In this work we develop a multiply interconnected system which involves objects, agents and interactions between them from the use of ICT applied to open repositories, users communities and web services. Our approach is applied to Architectural Cultural Heritage Environments (ACHE). It includes components relative to digital accessibility (to augmented ACHE repositories), contents management (ontologies for the semantic web), semiautomatic recognition (to ease the reuse of materials) and serious videogames (for interaction in urban environments). Their combination provides a support for local real/remote virtual tourism (including some tools for low-level RT display of rendering in portable devices), mobile-based smart interactions (with a special regard to monitored environments) and CH related games (as extended web services). Main contributions to AR models on usual GIS applied to architectural environments, concern to an interactive support performed directly on digital files which allows to access to CH contents which are referred to GIS of urban districts (involving facades, historical or preindustrial buildings) and/or CH repositories in a ludic and transversal way to acquire cognitive, medial and social abilities in collaborative environments.


INTRODUCTION
Progressive deterioration or even disappearance of CH resources, economic crisis (emigration and unemployment, as undesirable effects) and the lack of economic return (arising from institutions and private entities) have generated a serious problem for knowledge and mise-in-value of Cultural Heritage resources as cultural, social and economic actives.It is necessary to dynamize and articulate available resources which can provide an educational, social and economic return in terms of a social cohesion and more sustainable Cultural Tourism, specially in less-developed regions with very meaningful CH resources.To ease the reuse of multimedia contents available at Internet and users interaction, it is convenient to start from Web 2.0 framework and migrate towards Web 3.0 by extending usual approaches.We have developed this extension by using RDF schemata for ruins of churches in rural zones and small villages of Valladolid (Spain).This strategy enables end user creation of web content, and thereby encourages social networking which are crucial to improve communication of CH contents.His approach allows to recover a memory of unfortunately destroyed past and to understand the current reality.However, it does not generate a meaningful return to dynamize cultural zones which are depressed from the economic viewpoint and with a scarcely articulated social tissue due to the lack of resources.
Our proposal for cultural dynamization uses mobile devices for in-situ AR-based interaction as support.It is based on the development of multiple articulations between production, extraction and management of architectural heritage digital contents.The design and implementation of multiple articulations are focused to enhance and reinforce the economic, educational and cultural value of digital actives.Usual integration is holding at horizontal level, i.e in terms of access and reuse of a distributed network of digital repositories with CH contents belonging to different regions or countries; reuse is performed in terms of a shared lexicon.Instead, our approach is more focused towards the integration at vertical level i.e. towards an interaction between documentation, information and management systems (DIMaS).This scheme involves to digital objects, processing and analysis tools (for image-and range-based information) and services (including assessing rehabilitation tasks) to be applied to CH goods in the framework of the Semantic web.The fixation of an appropriate Ontology makes possible queries in depth (from the lexicon), discovery of relations between resources/tools (from thesauri) and the application/development of systems of rules (from propositional or descriptive logics).To achieve this goal in an effective way, we have developed an Ontology which is transversal to the three levels along vertical direction.This CH Ontology has been applied in architectural Cultural Heritage including documentation (photogrammetric surveying and monitoring referred to updatable 3D models), information (by following typical GIS strategies) and management modules (for assisting decision making in accessibility issues, e.g.) We have applied this methodology to give support to physical and digital accessibility issues, which is directly linked to real and/or virtual visits to architectural and archaeological sites.So, the application of ICT to digital repositories allows to perform the vertical incorporation of DIMaS for CH in a globally integrated system.Our approach is scalable and can be extended to different public repositories by means of Ontologies alignment; to achieve it it is necessary to specify the lowest level of semantics corresponding to lexicon.Scalable integrated models are crucial to articulate public and private initiatives in a common framework which can be reused from mobile devices (smart phones and tablets, in our case).In the Internet era, the development of cultural industries is strongly linked to the design and implementation of web services which must accessible by mobile devices.At web 2.0 level, web services are linked to interactive consults to large CH repositories, which can include the individual cre-ation of new contents in the most advanced stage.This solution is slow and, consequently, expensive for mobile devices.In the Web 3.0, the availability of shared open source software tools provides a support to interact everywhere (cloud computing) in a more effective way with multimedia digital contents, by enhancing redistribution and creation of more advanced CH contents.
Next step concerns to the incorporation of Augmented Reality (AR) to the current surveyed state of CH architectural objects.The development of efficient solutions for mobile devices is a non-trivial problem as it is shown in (Arth and Schmalstieg, 2011).A typical coarse-to-fine strategy starts with a 2D framework (cartographic representation) where all places of interest are georeferenced; in our case, we have identified more than one hundred CH sites in the city of Valladolid (Spain).Very nice global reconstructions of well-documented sites have been performed for large cities of the Roman empire in Europe, having into account historical research and archaeological excavations (Guidi et al., 2005) and including very illustrative advanced immersive projections (Giannakidis, 2011).
Our presentation tries of answering to several basic issues concerning to communication of CH contents by using Augmented Reality resources.Next section proposes an answer to what communicate in terms of an Augmented Reality (AR) model which can support additional layers linked to Advanced Visualization Tools; in particular, knowledge management can be performed by semantic layer which to solve interoperability issues.Next issue concerns to where and how to communicate; for an efficient Real Time (RT) interaction in outdoor environments we need to simplify environment representations and to display visualization of reduced meaningful contents.How and with whom interact poses the problem of fusion of information arising from embedded sensors in mobile devices (smart phones and tablets) for SLAM and image processing (transformation between polygonals) supporting the deployment of AR solutions.Last issue concerns to a justification of creation of simplified contents which is presented in section five.Our approach poses more advanced challenges for interactive visualization of disappeared urban environments with CH interest.Our solutions are described in terms of the ongoing work involving AR and Advanced Visualization Tools.

WHAT TO COMMUNICATE?
Most initiatives are organized following a quantitative approach, with a focus towards linking repositories in a distributed CH network following a "flexible, scalable, secure and reliable infrastructure; a typical recent example is provided by Europeana network.Following this approach, from a collection of "linked data enabled" cultural heritage information repositories will emanate a better knowledge and mise-in-value of Cultural goods and resources.Currently, there is a very large amount of information in CH repositories, which can be reused if interoperability issues are adequately solved.Interoperability is a semantic issue which requires complex solutions in the Web Semantic framework.To fix ideas, we restrict ourselves to the lowest level involving a lexicon, i.e a previously specified list of key words.Very often, annotation is very incomplete and involves only to a very limited number of elements which are considered as meaningful by the expert in charge of annotation.It would be necessary to develop a more systematic approach which could include the viewpoint of other experts in the framework of collaborative environments, and to try of developing expert systems in charge of reinterpreting available multimedia contents.
From linguistic and computational viewpoint, real-time semiautomatic recognition of architectural primitives is a far-reaching Figure 1: Real texture of an original cube of Alcazarejo goal which involves initially to the recovery of semiotic contents (in terms of a collection of key words) from images.This task requires iconological studies in regard to CH contents and software tools for knowledge representation and management in symbolic terms.This ambitious goal can be achieved in some cases; for example, some visible structural architectural elements (walls, columns or vaults, e.g.) can be easily recognized in terms of volumetric characteristics.However, the very large diversity of decorative elements makes still very difficult to provide a general solution for this problem.Hence, for remote access to CH repositories we restrict ourselves to annotated contents according to an explicit lexicon.Currently, there are different lexicons and thesauri which must be automatically aligned in the framework of the Semantic Web to ease the interoperability between different repositories.Unfortunately, there is no still an agreement about commonly accepted methodologies which can be applied to CH domains.The limitations of current methodologies and alignment techniques applied to Geospatial ontologies are discussed in (Delgado et al., 2013).The SKOS (Simple Knowledge Organization System) has developed a Primer (Miles et al., 2005) with specific recommendations and software tools in the W3C Working Group which can be applied to CH repositories.However, most repositories have not still organized following semantic criteria, and the lexicon is not adequately specified.
To improve the current state, it is necessary to combine educational and social contents supported by a visualization of CH contents.This combination requires standardization in contents as a common support for advanced ICT, and the development of new narrative styles for an appropriate contents transmission.We are adopting a scheme inspired by (Jacobson et al., 2009) in a very bounded architectural environment.
The application of typical architectural surveying techniques (image and range based) by means of calibrated cameras and laser devices are very useful for a professional presentation of the current state of archaeological sites.In particular, their combination has allowed to construct a three-dimensional model of some zones which are not accessible for most citizens.The figure 1 displays the remapping of original textures on a volumetric mesh corresponding to some underground vestiges.
To go beyond funny aspects concerning to the navigation of a 3D model, it is necessary to incorporate multimedia materials contained in CH repositories according to a well-specified semantics.Along nineties emerge complementary strategies which are based in different methodologies (Farbey et al., 2009) and/or different functionalities (Smithson and Hirschheim, n.d.) linked to the evaluation of Information Systems.Both provide quantitative measures and qualitative indicators to evaluate the reaching and performance.In practice, it is not possible to identify which are the "best" methodologies, because criteria are not comparable between them.The approach performed in (Smithson and Hirschheim, n.d.) has been developed in terms of three basic categories which are easily adaptable to the articulation between Documentation, Information and Management Systems (DIMaS).Indeed, This contribution has been peer-reviewed.The double-blind peer-review was conducted on the basis of the full paper.98 • Efficiency oriented methods where the performance or quality of a System is evaluated w.r.t.some predefined technical and functional specifications.This evaluation provides a link between Documentation and Information Systems relative to repositories: Its evaluation is obtained from dialogues with users, persistence of interactive navigation and suggestions performed through forums, blogs, and related social networks (including users and developers communities).
• Understanding-oriented approaches which is focused towards a deeper understanding of value generation by systems and their association with the organizational context.This category concerns mainly to the knowledge linked to Information Systems linked to repositories and can be measured by the interactive contributions of users to digital contents (including generation of new digital contents) • Effectiveness oriented methods which is oriented to try of evaluating the induced dynamics in terms of the economic activity generated in Management Systems, the development of intermediate structures (Living Labs to give academic support to contents producers or mash-ups as communities of advanced final users).
However, the above images provide only a static representation which is not understandable for most people.It is convenient to generate a 3D model, superimpose it to the current representation and link it with additional digital information contained in accessible repositories.Advanced Visualization Tools support these functionalities and furthermore they are transversal to the DIMaS hierarchy.Indeed, they allow not only an interactive navigation around surveyed objects (documentation), but an access to different kinds of information sources linked to the digital objects.In precedent papers, management issues were focused towards conservation and intervention strategies involving CH resources management by the Regional Administration.In this work, a videogame-based interaction replaces the above approach for management, and opens the door for enjoying cultural goods through technological resources.

An AR-based approach for historical contents
Our proposal is to provide a support to the "Back to the Past" to have a better understanding of the current reality throughout successive transformations of the urban space as a representation of social and historic phenomena.In particular, our approach is based on a previously specified repository concerning to the Spanish city of Valladolid which is semantically annotated.We have identified more than one hundred places with cultural, historic and/or artistic interest in the old villa, which are adequately georeferenced.Near to each selected place, it will be possible to In this way, a walk in the current city will provide different representations which can be composed to obtain a global understanding of the evolving historic environment.Our approach is understood as a simplified videogame, where it is necessary to discover meaningful elements which are interconnected with other places; so, we intend to contribute to generate a local representation of medieval buildings which is superimposed to the current urban trace.
To improve the understanding of past, we have developed a multiplatform solution which is based in Augmented Reality on mobile devices (Smart Phones and PC tablets).We illustrate the performed approach with a historic environment relative to a very limited quarter of Valladolid which concerns to the Alcazarejo (a contraction of "Alcazar Viejo") which is the primitive fortress in a vertex of the medieval city.There are archaeological vestiges which are not accessible for most public between 4 and 6 meters under the current level.Some simulations have been performed at a near square (Plaza Poniente) which are useful to provide another views of the environment.A schematic representation of our approach concerns to several key frames which are displayed in a consecutive way: 1. Right localization (position and orientation) of the citizen in regard to the environment which is performed by means of a mobile device able of processing georeferenced data (managed in terms of UTM coordinates in our case) 2. Understanding the current environment which is performed in terms of zoom-in from aerial views and superposition of the the contour of medieval villa on the current zone (see Figure 2 for a historic recreation).

3.
Replacement by an old pseudo-volumetric representation based on a selected zone of the annotated representation provided by the plan of Bentura Seco (1738).Identification of performed transformations.
4. Back to the medieval past: Successive representations from the 15th century till to the 12th century.where the Alcazarejo was already built.

Delimitation as a part of large medieval palace:
The Alcazarejo was extended to an Alcazar (a typical Royal Residence of Arabian influence) along 13th and 14th centuries, since its cession to the Benedictine Order. Figure 3 shows an interpretation of the royal complex Alcazar-Alcazarejo.
6. Focus on the building: Hypothetical visualization by analogy with similar fortresses which were built between 11th and 12th centuries.
From the step 3 all elements are hypothetical reconstructions.Up to some archaeological vestiges in the underground, there are no visible vestiges of the above described buildings.The building was constructed not only as a defense against Spanish arabs living in the South, but also as a defense of royal power against the population inside the walls.From 1390 the palace complex composed by the Alcazar Real and Alcazarejo were transformed along four centuries in one of the largest monasteries of the Benedictine order in West Europe.Thus, these buildings can only visualized by using AR techniques for their representation.
The already performed basic version displays the above contents in a consecutive way, with a small interval between consecutive frames with simple zoom-in effects to focus the attention towards the most meaningful historical facts.If the mobile device has a tactile sensor, each one of the above steps can be navigated in an interactive way.In next versions a low-level dialogue will be developed for each frame including progressively complex contents involving historical facts involving construction techniques, different uses of castle-residence along almost three centuries, local government by the ten lineages, rebellions of population, and/or daily life in a medieval villa around the royal residence.

WHERE AND HOW TO COMMUNICATE?
In a broad sense both issues (where and how communicate) concern to some of the oldest problems in advertising.The most immediate answers are Internet and interactive playing as support.
But at a more deep level, both issues concern to language to be used and the transparency of technologies to be used.Nevertheless its simplicity and generalized banality of contents, simplified representations (including popular cartoons) display some of the most efficient ways of communicating.Hence, no support or no language can be previously excluded to improve the communication of CH contents.
Recent developments of smart phones or tablets as support, cloud computing for web services everywhere and Augmented Reality for real-time interaction.Their fusion provides a general technological framework to answer to the initial two issues (where and how).Following (Arth and Schmalstieg, 2011) there are two hard problems for mobile AR: registration and precision.Small errors in localization can generate unacceptable results in the application.Thus, it is necessary a robust estimation of localization parameters and accurate visualization of navigable models in the mobile smart phone screen, by minimizing the computation and memory usage.In (Guardia et al., 2012) one can see a general introduction which is related to our approach.
Some of the bottlenecks to be solved are the following ones: (a) Environmental conditions (including illumination), (b) energy consumption (AR), (c) availability of information in the network (simultaneous access, complexity of 2D/3D information), and (d) performance of interaction and visualization (in terms of different interaction types).Our current contributions are focused towards the two following issues: • Availability of information: Our application supports textured models with more than five thousand control points.Low zooming, Refreshing, remeshing and/or remapping of textures under different illumination conditions can be currently performed.High resolution models are not accessible, still.
• Performance of interaction and visualization: Contrarily to smart phones, tablets require usually both hands for interact.
We have developed a software application which allows to interact with gaze control, by means the identification of 16 zones of the screen where attention is fixed.
Figure 4 shows screenshots of a proof of concept of the AR application running in a mobile device and performed in the environment of M. Delibes Campus of the University of Valladolid.
The above issues concern to new interaction ways which are related to the visualization of CH contents.To go beyond the current state such advanced visualization, it is convenient to combine the approaches performed by experts in Urbanism and/or Graphical Design.In order to facilitate the knowledge representation we use concept map, i.e. a diagrammatic representation which allows visualize representations between concepts in terms of relations between images.The management of the concept map is an extension of a cinematographical script, which incorporates different possibilities to advance in the story as it appears in videogames.We have adopted this reasoning way along successive approaches to the visualization of two specific zones in terms of pathways to be performed in place around two historical zones of Valladolid with their most typical monuments (La Antigua, San Benito).In both cases, there is a large amount of archaeological vestiges which provide the support for advanced visualization approaches for CH.
The interaction is focused towards different ways of interconnecting providers and consumers of CH resources, which are classically designed.A typical server-client approach provides the support for this classical (and passive) approach.From the late nineties, serious games for mobile applications are an emerging technological field going from the lowest level web-based solutions for individual players to more advanced mash-up applications for communities of users in more open collaborative environments.A recent survey can be seen in (Anderson et al., December 2010).Luckily, we have at our disposal a pseudovolumetric representation at urban scale which has been performed at 1738 by Bentura Seco (1738).
This representation provides a deformed model of the whole urban trace at the first half of the 18th century: It is remarkable that Valladolid has back down along 17th and 18th centuries because the capital of Spain has been definitely moved to Madrid (1606).
Nevertheless the Bentura's false perspectives, this representation provides a faithful visualization of the development at the 16th century (and not only for 1738) which corresponds to the largest expansions of the medieval villa.The existence of this documentation justifies our choice for simple videogames; our approach is focused towards an individual player who is immersed in 2.5D representation.The video-game approach based in third person camera is more expensive and demands more computational resources; furthermore, its incorporation to mobile devices displays some hard technological problems which are not still solved.
If we look at the first issue (how interact), a first answer arising from image-based modeling concerns to the adaptation of SLAM (Simultaneous Localization and Mapping) strategies to mobile devices.Nowadays, it is not possible to deploy on smartphones complex solutions which require memory intensive algorithms; most solutions have been originally designed for desktop computers.However, the increasing performance of miniaturized components allows to combine hardware devices (compass for orientation) and GPS (Global Positioning System) as a robust input which bounds the search range for queries involving meaningful buildings which has been applied to the already implemented solution.
Relative to CH contents, we have performed the documentation of a large zone of the historical center of Valladolid (Spain) including around two hundred sites of interest which are adequately georeferenced.
For some of these places (around La Antigua church and Monasterio de San Benito), we have taken a collection of images which play the role of reference elements.These elements are compared with the images captured by a user to generate a dialogue.
We have discarded other solutions (based in Google Street View, e.g.) to reinforce pedagogical aspects which very often are absent of commercial applications.In other words, the visitor must explore the CH environment according to the approximate location of archaeological vestiges.
The methodology includes several steps which involve to imagebased interaction.It is performed in terms of 1.Localization (position and orientation) of user in terms of GPS and compass of mobile device.
2. Comparison (in terms of homographies) between polygonals involving to the sky-line of buildings.
3. Automatic selection of zones of facades which are relevant from the CH viewpoint according to the information available at the CH database.
4. Optional deployment of virtual worlds for interactive experiences, as well as mixed reality games.
5. Combination of real and virtual interactions in CH environments.
Technologies involve to an appropriate combination of Computer Vision, Computer Graphics, VR/AR and videogames.Their integrations and application to CH environments displays some complexity.The general goal is the provision of real-time 3D visualization and simulation from user interaction.To fix ideas, we illustrate our approach with some examples arising from the Monasterio of San Benito (Valladolid) which was rebuilt since 1400 till 1750 on the demolished Alcazarejo and the old Real Alcazar, the palace-fortresses of kings of Castilla in Valladolid till the end of 14th century.
The Alcazarejo was a small castle of quadrangular plant consisting of a big tower with cylindrical elements at corners and at the middle of each wall; it contains a small indoor patio which distributes spaces and uses, following roman and Muslim tradition.
It was constructed at the first half of the 13th century, more than a royal residence rather by their defensive characteristics.Nowadays, there are some archaeological vestiges in the underground, including basis of cylindrical towers and some parts of walls.Unfortunately, most of these vestiges are at 5 or 6 meters under the ground level and they are not accessible for most people.In fact most of population ignores these archaeological vestiges, this part of the medieval history and the role of the city in the Castilla kingdom till 16th century.To construct a virtual model of the Alcazarejo, we have adapted materials arising from another similar castles which were constructed in the first half of 13th century, by following the tradition of Muslim kingdoms of taifas.

WHY CREATE NEW CULTURAL HERITAGE CONTENTS?
Currently, there is a very large amount of disappeared CH contents or which is hidden under current buildings.Their visualization heps to understand the past, and consequently the present.Furthermore, related physical and human resources which could explain are underemployed.ICT and multimedia production have a very large potential to help to understand the reality at different depth levels by following discovery strategies.Re-creation of CH contents is the first part, but more advanced developments can be incorporated, also.
A drastic reduction of volumetric information arising from laser scan or rectified photogrammetry provides a collection of cuboids which simulate the current state of the city.This approach has been applied to the French city of Toul (Chevrier et al., 2010).We have applied a similar strategy to an urban district of the World Heritage Spanish city of Segovia (Spain), but the economic and computational cost are very high.To ease the interaction we need to develop cheaper solutions.In other words, to be more effective some technical aspects must be simplified; In addition, it is necessary to organize CH contents following different depth levels which should be displayed in several steps to improve its dissemination according to users interests.This goal requires a higher effort in educational tasks for all ages and the This contribution has been peer-reviewed.The double-blind peer-review was conducted on the basis of the full paper.101 Seemingly, the proposed simplification is opposite to Photogrammetric Surveying, but it must be understood from a complementary viewpoint.Dissemination for large public is understood as a path for asking about more detailed information concerning to humanistic (historical, artistic, societal) aspects and professional solutions (photogrammetric surveying, historical urban GIS, 3D modeling, augmented reality).This strategy proposes a common framework to integrate all of them in a seemingly funny way for user, but with a view to understand and valorize the environment from the CH viewpoint.

CONCLUSIONS AND FUTURE WORK
Digitization of physical resources contained in large public repositories (museums, archives, Foundations) is the first step for valorisation of CH goods.Their reuse can be performed through distributed networks which are organized by means of specific Ontologies.The lowest level supporting this organization concerns to metadata.Multiple articulations can be developed from specific CH metadata towards EU multimedia data to provide an Internet support to create new digital contents and constructing an European citizenship aware of CH richness as a reference for diversity, tolerance and common values sharing.However, neither of these technological solutions contributes to make more sustainable the CH tourism if there are no visitors.In this work, we provide answers to several basic issues concerning to the communication of CH contents by following an interactive strategy based in games with CH contents in historic urban environments.To achieve this goal, we have designed and implemented a software application which includes Web services, Augmented Reality and Serious Games to enhance cultural experiences through immersive interactions performed at mobile devices.This application is illustrated with some examples corresponding to a disappeared zone of Valladolid (Spain) relative to one of the four royal residences in the Medieval Age (currently there is once, which has been converted in the enclosed Monastery of Las Huelgas).So, we intend to contribute to improve and valorize the knowledge of old times and their projection towards the future.
Next steps concern to the extension of this initial work to more complex buildings (Real Alcazar) or CH environments (around the Antigua church including already disappeared roman and medieval baths, and some parts of the wall of 11th century).In the future, we intend to extend this approach to the whole historic center of Valladolid (the old capital of Spain) including the extrusion of the villa from approximate representations of the 18th century, and relevant historical documentation, which is currently supported in a sparse way from planar information.This more ambitious goal will require additional resources involving scalable volumetric models, the incorporation of low-level Photorealistic Rendering to mobile devices for some limited zones, and more efficient compression techniques, between others.

Figure 2 :
Figure 2: Wall doors of Valladolid in the 11th century

Figure 3 :
Figure 3: The Alcazar surroundings in the 12th century activate the application and navigate around old representations in an interactive way.

Figure 4
Figure 4: a) A virtual 3D reconstruction of the castle of Villarejo over a real image taken from camera; b) The same model on a mobile device using augmented reality 3.1 Some technological contributions

Figure 5
Figure 5: a) The volumetric representation of the Real Street of Segovia using cuboids; b) A textured building model of the street creation of new digital contents in connection with multimedia industry.Seemingly, the proposed simplification is opposite to Photogrammetric Surveying, but it must be understood from a complementary viewpoint.Dissemination for large public is understood as a path for asking about more detailed information concerning to humanistic (historical, artistic, societal) aspects and professional solutions (photogrammetric surveying, historical urban GIS, 3D modeling, augmented reality).This strategy proposes a common framework to integrate all of them in a seemingly funny way for user, but with a view to understand and valorize the environment from the CH viewpoint.