INTEGRATING MULTIBAND PHOTOGRAMMETRY, SCANNING, AND GPR FOR BUILT HERITAGE SURVEYS: THE FAÇADES OF CASTELLO DEL VALENTINO

The conservation of built heritage is a complex process that necessitates co-operative efforts. Holistic, integrated documentation constitutes a crucial step towards conservation by contributing to diagnosis and by extension to the effective decision-making about the required preventive and restorative interventions. It involves the recording of interdisciplinary data to produce objective diagnostical conclusions concerning the state of preservation. Although the developments in close-range sensing techniques allow increasingly accurate and rich data recording for heritage building condition surveys, the problem of combining them (to allow integrated processing) often remains unsolved. This is particularly true when surveys include vastly heterogenous documentation data. This work aims to discuss methodologies and implications of such integrations through a monumental heritage survey case—the Castello del Valentino in Turin (Italy). Visible-spectrum and infrared imagery is combined with photogrammetric techniques, terrestrial LiDAR, and microwave measurements conducted on the historical façades’ surfaces, to examine the comprehensiveness of the data fusion results, as well as conclusions that can be drawn regarding previous interventions and the current condition of the monument.


INTRODUCTION
Built heritage conservation has significant educational, social, touristic, and political value, contributes to the environmental and financial sustainability of a location, and moreover to the sense of place through the architecture's visual aesthetic and cultural attributes (Lichfield et al., 1993;Worthing and Bond, 2008). Recording built heritage information is an integral part of conservation but also facilitates monitoring, management, and routine maintenance (Letellier, 2015).

Background
Recording data suitable to assist the decision-making process regarding conservation requires multidisciplinary approaches (Masciotta et al., 2016;Adamopoulos et al., 2017). The analysis of historic structures is an intricate task, due to the challenges and difficulties that the geometrical complexity, variability of historical materials, adopted building techniques, and extrinsic factors (including environmental characteristics) often induce (Binda, 2005). As a result, integrated documentation is required to assist investigations aiming to reveal information about the previous interventions, morphology of degradation, and overall preservation conditions (Kioussi et al., 2013). Thus, integrated documentation is a crucial part of the condition survey and an essential step towards identifying causes triggering damage and decay (diagnosis). It relies on topographic surveying, along with other non-destructive approaches that employ proximal sensing techniques to record rich data (Solla et al., 2020).
The integrated documentation depends on surveying techniques, not only for capturing the three-dimensional (3D) shape of the structure, but also to record the spatial reference based on which other metric and qualitative data can be integrated. Techniques proven to be efficient for architectural documentation involve close-range photogrammetry (CRP), terrestrial laser scanning (TLS), and combinations of both (Georgopoulos, Stathopoulou, 2017). However, both techniques are also sources of important radiometric data exploitable to further facilitate the diagnosis. Photogrammetric high-resolution ortho-photomosaics and color textured 3D models can be used to identify surface decay or to calculate damage indices (Russo et al., 2018;Randazzo et al., 2020). Reflectivity values recorded by TLS-which express the intensity of the backscattered laser energy-have been recently explored for mapping the alterations of historical surfaces (Del Pozo et al., 2015;Sánchez-Aparicio et al., 2018) as well as for surface moisture detection (Lerones et al., 2016).
Besides CRP and TLS, non-destructive methods that can deliver metric and radiometric data important for architectural heritage surveys include other optical methods, thermal and penetrating radar methods (Leucci, 2019). Near-infrared (NIR) imaging (0.75-1.4 μm wavelength) is an optical technique that has often been explored for the detection of historical building materials and the documentation of weathered architectural surfaces (Sánchez, Quirós, 2017;Rahrig et al., 2018). Thermal infrared (TIR) imaging (8-15 μm wavelength), frequently coupled with various image processing techniques, is not only useful for the study of alterations and material loss but also for understanding the distribution of moisture content and the identification of hidden defects and flaws (Maldague, 2001;Brooke, 2018). Ground-Penetrating Radar (GPR) techniques (microwave radiation) are increasingly used for the investigation of inner structure and damage of walls, buttresses, and columns of historical significance, as well as for locating moisture (Ranalli et al., 2004;Lampropoulos et al., 2017;Lachowicz et al., 2019). The complementarity of the abovementioned techniques in producing objective and comprehensive documentation results is evident, although the heterogeneity of recorded information and compromises made during data acquisition to reduce time and cost can reduce their capacity to be integrated.

Scope and Objectives
The fusion of multi-wavelength recorded data, which compose the integrated documentation, is decisive for maximizing their usefulness and avoiding misinterpretations that may lead to inaccurate conservation decisions. The task of integrating the heterogeneous data depends on aspects such as spatial and radiometric resolution, accuracy, and dimensionality of the fusion (Adamopoulos, Rinaudo, 2019). Integrative CRP and TLS approaches are well discussed and can ensure that density, accuracy, and color-resolution predefined specifications are met (Nex, Rinaudo, 2011;Munumer, Lerma, 2015;Chiabrando et al., 2019). However, the integration strategies with other types of recorded data are less frequently discussed. NIR images can be exploited towards constructing multiband image composites, useful for investigating architectural surfaces with nondestructive image processing-based methods (Lerma et al., 2012). The NIR's similarities with color imagery allow for the high-resolution texturing of historical buildings' 3D representations or the direct implementation of CRP-driven processing, thus facilitating integration with other data sources (Adamopoulos, Rinaudo, 2020). The problematics of integrating thermograms with metric data come from the inherent differences comparing with visible-spectrum (VIS) imagery and concern both the spatial (low-resolution) and radiometric (different observable features) characteristics. Methodologies on the fusion of thermal and geometric data depend on sensor registration (optical and thermal camera, or laser scanner and thermal camera), product registration (thermogram and point cloud, or thermogram and 3D mesh), or hybrid photogrammetric techniques (Lagüela et al., 2016). The choice of implementing one of these data fusion techniques largely depends on the scale of the survey and available equipment and can produce thermal-textured 3D point clouds or meshes. Data collected with non-destructive methods using microwave and ultrasound radiation methods can also be integrated when the position of utilized antennae is referenced appropriately; the integration refers either to 2D radargrams or to volumetric data produced by calculating surfaces with similar dielectric properties in the 3D space (Ercoli et al., 2016;Casula et al., 2020;De Giorgi et al., 2021). In the case of GPR, data quality is one of the most significant aspects towards multidisciplinary documentation and interpretation (Martinho, Dionísio, 2014).
Considering the above, the presented research investigates the fusion of multisource data recorded with close-range nondestructive sensing techniques (optical, thermal, and microwave radiation-based), aiming to produce rich integrated documentation results for built heritage. Through a case study of monumental architecture, the integration of multiband imaging, thermography, photogrammetry, TLS, and GPR is showcased. Workflows for overcoming multi-sensor data heterogeneity are discussed while paying attention to essential spatial and radiometric aspects.

Case Study
The Castello del Valentino is a suburban mansion and riverside royal residence already existing in the sixteenth century when it was bought by Emanuele Filiberto di Savoia. It belongs to the site "Residences of Royal House of Savoy", inscribed in the UNESCO World Heritage List in 1997. The castle is located on the left bank of river Po in Turin (Piedmont region, Italy). Throughout the last centuries, the castle has been subjected to numerous extensions, alterations, transformations, and restoration works. The overall condition of the front (west) façade (Figure 1) is visibly better conserved comparing with the riverside (east) façade (Figure 2), where several altered surfaces are visible predominantly in the lower parts of the façade, which can be partly attributed to the effects of moisture.

MATERIALS
As explained in 1.1, the integrated documentation considers the acquisition of multidisciplinary data which come from different sources. The recorded data are summarized in Figure 3. Topographic measurements were performed with a GeoMax Zoom PRO 3″ reflectorless total station. The accuracy was 0.9 mgon for the angle measurements and 2 mm + 2 ppm for the distance measurements. The TLS surveys were carried out with a FARO Focus3D X 330 phase-shift scanner. The laser scanner had a resolution of 0.3-0.4 mm at 10 m distance and a ± 2 mm ranging error. The transmitted laser beam's wavelength was 1550 nm.
The camera used both for VIS and NIR imaging was a Canon EOS Rebel SL1. This camera employed an APS-C CMOS sensor (22.3 × 14.9 mm) with a maximum resolution of 17.9 MP (5,184 × 3,456 pixels) and a pixel pitch of 4.3 μm. The crop-frame camera had undergone a conversion, including removing the internal NIR blocking filter, which was provided by 'Life Pixel Infrared Conversion'. A Canon EF-S 18-55 mm f/3.5-5.6 IS II lens was used to acquire all the optical images by attaching two external VIS and NIR-pass filters. The camera used for TIR imaging was the FLIR T1030sc with an uncooled long-wavelength infrared detector array (1,024 × 768 pixels, and pixel pitch 17 μm). The sensor's measurement accuracy is ± 1°C, thermal sensitivity < 20 Mk (NDET) at +30°C, and the spectral range 7.5-14 μm. The thermo-camera moreover had a build-in low-resolution VIS sensor.
The system utilized for the ground-penetrating radar surveys was composed of an Aladdin 2-GHz IDS antenna box (up to 400 kHz Pulse Repetition Frequency, 4768 scan/sec Max. Scan Rate) and an IDS K2 control unit.

Preliminary Surveys
The first phase of the surveys involved TLS range data acquisitions at the lower areas of the riverside façade and the main part of the east façade. The stations were placed to acquire homogeneously dense point clouds without occlusions, and thus two scans were performed on the east façade and six on the west, with large overlaps, which could facilitate accurate registration. The scans were used to generate general plans and lowresolution models of architectural surfaces after registering, denoising, and subsampling the 3D point clouds. Furthermore, the near-infrared scanning intensities were corrected (Suchocki et al., 2020) and exploited to create rapidly and cost-efficiently a representation of weathering stages (Figure 4), since the majority of the observed surface deterioration patterns belonged to a single typology-grey biogenic crusts. The second phase referred to the preliminary recording of many architectural elements by multiband imaging. VIS and NIR images were collected as parallel as possible to the surfaces of interest-avoiding any large deformation-and from consistent positions. For this scenario, each element was captured in a single image frame, retaining homogeneous spatial resolution for all the acquired imagery, at approximately 5 mm. Image quality was validated by calculating background noise and vignetting levels-systematic errors caused by camera electronics (Del Pozo et al., 2015). Uncompressed raw image data were exported with the RawDigger software, and visiblespectrum images were color-balanced.
Thermal-infrared images for preliminary recording were also collected as parallel as possible to the surfaces of interest, with approximately half the spatial resolution of the optical images, but capturing multiple images for every recorded element. Therefore, after applying the same temperature scales on the thermograms(ThermaCAM software), higher resolution TIR images were generated (same spatial resolution with optical imagery), by interpolating information from each quadruplet of images acquired at the same position.
All images were subsequently undistorted (utilizing parameters calculated by calibrating the involved sensors), then rectified by picking points that were either measured directly with the total station or indirectly by TLS, and finally merged with the HyperCube software to create multiband images. The multiband imaging data were analyzed by data reduction, decomposition, and compression methods to create 2D image visualizations that could be more easily interpretable. Figure 5 shows the results of one of the preliminarily surveyed areas on the main façade, where slight plaster detachments and a few surface cracks are visible through the elaboration of the multiband composites. Figure 6 shows another surveyed area on the riverside façade where extensive detachments, cracking, and biogenic crusts can be observed.  Wide-scale representations and 2D preliminary surveys led to identifying areas of interest for in-depth investigations by integrating higher resolution data and the elaborated GPR measurements. These two presented significant radiometric anomalies in the multiband analysis results (Figure 7).

Detailed Surveys
The areas identified to conduct further integrated documentation campaigns were a 0.6 × 3.8 m 2 planar surface on the lower riverside façade and the main façade's left column.
Imagery for photogrammetric processing was densely collected (> 80% bidirectional overlap) for both the planar and cylindrical geometries. For this scenario, a VIS and NIR spatial resolution of at most 1.5 mm was achieved to produce high-resolution mesh textures and ortho-photos. Thermal images for highresolution mesh texturing and ortho-mosaic production were also rigidly collected from a close range to match as possible the optical imagery's spatial resolution. A standard semiautomated methodology based on structure-from-motion (SfM) and multi-view stereo (MVS) algorithms was followed in Agisoft Metashape (Adamopoulos et al., 2021) with the EOS Rebel SL1 photos, using as reference measured coordinates of placed targets or points picked from the point clouds, achieving similar accuracy of approximately 5 mm for both spectra and surveyed areas. To tackle the problems of 3D modeling with the low-resolution thermal images, a hybrid photogrammetry-driven approach involving both optical and TIR images from the T1030sc was employed (Adamopoulos et al., 2020). The resulting accuracy was similar to that of the high-resolution image datasets, while maintaining a low noise level on the point clouds and models.
The GPR profiles were collected along horizontal (circular for the column survey) profiles with a vertical spacing of 5 cm. A wheel encoder was used to track the positioning of the antenna along the moving direction for each GPR profile acquired. Each profile's starting and ending point was also measured with the total station to obtain the same reference as the photogrammetric data. The GPR data were processed with the ReflexW software (Sandmeier, 2018), adopting standard processing steps. The obtained radargrams produced 3D amplitude point clouds that could then be visualized or used for computing isosurfaces.
The transdisciplinary results were integrated using the orthophoto-mosaics of visible, NIR, and TIR spectra to create multiband ortho-visualizations, which could be processed with the same approaches as in the preliminary surveys but also had spatial reference. This allowed the relative referencing between radar data (parallel and perpendicular to recorded surfaces), range information, and the other image-based data elaborations (Figures 8, 9). A significant correlation was observed for the riverside façades between material losses, high GPR reflection amplitudes, and surface thermal minima, which could be interpreted as high moisture content from water that had permeated the damaged structure's surface layers. Additionally, the cylindrical surface of the column could be developed-after calculating the average diameter from the TLS point cloud-to create a planar result, instead of only ortho-projections of every side of the column (Figure 10). This type of visualization made the interpretation of the results easier by facilitating the integration with the GPR data. Anomalies detected by on-site inspection, multiband imagery elaboration, and locating extreme amplitude values overlapped, led to the assumption that the identified volumes on the bottom part of the column corresponded to replacement material (mortar) originating from a previous intervention.   Meshes of the investigated surfaces were generated from the TLS-produced point clouds and textured by VIS, NIR, and TIR imagery in Agisoft Metashape. GPR data were recalculated in cylindrical form. Specific amplitude values corresponding to speculated anomalies were selected and turned into isosurfaces in CloudCompare. The last step of the data fusion was integrating the multi-wavelength products to visualize better areas of interest in three dimensions (Figure 11, Figure 12). Figure 11. Integration of thermal-textured mesh and GPR grid data for the bottom part of the column.

Figure 12.
Integration of color-textured mesh and GPR isoamplitude surfaces for the riverside façade.

DISCUSSION AND CONCLUSIONS
The combination of metric data with multi-wavelength results derived by laser scanning, imaging, thermography, and GPR surveys enhances the richness of recorded information for historical architecture and adds to data validity, integrated processing, and facilitates interdisciplinary interpretation. These non-destructive methods are complementary and can produce documentation content for vastly different typologies of building elements. However, as they operate at different wavelengths, they also collect data at different depths and possess distinct spatial characteristics. Thus, defining the parameters of acquisition in detail, such as spatial resolution, accuracy, and metric implications of their fusion, is crucial to reassure the quality of their integration, which can assist further diagnostic steps and conservation-related decisions.
Knowledge of a heritage building surfaces' defects and previous interventions is of paramount importance for diagnosis, and the first step to achieving it is interdisciplinary documentation. In this study, the documentation workflow started from collecting high-accuracy topographic measurements, point clouds, images, and thermograms. By methodically capturing, correcting, and rectifying imagery and artificially enhancing the spatial resolution of the close-range thermograms, multiband image composites were created to perform preliminary investigations of various buildings' elements. Their 2D elaboration helped the documentation of weathering, cracks, and healthy surfaces, the localization of previously restored parts, and the identification of areas of interest for more detailed surveys. Additionally, TLS data exploitation produced visualizations of the façades' geometry and weathering indices in a rapid way.
To conduct detailed documentation surveys, densely captured image datasets were processed through SfM/MVS-based photogrammetric approaches creating multiband high-resolution orthomosaics and textured models. These were enhanced by radargrams, grid data, and isosurfaces of amplitude values from GPR measurements. Although the accuracy and resolution of GPR may have been of double to triple values comparing to the other close-range sensing techniques, the significance of providing depth information in a non-destructive way became apparent. The fusion of all involved techniques gave a clearer picture of the nature of surface thermal anomalies, the effect of surface weathering, and the 3D volume of restoration materials. However, it should be mentioned that integrated processing of the heterogeneous data will require downscaling of the photogrammetric results to match the GPR properties and is a point of the authors' future research.
The major challenge faced in order to guarantee integrability of the multisource data was ensuring that during acquisition, their metric parameters and relative position were known. Since sufficient surface features cannot always be identified in thermograms, placing special reflective targets proved essential for thermographic recording. Regarding the GPR measurements, having already established a grid of the antenna's movements with known position parameters helped the registration with other data. A crucial advantage of cross-examining the multiband data was that certain misinterpretations were avoided. For example, it could be determined whether certain reflectance anomalies in the infrared spectrum were caused by dark crusts or moisture concentration.
To conclude, the authors would like to highlight that of the workflows presented in this work, for practical reasons, the approach that integrated multiband orthomosaics/developments and GPR slices parallel to the investigated surfaces proves the most promising. This approach not only allows for the straightforward integration between the spatially referenced heterogeneous measurements but can potentially provide an easier transition to semantic annotations and conservationoriented building information modeling (Chiabrando et al., 2017;Solla et al., 2020;Croce et al., 2021).