AUTOMATIZING DEGRADATION MAPPING OF ANCIENT STELAE BY DUAL-BAND IMAGING AND MACHINE LEARNING-BASED CLASSIFICATION

Degradation patterns are the visible consequence of the impacts of environmental factors and biological agents on stone heritage. Accurately documenting them is a key requisite when studying exposed stone antiquities to interpret weathering causes, identify conservation needs, and plan cleaning interventions. However, a significant gap can be identified in practical automatized procedures for mapping patterns on stone antiquities, such as ancient stelae. This work evaluates a workflow that uses visible and near-infrared imaging, combined with machine learning-based digital image segmentation tools, to classify degradation patterns on marble stelae correctly and cost-effectively. For this work, different classification methods are considered. Results are analyzed using error matrixes and reference degradation maps. The application cases include stelae displayed in the courtyard of the Archaeological Museum of Eretria (Euboea, Greece). The proposed methodology aims at being easily adapted to facilitate the conservators’ work.


INTRODUCTION
Stelae-slabs or blocks of stone, usually standing upright on the ground-were often used for dedicatory purposes in ancient Greece. They serve as essential sources of evidence for ancient history through their inscriptions (Bodel, 2012) and materiality. Typically made of marble, they were carved in relief and often decorated. In the same way as other stone antiquities, ancient stelae are subject to degradation due to weathering caused by environmental factors and biological agents (Savković et al., 2016). Assessing and understanding the complex process of stone weathering is crucial for the conservation of stelae. Defining and mapping their surface patterns is the first critical step towards diagnosing degradation-inducing factors and agents (Delgado Rodrigues, 2015). The method of mapping historical stone surfaces is a well-established non-invasive and non-destructive technique that allows the registration of information about degradation forms in a detailed and reproducible way (Heinrichs, 2008;Delegou et al., 2013). Mapping the formation, intensity, and spatial distribution of stone patterns can accelerate drawing conclusions and reduce required diagnostic surveys and sampling. This procedure is often conducted by hand, manually utilizing computer-assisted design (CAD) and spatial information systems (SIS)-based platforms and photographs as reference, or with specialized software (Puy-Alquiza et al., 2019;Siedler, Vetter, 2015). Digital image processing (DIP) can serve as an effective mapping alternative (Cossu, Chiappini, 2004;Moropoulou et al., 2013;Adamopoulos, Rinaudo, 2021), and in order to produce semantically accurate classification results, it requires the conservators' input.

Scope and Aims
Recently, advances in DIP have allowed the automatization of segmentation and classification for cultural heritage applications through machine learning (ML)-based methods (Grilli et al., 2018). However, implementations aimed towards degradation mapping are often case-specific, designed for heritage of large dimensions, or require substantially large image datasets to successfully train the involved algorithms for optimal results (Kwon, Yu, 2019;Pathak et al., 2021)especially when deep learning algorithmic processes are involved. These characteristics make mapping approaches not easily adaptable and impractical for the detailed degradation documentation of small-sized stone antiquities. Therefore, more universally useful and agile solutions should be considered that have the ability to contribute to conservation planning.
The presented research takes into account that methods for mapping the degradation of marble stelae (and other stone antiquities) should be practical, reproducible, and cost-efficient. A workflow for two-dimensional mapping based on low-cost visible and near infrared-spectrum imaging and ML tools is designed and then evaluated for annotating identified surface patterns. In section 2 of this paper, the methodology is described, including image capturing, preparation, and mapping approach. In section 3 follows a summary of the application, which addresses instrumentation, ML algorithmic implementations utilized for mapping and the evaluation approach. The presentation and evaluation of results follow in section 4; in section 5, some concluding remarks are attempted.

METHODOLOGICAL APPROACH
Taking into consideration that visible and near infrared (NIR)spectrum images as well (De Roy et al., 2016;Adamopoulos, Rinaudo, 2020;, are useful for the digital mapping of sculptured stone surfaces' preservation state, the proposed procedure employs dual-band imaging. The methodology starts from the acquisition of the required images, continues with the pre-processing of the captured images to acquire spatially and radiometrically corrected data input for the classification, and ends with the thematic mapping of the degradation patterns through ML-based classification of the false-color imageswhich combine RGB and NIR information. The summary of the methodological workflow is presented in Figure 1.

Image Acquisition
The proposed method employs affordable sensing techniques to acquire the true color (RGB) and NIR-reflectance images. NIR images can be captured using consumer-grade DSLR cameras after modified by replacing the internal NIR-blocking filter with a visible opaque filter, significantly increasing the camera's NIR sensitivity (Verhoeven, 2008). Inserting a NIR-pass filter in front of the lens will allow pure NIR imaging and can be chosen to capture particular wavelengths depending on specific needs. Low ISO values are being selected during acquisition to reduce on-image noise. The images are acquired as parallel as possible to the object, with the same camera positioning (to avoid varying occlusions) and homogeneous exposure. They are captured to fit the object in a single frame, while simultaneously achieving resolutions sufficient for the degradation mapping. Alternatively, images can be acquired with a professional camera and then down-sampled during the data preparation phase based on scale specifications.

Data Preparation
Low-cost cameras allow versatility during data acquisition (Adamopoulos et al., 2019); however, harnessing the correct radiometric information in order to perform the degradation mapping or other classification accurately requires validation of the quality of collected images (Del Pozo et al., 2016)especially when modified cameras are employed (Webb et al., 2018). Therefore, in order to ensure their quality, images are exported in uncompressed format, then reverse-debayered and color corrected (only the RGB images) with RawDigger software, corrected from vignetting and inhomogeneous illumination with ImageJ (Abrà moff et al., 2004), undistorted and rectified with HyperCube. Finally, synthetic pseudocolor images are composed of the green and red channels of the RGB image and the NIR-reflectance image (NIRRG image) after the latter has been manually matched to the former using common points.

Mapping
The classification of the degradation from the images is performed using machine learning models embedded in WeKa (Witten et al., 2011). Specifically, the 'Trainable Weka Segmentation' (Arganda-Carreras et al., 2017) plugin of ImageJ's Fiji distribution is used (Schindelin et al., 2012). These models are trained by manually annotating small patches/regions of interest that correspond to each type of degradation identified. The categories of degradation vary depending on the case study. They are selected after visual inspection, using as a guide the manual or guidelines adopted by the respective archaeological/restoration agency. For this work, the ICOMOS-ISCS (2008) Illustrated glossary on stone deterioration patterns is considered.

Materials
The instrumentation used for the dual-band image acquisition included a modified Canon Rebel-SL1 digital camera (with a resolution of 5184 × 3456 pixels) with an external NIR-pass filter and a Canon EF-S 18-55 mm f/4-5.6 IS II zoom lens for NIR-reflectance imaging, and a smartphone employing a Sony IMX600 sensor (with a resolution of 3648 × 2736 pixels, at RAW capturing mode) with LEICA optics for color imaging. The antiquities involved in this work were eight inscribed and uninscribed stelae from the Archaeological Museum of Eretria in Euboea, Greece (Figure 2). Images of the stelae were captured, exported and prepared for digital processing as described in Section 2.2, considering a 1:2 scale.

Classification
The main identified categories of degradation on the stelae were moss-lichens, discoloration, crusts and black crusts, and were therefore selected as classes for the classification, along with an additional class for relatively healthy marble surfaces. Given the manual nature of selecting the classification classes and annotating the sample regions for training the classifiers, four different classifiers were trained to allow comparisons. All classifiers used for the degradation mapping were based on a decision tree learning method. The evaluated in ImageJ classifiers were the Random Tree, Random Forrest, Fast Random Forrest (Breiman, 2001), and j48 (Quinlan, 1993). The ML techniques were tested for four stelae, and then the trained classifiers were applied to the other four, which presented similar degradation patterns.

Evaluation
To evaluate the classification results, reference maps were constructed. RGB images were also involved in the evaluation to estimate the contribution of introducing near infraredspectrum data for improving the classification. Error matrixes were calculated as an accuracy metric to compare performances. The error matrix is a square array of values, which expresses the number of pixels assigned to a particular class in one classification relative to the number of pixels assigned to a particular class in another, reference, classification (Congalton, 2020). The columns represent the reference map, while the rows indicate the classification generated from the ML techniques. The ration of correctly classified to the total number of pixels is another important metric, derivative of the error matrix.  All approaches generated results with over 70% accuracy. Among them, consistently high accuracy results were achieved with the Random Tree implementation for the more decayed stelae and with the Fast Random Forest for the less decayed stelae. Figure 3 and Figure 4 present the comparison between reference maps and generated results. The ability of the random forest method to tackle problems in decision tree learning such as learning unusual irregular patterns and overfitting training data, by assigning random subsets of the training and from them random input feature subsets, facilitated better accuracies when the distinction between pathologies and cleaned/cleaner surface was clearer. However, when there was overlap between degradation forms, and when small enclaves of moss-lichens pixels were present inside discolored areas, the decision tree method, which uses all features of interest for the training, proved more efficient.

RESULTS AND DISCUSSION
It should be highlighted that the complexity of the stelae's surface degradation can contribute to the misclassification of pixels. This becomes apparent from the higher accuracy values observed for Stelae M.E. 980 and M.E. 1131. The reference data consider the predominant form of degradation present at each part of the marbles' surface, but, in fact, the different degradation forms can be overlapping, which means that these data are not perfect. Therefore, classifiers assume the reference data correct, but there is an uncertainty present, which rises as the number of degradation categories increases. This reflects on the misclassified pixels, which may not always be assigned to an incorrect class, but on a lesser dominant degradation category at a particular region of interest.
Additionally, inhomogeneous reflectance caused by features induced due to material loss-on the micro and macro scale (i.e., erosion, pitting, missing parts)-can affect the performance of the classification and can usually not be mapped as they are overlapping and often caused by weathering phenomena such as biological colonization (Urzi, 2004).
It is evident from error matrixes Table 2 and Table 3, that when multiple degradation forms were present, the most common misclassification was between moss-lichens and discoloration, but also clean surface and discoloration when discoloration was not very intense (considering the dual-band Random Tree approach as the most accurate for stelae 18084 and 1357).
Including the near-infrared data generally improved accuracy slightly, and particularly for the more decayed stelae, which can be explained by the vastly different reflectance characteristics of the biodegraded areas at the NIR spectrum.  The trained and tested classifiers were applied to the image composites of the other four stelae to complete the degradation mapping ( Figure 5).

CONCLUSIONS
The proposed approach allows the detailed 2D mapping of degradation caused by weathering on stelae and other stone antiquities. Forms of degradation are identified by visual inspection and then automatically annotated through a supervised machine learning-based approach, applied on images combining spatially and radiometrically corrected data collected at the visible and near-infrared spectrum. The methodology employs affordable equipment and few processing steps that do not require particular training or specialist software. This process speeds up the documentation steps that are usually conducted manually before conservation treatment of the stone surfaces and facilitates decisions regarding the required cleaning techniques. The proposed method can potentially assist the identification of areas for sampling and, subsequently, the in-depth investigations through laboratory testing. An interesting perspective would be integrating with other mapping and non-destructive on-site monitoring techniques (Akoglu et al., 2020). The approach is very adaptable, as it performed well for different case studies of stone stelae that presented dissimilar surface pathology. For practical reasons, some of the steps followed in this work may be skipped. If, for example, the camera sensor cannot be geometrically or radiometrically calibrated, the proposed approach can still produce degradation maps useful for conservation purposes. Including NIR images in the approach proved to increase the accuracy of the results for some cases, especially when biodeterioration was present; however, including only the true color images still provided high-accuracy results. Therefore, an unmodified camera can also be used to provide the necessary input for ML-based segmentation.
To conclude, a critical issue observed in this work stems from the complexity of the mapped surfaces. Overlapping of degradation forms can cause some misclassifications, while more easily definable deterioration patterns can produce higher accuracy results. The subjective visual identification between less and more degraded surfaces that present the same pathology can also affect the approach's performance. Features causes by material loss cannot be easily detected if they are not the cause for other degradation patterns (such as moss caused by concentrated moisture in the created cavities), and thus combining 3D approaches could be helpful in that direction.

ACKNOWLEDGMENTS
This project (part of the 'Technologies for Cultural Heritage' Ph.D. program at the University of Turin) has received funding from the European Union's Framework Program for Research and Innovation Horizon 2020, under the H2020-Marie Skłodowska-Curie Actions-COFUND scheme (Agreement ID: 754511) and by the foundation Compagnia di San Paolo.