IMAGE-BASED RECONSTRUCTION AND ANALYSIS OF DYNAMIC SCENES IN A LANDSLIDE SIMULATION FACILITY

The application of image processing and photogrammetric techniques to dynamic reconstruction of landslide simulations in a scaleddown facility is described. Simulations are also used here for active-learning purpose: students are helped understand how physical processes happen and which kinds of observations may be obtained from a sensor network. In particular, the use of digital images to obtain multi-temporal information is presented. On one side, using a multi-view sensor set up based on four synchronized GoPro 4 Black® cameras, a 4D (3D spatial position and time) reconstruction of the dynamic scene is obtained through the composition of several 3D models obtained from dense image matching. The final textured 4D model allows one to revisit in dynamic and interactive mode a completed experiment at any time. On the other side, a digital image correlation (DIC) technique has been used to track surface point displacements from the image sequence obtained from the camera in front of the simulation facility. While the 4D model may provide a qualitative description and documentation of the experiment running, DIC analysis output quantitative information such as local point displacements and velocities, to be related to physical processes and to other observations. All the hardware and software equipment adopted for the photogrammetric reconstruction has been based on low-cost and open-source solutions.


INTRODUCTION
Scaled-down facilities for landslide simulation have been largely developed for research purpose (see a review in Scaioni et al., 2013).Besides this application domain, such platforms may be also used as tools for active-learning in education environments (Rutzinger et al., 2016): students have the chance to set up a complete landslide simulation experiment as well as to see somehow the soil configuration (material, slope, compactness, moisture, etc.) and the triggering factor (i.e., artificially induced rainfall) may control the development of a slope failure.This way, students may physically realize geohazard events that are very difficult to see directly, even better than watching videos.The deployment of a sensor network may be also used to demonstrate somehow different techniques are able to gather observations inherent to the prediction of a slope failure.Such a landslide simulation facility (see Sect. 2) has been implemented on Lecco Campus of Politecnico di Milano university, Italy.Beyond its employment for testing the behaviour of different types of soil material in slopes solicited by intense rainfall, this platform has been used during spring 2017 as an active-learning tool for students engaged in the Civil Engineering for Risk Mitigation MSc degree course (www.master-cerm.polimi.it).As illustrated in Subsection 2.2, different types of sensors have been implemented to record parameters, signals or images/videos during experiments.Among these sensors, some innovative technologies have been applied as well to explore their use in Landslide Science.For example, geoelectrical monitoring (Supper et al., 2014) and a 1 Corresponding author coherent fibre optic-based system (Ferrario et al., 2016) have been used.
In this paper the focus is given to the use of a system consisting of four GoPro 4 Black ® cameras to capture multi-view stereo (MVS) image sequences of a landslide simulation experiment.These have been processed using a popular low-cost photogrammetric software package for close-range photogrammetry, i.e., Agisoft Photoscan Professional ® (in the following APP) ver.1.2.5 (www.agisoft.com).The output consists on a 4D model of the simulation experiment, which is made up of a sequence of 3D coregistered models showing the evolution of the slope failure within different epochs up to the final collapse.The resulting 4D model offers the opportunity to be seen and analysed at variable speed, to allow measurements between points in each 3D model and across diverse epochs, and to change the viewpoint at any time.The 4D model becomes itself an additional active-learning tool that may be exploited to show somehow a certain type of landslides or soil erosion processes (Rieke-Zapp & Nearing, 2005;Hungr et al., 2014, Longoni et al., 2016) happened depending on specific input parameters and boundary conditions.
In Subsection 3.1, after reviewing the application of imagebased systems in landslide simulation facilities, the hardware applied here is described.In Subsection 3.2, the automatic processing workflow implemented within APP is addressed.Notwithstanding this software package is well known and static 3D reconstructions are easy to be obtained, its application to derive dynamic 3D models is less common.Consequently, it is worth to communicate to the scientific community the methodology to obtain such 4D models that are fully compliant with the concept of Virtual Geology (Jaboyedoff et al., 2015).In general, in Virtual Geology static 3D models and point clouds are used, but dynamically evolving 4D models might be also entailed.In Subsection 3.4, the application of digital image correlation (DIC -see Baker et al., 2011) has been focused to track surface points from the image sequence obtained from the camera in front of the simulation facility.While the 4D model may provide a qualitative description and documentation of the experiment running, DIC analysis outputs quantitative information such as local point displacements and velocities, to be related to physical processes and to other observations.Eventually, Section 5 discusses and reviews the main key-points of the methodologies described in this paper, and tries to highlight open problems and future developments.

Description of landslide simulator
A landslide simulator is a laboratory machine that reproduces on small scale a real slope with its characteristics and destabilizing conditions.With this simulator, we may study somehow external triggering factors (e.g., rainfall or earthquake) as well as some slope parameters (e.g., inclination) may influence the stability and trigger a landslide.The experimental work on a laboratory model allows for the investigation of various phases which characterize the instability conditions of a shallow landslide: from the onset to the kinematics and post-event study.
The main aim of the landslide simulation platform built up on Lecco Campus of Politecnico di Milano is to investigate the modes of collapse and the factors which may control it: material property, rainfall precipitation intensity, initial moisture content, as well as gradient of the inclined topographic surface.The simulator is composed of two adjustable metallic surfaces.The upper part of the flume has dimensions 2 m x 0.8 m and could be lifted up to an inclination of 45°.The lateral sides of this flume are made of plexiglass for visual inspection while the bottom is supplied with a geogrid so as to retain friction between the soil and the structure.Rainfall is simulated with a sprinkler system designed for the purpose.Initially, the material used for landslide simulations is homogeneous fine sand.In Figure 1 the landslide simulation facility is shown.

Implemented sensor network
The sensor network that may be installed on the landslide simulation platform is made up of several instruments to monitor the ongoing processes.So far, no efforts have been put on the automation and centralization of data acquisition.This is motivated by the aim to set up a prototype simulation platform and to test new types of sensor technologies, rather than to establish a platform for routinary operations.In addition, large space has been given to the use of this simulation facility for active-learning purpose within the MSc and PhD programmes at the university.
A TDR (time-domain reflectometer - Su et al., 2009) probe was inserted at a predefined position within the soil layer in order to keep record of the volumetric water content during the experiment.Three piezometers were buried at different positions along the slope to measure the water table level.To monitor the variations in water distribution along the main axis of the slope we applied the DC-resistivity method by installing a miniaturized array of 48 electrodes.This method is normally used for geophysical investigations and in recent years its use as a permanent monitoring system has been tested by several authors for applications on landslides and artificial earth structures such as dams and levees (Arosio et al., 2017).Ultimately, two kinds of fibre optical-based sensors were buried in the soil in order to monitor precursory signals of displacements, which could indicate the onset of instability within the terrain during a landslide simulation experiment (Ferrario et al., 2016).Sensors were disposed in different configurations so as to define the best setup for monitoring.
The platform may be also equipped with imaging systems.These have a twofold aim: (i) capturing videos for documentation and archiving of the experiments; and (ii) applying images for 2D image-based analysis and 3D photogrammetric reconstructions.Imaging sensors may be hung up to the upper part of the outer frame, and also installed on stable tripods in front of the scaledown landslide model.The presentation of some results obtained using the imaging system implementing GoPro 4 Black ® cameras will be the main aim of this paper (see.Sections 3 and 4).

State-of-the-art
The small/medium size of typical scaled-down landslide simulators perfectly fits with the application of closerange/terrestrial photogrammetry, see Luhmann et al. (2014).In fact, such simulation facilities span over a volume of a few cubic metres in laboratory implementations, and a few hundreds when they are constructed on natural slopes (see, e.g., Travelletti et al., 2008;Ochiai et al. 2004).
In general, applications reported in the literature may be grouped in two main categories (see Scaioni et al., 2015): 1. Single-camera systems, where the same scene is focused from a fixed camera, which may be positioned in front of the simulation facility or on a lateral or upper setup.In the case of a single image, no 3D information may be extracted in object space, unless the object is flat or a digital surface model (DSM) is already available; and 2. Stereo-camera systems, where two cameras with parallel or slightly convergent axes are used to capture the scene during the experiment.The chance to record stereoimages allows for 3D reconstruction to be transformed in object space using ground control points (GCP's).
Of course, more than two cameras (MVS systems/ photogrammetric networks) may be also installed with the aim of rising up the redundancy of the observations (see Luhmann et al., 2014) or to cover more sub-portions of the same facility (as in the case described in Subsect.3.2).
In the Case (1), distinctive natural or artificial features (markers) may be automatically recognized and tracked within the image sequence using an optical flow technique (Baker et al., 2011;Chao et al., 2014).Some relevant profiles may be also extracted, as reported in Scaioni et al. (2015).As mentioned above, only in the case the object is flat, the reconstruction may be completely transformed from image (2D) to object (3D) space in non-ambiguous way.A homography transformation is applied, see Hartley & Zisserman (2006).But in the general case when the object has a real 3D shape, the reconstruction from a singlecamera sequence cannot be accomplished.A 3D model of the object, if available, may be exploited to project 2D information obtained in image space (for example, feature displacement vectors) on 3D space, as proposed in Travelletti et al. (2012).On the other hand, during the most landslide simulation experiments the surface topography undergo severe changes.Consequently, the initial 3D model might not be adequate to project all the images of the sequence.This limitation is overcome by using a couple of fixed cameras (Case 2).Although in such a case the complexity of the system may be involved, the quality of outputs is also increased.In the case of very fast simulation processes, the synchronization of both sensors become a critical issue, to be obtained using solution at hardware or image-processing level (see Raguse and Heipke, 2009;Li et al., 2017).
While the calibration and orientation of cameras are quite standard in both Cases (1) and ( 2) and in multi-station networks (see Luhmann et al., 2014), the extraction of relevant information from the image sequence is the crucial point to help understand the ongoing geo-processes.Single-image sequences may be exploited to derive point displacements and velocity field, to be displayed using graphical tools.These outputs may be compared and correlated with observations from other sensors and with major events occurred during a simulation (for example, when intermediate failures happen), as carried out in Scaioni et al. (2013).Stereo-images may output 3D models to describe surface changes during landslide simulation, to compute volumes, to study soil erosion processes and other geomorphic features.Surface features and their displacements may be also tracked in stereo-pairs (Feng et al., 2016).

Hardware implementation of the imaging system
The landslide simulation facility presented in Section 2 has been equipped with a MVS camera setup consisting of four GoPro 4 Black ® (www.gopro.com)sensors.This kind of cameras that has been designed for the acquisition of videos and image sequences during outdoor leisure and sport activities, has some characteristics that deserve consideration for photogrammetric applications.Indeed, GoPro 4 Black ® cameras are very easy to use, lightweight, robust as well as waterproof.This latest property is quite important for the implementation in the landslide simulation facility, because of the water poured from the upper part to simulate rainfall.The GoPro 4 Black ® camera has also fix focal length, property that avoids accidental changes of this characteristic during image acquisition with consequent problems in the interior orientation (see Par. 3.3.1).The timelapse data acquisition mode allows the recording of an image sequence to be used for further processing.In addition, the GoPro Smart Remote ® controller may be used to set up and make synchronous the acquisition of image sequences by multiple GoPro 4 Black ® cameras.The cost of the adopted camera is quite small, and this help the sustainability of its application.Roughly, a set of four GoPro 4 Black ® cameras and the remote controller may totally cost approximately 1,500 Euros.In Table 1, some technical properties of the adopted cameras are shown.Four GoPro 4 Black ® cameras have been installed on the outer frame of the landslide simulator according to the scheme shown in Figure 2. The position and spatial orientation of each camera has been fixed so that the whole area interested by the expected failure process is captured in at least two sequences.Four sensors have been installed on the top of the platform looking downwards.Special care should be put on fixing cameras in stable positions, and on controlling the light conditions.The presence of external objects that could shadow the slope should to be avoid, since it may heavily interfere with image processing.

Camera features Values
A set of 17 markers has been temporarily positioned on the slope to be used as GCP's to establish the object reference system for the photogrammetric reconstruction.Other GCP's have been also placed on the outer frame of the platform, which may assume a variable geometry thanks to the possibility of changing the inclination of the slope bed.All GCP's have been measured with a total station, so that their position in a topographic reference system could be determined (i.e., with the z axis aligned along the vertical plumb line).This type of geo-referencing is important since the inclination of the slope has to be known.In the following Subsection 3.3, some issues related to the following processing steps of the image sequences collected using GoPro ® cameras during the simulation experiment are reported.

Camera calibration:
At the current state-of-the-art of close-range photogrammetry (see Luhmann et al., 2016), in the most projects a large number of images are captured and used for static 3D reconstructions.Consequently, the camera selfcalibration may be accomplished concurrently with the orientation of images based on a Structure-from-Motion (SfM) approach, as implemented in APP (Hartley and Zissermann, 2006;Luhmann et al., 2014).On the other hand, when the number of camera stations is rather small (less than five images), self-calibration is not the optimal solution, but each camera should be independently calibrated.Then the calibration parameters of each camera are used as input data during SfM and successive dense image matching.For a stable independent calibration of each GoPro 4 Black ® camera, at least 16 images have been captured to focus a set of coded markers provided with APP.Markers were placed on the floor surface but at variable heights.Convergent photos have been taken from different viewpoints, including some photos with the camera rolled 90° to mitigate potential correlations between EO and calibration parameters (Zhang et al., 2018).The adopted calibration setup is displayed in Figure 3. GoPro 4 Black ® cameras may gather time-lapse image sequences at full or reduced format size (see Table 1).Here the largest (12 MPx) format size has been used.Of course, the calibration has been done using images at the same size.
The calibration block has been processed in APP by using standard Brown's model (see Luhmann et al., 2014).Markers have been automatically detected and measured, to be used as observations for a free-net bundle block adjustment (BBA).This has worked out the EO of each camera and the set of calibration parameters, which have been saved and exported in a XML file to be used afterwards.It should be noticed that the following calibration parameters have been estimated according to the Brown's model: principal point offset (xp, yp), four coefficients for compensating radial-symmetric distortion (k1, k2, k3, k4), affine distortion coefficients (b1, b2), and two coefficients for compensating decentering distortion (p1, p2).In Table 2, the estimated parameters obtained for one of the four cameras are shown.The accuracy of calibration parameters was checked out by using a test field including a subset of independent check points as proposed in Zhang et al. (2018).Root mean squared errors (RMSE) resulted in the range between 1-2 mm, depending on the camera.Table 2. Example of camera calibration parameters estimated for one of GoPro 4 Black ® cameras.

Data acquisition:
After the setup of all cameras and before starting the simulation experiment, some removable coded markers (GCP's) were placed on the slope and measured with a total station.This phase can be seen in Figure 1.
A set of images have been collected before removing markers, to be used for computing the EO at post-processing stage (Epoch t0).After this task, markers have been removed and the experiment has been run up to the slope failure (Epochs t1tn).Concurrently, images from the four sequences have been captured using an acquisition rate of 0.5 frame-per-seconds (fps), according to the expected speed of displacements and not to miss intermediate events.Of course, the frame rate depends on the type of process: if the slope failure evolves slowly, the acquisition rate will be slower.The selected rate has resulted in sequences made up of 2,560 images each for a total time of approximately 40 minutes.As already introduced, the synchronization has been gained by using the GoPro Smart Remote ® control as common external trigger.
Image sequences have been structured in distinct directories.When a sequence is uploaded in an APP project, the function 'Workflow->Add folder' allows to add a full sequence.By adding more directories, multiple sequences may be read.Each sequence is automatically assigned a camera, whose calibration may be imported by loading the corresponding XML file.When processing more full sequences, the user may select to compute a specific task (e.g., 'Image alignment,' i.e., EO) on the images corresponding to a single frame rather than repeating in autonomous way the same task on all frames, as in the case of dense matching.

Camera exterior orientation:
The block of four images acquired at Epoch t0 have been used to compute the EO.In a first stage, the EO was computed in a free-net fashion using the SfM function implemented in APP ('Image alignment').This task resulted in a RMSE of re-projected tie points (TP) automatically extracted and matched during SfM of 1.3 pixels.This result shows the intrinsic precision of computed EO.In a second stage, six GCP's were manually measured on the images.The BBA including GCP 3D coordinates in the topographic ground reference system has yielded the definitive EO of four camera stations.RMSE of residuals on 3D coordinates of GCP's has resulted 1.3 mm.This result also has confirmed the absolute accuracy of EO, to be used for all the remaining images from Epochs t1 to tn, since cameras were supposed to remain stable during the entire experiment.

Dense image matching to provide 3D models:
Each block of four synchronous images corresponding to each generic Epoch ti has been processed in APP to obtain a 'Dense point cloud' for modelling the surface of the slope.The main parameters to control this task are the reconstruction 'Quality' level (from 'Low' to 'Very-high') and the depth filtering (from 'No one' to 'Aggressive').The former parameter affects the resolution of the images adopted in dense matching stage: in the highest 'Quality' level, the full resolution of the images is exploited; in the lowest, subsampled images are employed.Besides the fact that a higher 'Quality' level implies a longer processing time, this option may not work well in the case of noisy images, for which a 'Medium' or 'Low Quality' might be preferred.For example, in the case of the reconstruction of a single epoch of the simulated landslide, the 'Low' level has provided more complete models than the top levels.Thus, the choice of the 'Quality' level also depends on the quality of the images.The dense cloud obtained per each epoch is made up of approximately 150,000 points.

Building of dynamic 4D model:
The time series of n 3D point clouds may be composed together to create a 4D model.This operation has been preceded by the creation of a triangular mesh to be textured with the image content.These tasks have been automatically done on all the frames of the sequence.
Due to the large numbers of frames (2,560), two different 4D models have been produced: 1.A 'complete 4D model' covering the entire experiment but with a reduced number of images (in total #504), corresponding to 0.1 fps; 2. A 'detailed 4D model' at the original acquisition rate (0.5 fps) but limited on a time window starting after 23.2 minutes from the beginning, when the first crack appears, and ending when the second slope failure happens (#556 overall images).
A video illustrating the final 4D models may be retrieved online at the following link: goo.gl/1GDyKR.

Methodology:
The application of DIC has been intended to provide quantitative information on the slope surface displacements and velocities.In order to do this, the image sequence recorded by GoPro 4 Black ® camera No. 3 (see Fig. 2) has been selected.Indeed, this sequence may provide the best view on the area where the largest cracks occurred.To remove the effect of lens distortion, previously computed calibration parameters (Par.3.3.1)have been used to output distortion-free images from APP.
An open-source code for DIC running in Matlab ® environment has been utilized (Eberl et al., 2010).This code requires the list of the images in the sequence to be analysed, whose names have to be specifically codified.The same time rate (0.1 fps) adopted for computing the 'complete 4D model' has been used here.A grid of points to be tracked along the sequence has to be defined on the first image.In this case, a grid made up of 25 x 27 points has been set up with spacing 50 pixels x 50 pixels along both orthogonal directions of the grid, respectively (see Fig. 4).The DIC code has output displacements in correspondence of grid points.These observations have been done in pixel coordinates.Transformation into metric units has been carried out by using the average pixel size (0.42 mm/pixel).The DIC analysis has suffered from the presence of shadows on the slope.These prevented to track point displacements in a central region and in other smaller areas.A video showing the surface displacement field may be found at web link: goo.gl/15fH82.

Results
From both the 4D model and the analysis of surface point displacements obtained from DIC, the landslide dynamic may be separated in three blocks that moved independently, as can be seen in Figure 4 on the right column.In Figure 3, 3D models and maps of contemporary point displacements from DIC analysis are reported in correspondence of four relevant steps of the experiment running.Each 3D model extracted from the 4D model is displayed using two-colour anaglyph representation, which allows 3D stereo-vision when watching with proper glasses.A suck kind of low-cost 3D visualization also contributes to render a Virtual-Reality documentation of the experiment.
In order to carry out a deeper analysis, three specific points have been identified in correspondence of Piezometer 1 in the lower part of the slope, Piezometer 2 in the central part, and Piezometer 3 in the upper part (Fig. 5).Since single points tracked with DIC are noisy, the average surface displacement in the nearby of each piezometer has been computed from points tracked within a radius of 10 cm.Displacements in correspondence of Piezometers 1 and 2 appear quite significant, as shown in the plot in the upper part of Figure 6.On the other hand, Piezometer 3 seems to be quite stable and it will not be considered in the following analysis.The time series of point displacements corresponding to Piezometers 1 and 2 have been used to compute local velocities, as can be seen in the plot in the lower part of Figure 6.Velocity in correspondence of Piezometer 2 has slowed down immediately after the first collapse, and then has started again to go up.

Discussion
A first consideration should be made about the adopted equipment for the imaging system.The use of GoPro 4 Black ® cameras is positively evaluated, because of low-cost, robustness, user-friendliness, chance to synchronize more cameras using a specific tool (GoPro Smart Remote® control).The calibration and orientation of this kind of sensors has been possible using standard procedures and mathematical models adopted in closerange photogrammetry.The recourse to specific models (Perfetti et al., 2017) for wide-angle lens has not been necessary, despite of the short focal lens.In general, the use of other models of GoPro ® series resulted in some severe problems with camera calibration, preventing their use at the maximum sensor format available.
The APP software package has provided a satisfying environment to process the image sequences and to obtain the final 4D models at different time rates.Coupling APP and GoPro 4 Black ® cameras offered a complete low-cost system for dynamic image acquisition and modelling.In addition, the availability of 30-days demo licenses offer the students the opportunity to develop their applications and to be really engaged in an active-learning fashion.
The integration of 4D models and DIC analysis has yielded a complete description of the experiment.The former provides a qualitative visualization of what happened, at a level of realism quite superior to standard 2D videos or images.In addition, such a 4D model has a metric structure that also allows to derive geometric quantities (point coordinates, distances, areas, volumes) within and across 3D models.The latter yielded a precise displacement field of the scaled-down slope model during the development of a rainfall-induced landslide.
Of course, the full potential of this analysis in the research field may be achieved when those outputs from image-based analysis are integrated and compared to other sensors, as demonstrated in Feng et al. (2016).

CONCLUSION AND FUTURE DEVELOPMENTS
In this paper the application of an imaging system made up of four low-cost GoPro 4 Black ® cameras in a scaled-down landslide simulation facility is presented a discussed.By using a popular low-cost software package (Agisoft Photoscan Professional ® ), a 4D model describing the changes of the 3D surface of the slope within a simulation experiment was obtained.Concurrently, one of the image sequences was used for digital image correlation analysis to track surface point displacements.Coupling these two techniques offered a documentation of the experiment under both qualitative and quantitative viewpoints.
The imaging system perfectly fits in a corroborative manner within the sensor network that may be installed on the simulation platform.This option offers unprecedented opportunity to study materials, mitigation solutions and mechanical models in Landslide Science.
In the future, the development of the simulation platform and related research will go in the direction of testing new sensors (for example, 3D gaming cameras) and to analyse and integrate data already archived from completed experiments.
On the other hand, in this paper we also demonstrated somehow such a simulation platform may be an important tool to do innovative teaching by using active-learning techniques.
Tresoldi.We are thankful to Fondazione Cariplo (grant No. 2016-0785) for partial support of this research.

Figure 1 .
Figure 1.The landslide simulation facility set up on Lecco Campus of Politecnico di Milano, Italy.

Figure 2 .
Figure 2. Locations of four GoPro 4 Black ® cameras on the landslide simulator.The bottom figure also shows the field of view of each camera.

Figure 4 .
Figure 4. On the left column, 3D models corresponding to relevant events during landslide simulation are depicted in anaglyph representation for 3D stereovision with coloured glasses.On the right column, corresponding point displacements obtained from DIC analysis are shown: blue circles represent positions of initial surface points to track, while red crosses give the shifted position of points at the same time of the contemporary 3D model.

Figure 5 .
Figure 5. Positions of piezometers: layout scheme (on the left); photo of the experiment setup after inserting piezometers in the ground (on the centre); and 3D model showing how the slope model has split in three portions, each of them including a piezometer.

Figure 6 .
Figure 6.Plots displaying displacements (on the top) and corresponding velocities (on the bottom) of Piezometers 1 and 2, as measured using DIC analysis of the sequence from GoPro 4 Black ® camera No. 3.

Table 1 .
Main technical properties of GoPro 4 Black ® cameras.