DETECTION AND CLASSIFICATION OF BRIDGE CROSSING EVENTS WITH GROUND-BASED INTERFEROMETRIC RADAR DATA AND MACHINE LEARNING APPROACHES

In this paper, we investigate the potential of detecting and classifying vehicle crossings (events) on bridges with ground-based interferometric radar (GBR) data and machine learning (ML) approaches. The GBR data and image data recorded by a unmanned aerial vehicle, used as ground truth, have been measured during field campaigns at three bridges in Germany non-invasively. Since traffic load of the bridges has taken place during the measurement, we have been able to monitor the bridge dynamics in terms of a vertical displacement. We introduce a methodological approach with three steps including preprocessing of the GBR data, feature extraction and well-chosen ML models. The impact of the preprocessing approaches as well as of the selected features on the classification results is evaluated. In case of the distinction between event and no event, adaptive boosting with low-pass filtering achieves the best classification results. Regarding the distinction between different class types of vehicles, random forest performs best utilising low-pass filtered GBR data. Our results reveal the potential of the GBR data combined with the respective methodological approach to detect and to classify events under real-world conditions. In conclusion, the preliminary results of this paper provide a basis for further improvements such as advanced preprocessing of the GBR data to extracted additional features which then can be used as input for the ML models.


INTRODUCTION
Worldwide, the service life of newly constructed bridge infrastructure is designed for several decades. In Germany, the targeted lifetime is about one hundred years. However, increasing traffic frequency and vehicle loads may have a negative effect on the structural integrity. For example, excitations due to crossing traffic may have higher amplitudes or may be qualitatively different compared to the considerations when planning the bridge. Therefore, a frequent monitoring is crucial for public safety.
Conventional systems used for structural health monitoring (SHM), e.g. strain gauges, require a permanent installation to an infrastructure such as a bridge. The installation process and a long-term maintenance can be difficult for only remotely accessible bridges. In addition, commonly used systems are often invasive meaning their installation introduces damage at the structure. Furthermore, analytical approaches of SHM, such as Operational Modal Analysis (OMA), investigate solely the response of the bridge based on, for example, the acceleration. In this case, however, the cause of the acceleration is not considered explicitly due to the difficulties of measuring the exact excitation. At the same time, changes of the bridge's behaviour and its dynamics can be caused by changes of the traffic volume itself (e.g. Magalhães et al., 2012). Detecting changes of a bridge's characteristics under dynamic conditions of traffic is an essential part of SHM. Altogether, there is a need for an innovative approach in SHM which includes non-invasive measurements of the dynamic behaviour of the bridge and which is able to detect and distinguish changes in the resulting data either caused by traffic or the structure itself.
To achieve these objectives, a first approach is to extract vehicle crossings on the bridge based on the signal data so that the causing excitations can be defined. In this contribution, we are following this approach to detect vehicle crossings on a bridge. For this purpose, we rely on a ground-based interferometric radar (GBR) as a sensor recording the bridge dynamics stimulated by vehicles crossing the bridge. We use solely the GBR data combined with machine learning (ML) approaches to detect and classify vehicle crossings. We avoid threshold-based methods, since they need adjustment for each structure, preventing a generalisation. To evaluate our proposed approach and its ability in the context of SHM, we choose a dataset which we have recorded in three real-world measurement campaigns at bridges in Germany. Among others, an important challenge arises from the excitation of the bridge caused by unspecified, random daily traffic. The main contributions of this paper are: • a detailed description of the measurement setup with the GBR sensor monitoring three bridges, • a profound analysis of time-series data extracted from the GBR measurements to calculate the vertical displacement of the bridge caused by vehicle crossings, • a schematic approach including all steps starting with the time-series data, preprocessing and subsequent feature extraction, • an appropriate ML framework with eight models applied to (1) detect and (2) classify vehicle crossings (events), • and finally a comprehensive evaluation of the proposed methodological approach according to the classification performance.
We give a short overview of related work regarding distinct approaches in bridge monitoring, event detection and vehicle classification in Section 2. In the following section, we describe the measurement setup at three different bridges with a GBR sensor. Section 4 introduces the acquired GBR time-series data and describes the generation of bridge displacement data. Addition-ally, we present and analyse the vehicle crossing events. Section 5 contains the proposed methodological approach consisting of three steps: preprocessing, feature extraction, as well as ML models to detect and classify vehicle crossings. The results are presented and discussed focusing on the feature importance of the ML models in Section 6. Finally, we conclude our studies in Section 7 with a brief summary and an outlook to further research topics.

RELATED WORK
Alternatively to acceleration-based or strain-based monitoring of bridges (see e.g. Cho et al., 2010), the GBR-based measurements have gained a growing reputation in structural health monitoring (SHM). This increasing interest mainly results from its simple usage and the non-invasive measuring principle to recognise the vertical displacement of a bridge. Gentile and Bernardini (2010) introduces a new microwave radar-system (IBIS-S) and compares its sensitivity to acceleration sensors. To increase the sensitivity of the GBR signal, so-called corner reflectors can be installed at the bridge. In general, the displacement sensitivity exceeds 0.02 mm depending on the underlying measurement setup. The IBIS-S is used to perform static and dynamic measurements on a bridge near Florence, Italy in order to acquire its natural frequencies (Pieraccini et al., 2007). Furthermore, Hsu et al. (2016) carry out a long-term monitoring of two highway bridges in Taiwan. After measuring each bridge annually and over 5 to 6 years, one bridge has shown a change in its natural frequencies as well as a clear visual degradation. As an alternative measurement setup of the IBIS-S, Michel and Keller (2020) proposes and evaluates the mirror mode during measurements at a bridge in Karlsruhe, Germany. This setup allows the usage of corner reflectors without the need for a permanent installation of these reflectors directly at the bridge. A corner reflector is placed opposite the GBR beneath the bridge reflecting the beam which is scattered from the flat surface of the bridge underside back to the GBR.
In the context of vehicle classification, deep learning combined with data recorded by visual sensors is commonly used (Nguyen et al., 2018). Further sensors applied for event detection and classification are strain gauge sensors. For example, Zhang et al. (2008) Rajab et al. (2014) have installed multi-element piezoelectric sensors beneath the surface of a roadway in the USA. Based on this data, they extract vehicle length, axle spacing and the number of tires as features for subsequently threshold-based classification with a overall accuracy of 86.9 %.

MEASUREMENT SETUP
To evaluate our approach for detection and classification of vehicle crossings on a bridge based on GBR measurement and ML, we rely on a dataset which has been recorded during several measurement campaigns in 2019 on three bridges in Germany. Since real-world conditions are given at all three bridges, the possibility to transfer the applied approach to further bridges is maintained.
The measurement setups at all monitored bridges are described in Section 3.1. Subsequently, we give a brief introduction of the GBR measurement principles (see section 3.2). A more detailed explanation of the GBR measurement principles with respect to the processing of the raw data is given at Coppi et al. (2010).

Measurements at Three Selected Bridges
Three selected German bridges are the centerpiece of the measurement campaigns. They have been monitored over several hours at different days and, thus, cover a broad variance of environmental influences. We have equipped two of the three bridges with corner reflectors to gain a high signal-to-noise ratio (SNR) of the GBR data. Table 1 provides an overview of selected details of all three bridges.
With respect to the vehicle crossings, we can divide the bridges into two groups. The vehicles crossing bridges A and B drive at a constant speed. In contrast, bridge C serves as entrance and exit to the state road on top of it. Therefore, vehicles will accelerate and decelerate while crossing. In our analysis, we focus solely on a single field of each bridge for reasons of comparability. Figure 1 shows the respective measurement setups at each of the three bridges from different perspectives. To explain the measuring principle of the GBR and to illustrate the GBR raw data, we use the bridge A in the following subsections as an example.

Measurements with the GBR
Two fundamental measurement principles are combined in the IBIS-S GBR to ensure the precise measurement of spatial displacements: frequency modulation and interferometry. By applying frequency modulation, the GBR monitors several points in line of sight (LOS) which we refer to as range bins. With the speed of light c = 3 × 10 8 m s −2 and a bandwidth B = 200 MHz, the spatial resolution ∆r is The LOS distance R between the GBR and each range bin equals the spatial resolution times the number of the range bin. Four of such measuring points with their corresponding numbers are highlighted in Figure 1a. The IBIS achieves an accuracy of up to 0.01 mm, depending, for example, on the inclination of the GBR (Rödelsperger et al., 2010). To monitor a bridge, the GBR is placed underneath it as shown in Figure 2. For each range bin a phase shift ∆φ is measured with a sampling rate of up to 200 Hz through interferometry. This shift can be transformed to a radial displacement ∆R along LOS according to Its high sampling rate enables IBIS-S to monitor continuously and to record vibrations caused by vehicle crossings. The vertical displacement ∆z as a projection from ∆R can be deduced from the height difference h between the bridge and the radar corresponding to

GBR TIME-SERIES DATA
First, the selection of useful range bins is described in Section 4.1 which are then used to illustrate time-series signals of crossing ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume V-1-2020, 2020 XXIV ISPRS Congress (2020 edition)   Figure 2. Schema of the measuring principle of the GBR exemplary illustrated for bridge A. R is the distance in LOS between each range bin and the GBR, h the height difference between the GBR and the lower side of the bridge. ∆R represents the measured displacement in LOS and ∆z is the respective vertical projection.
events in Section 4.2. We refer to crossing events as vehicle crossings of the bridge. Second, we will define classes to distinguish between events (see Section 4.3).

Range Bin Selection
In time-series analysis, it is essential to have a signal with a high SNR meaning a powerful signal. In the context of GBR measurements, range bins with a high SNR correspond to parts of the bridge with high backscattering. We can generate high reflectivity artificially by installing corner reflectors to the underside of a bridge. This has been performed at bridge A and bridge B. As illustrated in Figure 3, high peaks indicate data samples with a higher SNR which are used for further analysis. The marked range bins of Figure 1a correspond to measuring points with a high SNR in Figure 3. We select these four range bins for further time-domain analysis.

Time-Series Signals of Events
The local maximum displacement at a specific point (see Figure 4, second row) on the bridge depends on several parameters such as the weight of the vehicle and the driving direction. Bridge crossings of different vehicle types such as trucks or cars result in distinct time-series signals of the measured displacements as illustrated in Figure 4. When analysing the time-series signals and the corresponding events on the bridge, we suggest that (a) the relative vehicle weights can be approximated based on the maximum displacement for each vehicle and (b) the bridge site, on which the vehicle is driving, can be estimated as the relative maximum displacement of each range bin within one event. We have to distinguish two cases: In case of a single event, i.e. between the entering time and exiting time only one vehicle is on a bridge, the driving direction can be deduced from the driving site. However, at several bridges we observe multi-events, sometimes in the form of overtaking manoeuvres. For multi-events 7LPH>V@ the driving direction is more challenging to detect. It is necessary to understand which vehicles are on the bridge and which type generated the corresponding signal measured by the GBR.
To establish ground truth concerning the types of vehicles on the bridge, we have used an unmanned aerial vehicle (UAV) to monitor the surface of the bridge. The UAV was employed in measurement campaigns at the bridges A and B. After some vehicles exit the bridge, an undisturbed swing-out process can follow (see Figure 4(b) and Figure 4(c)). Note that an event is defined as solely the crossing of each vehicle without the respective swingout process. In sum, the time between the entering and the exit of a vehicle is regarded. This study focuses on the events, but as a byproduct the characteristic natural frequencies of bridges can easily be extracted from a swing-out process with a fast Fourier transform (FFT). For each bridge we extracted the first natural frequency, as shown in Table 1. Figure 5 shows the vertical displacement from the selected range bin 21 as a time series recorded for approximately 2.7 h. In general, the vertical displacement in the time series of range bins is characterised by a drift in a non-linear manner over time (see e.g. Figure 5). Similar to all electronic devices, the GBR is influenced by environmental parameters of the measurement surroundings such as temperature, relative humidity and air pressure. However, these parameters affect the GBR device on the one hand and the GBR signal during the transmission on the other hand. Furthermore, the bridge is also affected by the dynamics of the environmental parameters. For example, the bridge expands under rising temperature which leads to a slow horizontal movement of the corner reflectors. It is challenging to account for all these parameters without interfering with signal components necessary to detect events within the time-series data. A description of our applied preprocessing follows in Section 5.1.

Class Extraction
To define potential classes of events, respectively vehicle crossings on the bridge, we analyse the duration of different events vs. the maximal vertical displacement (see Figure 6). The event duration depends on the speed of each vehicle, but the bridge is usually crossed with approximately constant pace. Figure 6 7LPH>V@ 9HUWLFDO'LVSODFHPHQW>PP@ 5DQJH%LQ Figure 5. The vertical displacement of the selected range bin 12 as a time series over about 10 000 s. The occurring peaks in the time series correspond to different events. The time-series data has been low-pass filtered (see Section 5.1).
shows three distinct event classes. Class 1 contains for example cars, class 2 contains vehicles such as small trucks and class 3 contains trucks. We have evaluated these three classes based on high-resolution UAV data. Note that the defined classes do not correspond to official vehicle types. For a preliminary classification presented in this study, we rely on the three classes defined above.

METHODOLOGY
Our proposed methodological approach consists of three steps: the different preprocessing approaches, the feature extraction and the ML models to classify the events. Figure 7 provides the schema of the three applied steps.

Preprocessing
The GBR monitors a bridge dynamically with a sampling frequency of up to 200 Hz. However, most significant signal components of bridges are to be expected in a lower frequency range (Mehlhorn and Curbach, 2014). Therefore, low-pass filtering is a useful preprocesssing step to suppress high-frequency noise. We apply a Butterworth filter (Bianchi and Sorrentino, 2007) with two different settings which we refer to a LP1 and LP2. The distinction between LP1 and LP2 is that the latter has a higher degree of filtering than LP1. The raw GBR data without any filtering is defined as baseline.
We analyse different approaches for preprocessing (see Figure 7: None, LP1, LP2). In the test phase of the ML models we compare the results of the classification based on features extracted from the three preprocessing approaches. Note that no scaling has been applied on the time series and the features in order to avoid misclassification of events including heavy vehicles. Table 2 shows the number of labelled data samples according to each bridge and event class as illustrated in Figure 6. The event duration is not constant. Since the further steps of our methodological approach require features extracted from time-series data of equal length, all events are split into multiple time series of a length of 0.5 s. For each bridge, we consider only the range bin with the highest SNR. As a result, the complete dataset consists of 4490 time-series samples of 0.5 s for each preprocessing approach. In addition to real events, we include time-series samples with no event on the bridges to the current dataset (Class 4). We do not balance our data since it is not possible without heavily reducing the dataset considering there are only few Class 3 events. For comparability, we also omit balancing for the event detection task.

Feature Extraction
Seven features are extracted from each time-series sample. We use common statistical and signal processing values such as the variance and the signal energy. Since the displacement signal of the GBR has a baseline drift caused by environmental parameters (see Section 4.2), we have excluded features such as the mean value which are sensitive to this drift. Where meaningful, e.g. Maximum, Minimum and Energy, the influence of the mean value was eliminated. All the features are extracted directly from the time series (time domain) without performing a modal analysis.   Table 3. Seven features extracted from the GBR time-series data.
x represents one data sample of 0.5 s.
Feature No. Name of Feature Formula Kurtosis mean((x − mean(x)) 4 ) In Table 3, the extracted features as well as their corresponding formulas are summarised.

Machine Learning Models
In the last step (see Figure 7), we select eight ML models (1) to distinguish between event and no event and (2) to detect events and classify these events according to four proposed class types.
Before the training the complete dataset is randomly split into a training subset and test subset in the ration 80:20. All selected ML models are trained on the training subset using the extracted features as input and the class labels as target label. The hyperparameters for each model are found using grid search. Note that only the SOM performs the training phase unsupervised while all other models are supervised learners. For details on the SOM specifications see Riese et al. (2019).
During the subsequent test phase, the trained classification models classify (1) the events meaning event vs. no event as well as (2) the four classes based on the extracted features of the test subset. The predicted classes are compared to the labelled values. The classification performance is expressed by the overall accuracy (OA), precision (P) and recall (RC). Ensemble models such as ET and AB provide the feature importance of the input features as further information of the classification task.

RESULTS AND DISCUSSION
The main objective of this study is to evaluate the potential of ML models to detect and classify events based on GBR data from bridge monitoring. Such events are, for example, vehicle crossings on the bridge. Table 4 shows the results for the event vs. no event classification task of the selected ML models and the three preprocessing approaches. When comparing the baseline, i.e. no applied preprocessing and hence more high frequency noise, to the LP1 and LP2 filtered preprocessing approach, we obtain a better classification performance in terms of OA. By applying the two lowpass filters as preprocessing, the eight ML models perform differently. Within the ensemble methods, ET, for example, achieves the highest OA for the unfiltered input data. The filtered input data leads to a lower OA. In contrast, RF performs the best with the most filtered input data (OA of 83.2 %) as well as AB with an OA of 83.8 %. With focus on P and RC, both measures are low despite OAs over 70 %. The reasons for this is the unbalanced input data (see Section 5.2).
1 scikit-learn 2 Implementation from Riese (2019) The classification results of the class type classification are similar to the results of the event vs. no event classification. For LP1 vs. LP2 half of the ML models achieve higher OA with the LP2 preprocessing approach (see Table 5). As the class type classification is a more sophisticated task than the distinction between event and no event, high-frequency noise has a more negative effect on the classification performance (see Table 5, Baseline columns). ANN trained on the LP2-filtered data has the highest OA of 77.5 %. With the LP1 filtered data, AB classifies the class types best. Although with LP1 RF is the best ML classifier for the event vs. no event classification, it performs poorly in the task of distinguishing several vehicle classes based on the same data.
Since the difference in OA between LP1 and LP2 event detection for RF and AB is only small, we will use LP1 in the remainder in order to avoid suppressing potential significant bridge deformations. Figure 8 shows the relative feature importance of the input features (see Table 3) of the LP1 preprocessing approach provided by the ensemble models, RF and AB. We select these two ML models since RF is the best ML model to detect events with LP1 preprocessing and AB is the best ML model to classify class types with the LP1 preprocessing. As for the differences in the classification performance of RF and AB, the latter has a balanced classification performance for the two underlying tasks, while RF has a significant lower OA for LP1 class type classification. When only focusing on the features which are important for both classification tasks, RF priors the features energy, variance and standard deviation in this order. However, AB either omits all three features or rates them low. Especially the relative weighting of the variance shows strong deviations within both models. This finding suggests that the feature variance can be useful as an indicator for event detection (event vs. no event) but it can be omitted for the decision of the class type. We can observe, that the feature kurtosis is dropped by RF completely as for AB, it is an important feature for the class type classification.
In sum, the classification performance of the ML models is satisfying. The best classification performance independent of the classification tasks is an OA of 83.8 %. We state that for any classification improvement, the selection of the class types depending on the vehicles needs to be enhanced. In addition, we cannot create further features out of the GBR data due to the included drift. Thus, we need to apply advanced preprocessing approaches to eliminate this drift and to extract more potential features. Also note that the impact of further features in the ML classification tasks needs to be evaluated with respect to each model. To allow for an unknown weight range, we have omitted data scaling which can be another factor influencing the prediction accuracy  Table 3.
(see Section 5.1). Obviously, the classification performance depends highly on the selected time-series data. For example, we may miss the start of an event, if the impact of the increased bending of the bridge is too small to be detected in the respective window. To avoid these kind of misclassifications, we plan to use overlapping windows in further studies.
To summarise and visualise our results, we apply the two ML models performing the best with LP1 preprocessing on the event detection task (RF and AB) to classify a time series which is previously unknown by these two models. We then combine these two models in a voting classifier implementing soft voting to detect events (see Figure 9). By employing data of an unknown range bin with a different GBR position, we demonstrate the potential of the combination of ML approaches and GBR data to generalise in case of a event vs. no event classification. As Section 4.2 shows, all events of this 350 s time-series section are classified and labelled. Bridges are dynamic infrastructure objects and vehicle crossings stimulate their oscillation. Hence, we have to deal with swing-out processes which are characterised partly by high amplitudes. As the classification of the events in general and the feature extraction is based on, for example, ratios between high amplitudes, the swing-out processes are challenging. At several occasions noise is falsely detected as an event, mirroring the precision of around 75.8 % and 74.9 % for RF and AB.

CONCLUSION
In this paper, we address the challenge of detecting vehicle crossings, which we refer to as events, and their classification based on GBR data using ML. In contrast to widely used event detection approaches, which exploit acceleration sensors or strain gauge data, we extract the bridge displacement directly from GBR timeseries data avoiding any threshold-based approaches. The GBR time-series data has been recorded during real-world measurements at three bridges in Germany with random daily traffic. As ground-truth data for the vehicle types we have used UAV data of the vehicles that have been on the respective bridge during the measurement. Our main objective is to investigate the potential of GBR data for (1) detecting events and (2) classifying time-series excerpts according to four class types of vehicles.
To solve these classification tasks, we introduce a methodological approach involving an optional preprocessing step with lowpass filtering and eight ML models. The classification results of both cases reveal the high potential of data-driven models and the exploited GBR input data at varying measurement conditions at the three bridges. Furthermore, the performance of the methodological approach is satisfying with respect to the limited size of (ground-truth) data and without providing any prior domainknowledge. Except for the low-pass filtering, no further preprocessing of the GBR data has been applied. Thus, the mentioned drift in the GBR time-series data, caused, among others, by environmental influences during the measurements, has not been excluded in this preliminary study. 7LPH>V@ When focusing on the monitoring of the bridge dynamics under load, the application of GBR sensors provide a non-invasive and remote alternative to sensors which have to be directly installed on the bridge or even in the bridge structure. Therefore, GBR sensors can contribute to structural health monitoring in combination with appropriate methodological approaches.
To evaluate the generalisation abilities of the proposed approach, we apply a voting classifier (RF and AB) on time-series data of an so far unknown range bin from data of another GBR sensor at Bridge A. Aiming at distinguishing between event and no event, the classification results are satisfying in general. Although, in some cases, noise is misclassified as an event, the events are all detected properly.
The results of this first study are very promising. As a direct consequence, we will further improve the ML-based approaches in order to remove the baseline drift. With respect to the class type classification of different vehicles, we will then be able to extract additional features from the time-series GBR data. Then, we can evaluate the impact of these features on the classification performance due to avoiding misclassifications caused by longterm drifts. Furthermore, we consider to include a new class which covers the swing-out processes. To evaluate the generalisation abilities in detail, we will expand the dataset in upcoming measurement campaigns. Additionally, we will exploit further range bins of each bridge. Finally, to overcome the limited size of labelled data, extended UAV flights simultaneously to the GBR measurements are planned. As the dataset size is increased, advanced ML approaches such as convolutional neural network can be applied and evaluated.