STRUCTURED-LIGHT BASED 3D LASER SCANNING OF SEMI-SUBMERGED STRUCTURES

In this work we look at 3D acquisition of semi-submerged structures with a triangulation based underwater laser scanning system. The motivation is that we want to simultaneously capture data above and below water to create a consistent model without any gaps. The employed structured light scanner consist of a machine vision camera and a green line laser. In order to reconstruct precise surface models of the object it is necessary to model and correct for the refraction of the laser line and camera rays at the water-air boundary. We derive a geometric model for the refraction at the air-water interface and propose a method for correcting the scans. Furthermore, we show how the water surface is directly estimated from sensor data. The approach is verified using scans captured with an industrial manipulator to achieve reproducable scanner trajectories with different incident angles. We show that the proposed method is effective for refractive correction and that it can be applied directly to the raw sensor data without requiring any external markers or targets.


INTRODUCTION
Certain applications require scanning objects that are partially submerged in the water.Removing the object of interest from the water is undesirable or even impossible because this can result in damage or changes of the scene.In particular, forensic and archaeological applications ideally require capturing the scene as it is to ensure authenticity and integrity of the acquired data.
Another example are surveys of ship accidents, such as the collision of the Costa Concordia off the coast of Isola del Giglio in 2012.In this particular case different techniques were used to capture the damage to the Costa Concordia.Menna et al. (2013) applied photogrammetry above the water and underwater to create a complete 3D model of the ship.For aligning the models captured in air and in water with each other, multiple photogrammetric targets were mounted to the ship hull (Menna et al., 2015).Another survey was conducted by "ADUS DeepOcean", using a terrestrial laser scanner combined with a triangulation based underwater LIDAR system and sonar imaging (ADUS DeepOcean, 2013).The sensors were mounted to a ship, using a pole with the terrestrial scanner mounted on top of the pole to scan above water and the subsea LiDAR and multibeam sonar mounted below water to simultaneously capture the ship hull underwater.Navigation data was recorded using global navigation satellite system (GNSS) and an inertial navigation system (INS).
If the above-water and the underwater data is captured separately, it is sometimes difficult to capture a consistent model without gaps.Tidal changes between multiple scans can help to create some overlap.To apply laser scanning to semi-submerged structures, only few commercially available laser scanners can be applied in air as well as in water.Many terrestrial laser scanners employ infrared wavelengths which cannot penetrate far into the water.Some aerial laser scanning systems employ green wavelengths and can penetrate multiple meters into the water.However, typically these scanners are not built for underwater use and quite expensive.In the presented work we are working towards capturing abovethe-water and underwater 3D information with a single sensor system.To this end we employ a prototype triangulation based underwater laser scanner as depicted in Fig. 1, and investigate if it is possible to scan through the water surface and correct for refraction.In this work we assume the water surface to be completely flat.Since this is the first phase of development, we do not consider surface effects, e.g., wavy water surfaces, because of the difficulty to measure and estimate robust local water surface normals, and the additional modeling complexity.The goal is to simultaneously acquire 3D information in air as well as in water to capture a scan without any gaps.
We propose a structured light based scanner setup and ray-based refractive correction approach for correcting 3D scans captured through the water surface.In experiments we demonstrate the effectiveness of the method and show how it can be applied without requiring any external markers to detect the water surface.

RELATED WORK
Correction of optical measurements through the water surface has been studied for various applications of laser scanning and photogrammetry, e.g, airborne laser scanning of coast lines (Hilldale and Raff, 2008;Wang and Philpot, 2007;Irish and Lillycrop, 1999;Saylam et al., 2017) or sunken archeological sites (Doneus et al., 2013), photogrammetric measurements of river beds (Westaway et al., 2003) or convection flow estimation in a glass vessel (Maas, 2015).Narasimhan and Nayar (2005) investigated the application of light stripe projection and photogrammetric stereo on multimedia scanning.They employ a camera-projector setup in air that measures objects in a water tank.To correct for the refraction at the transition between air and water the refraction of the camera ray as well as the projector ray need to be considered.For calibration two planes at known distance are placed in the water tank.Moreover, they studied the effects of scattering media by adding milk to the water and developed a method to correct the color of the scans.Klopfer et al. (2017) apply a Microsoft Kinect RGB-D camera to capture bathymetry with the sensor placed above the water.
Although the infrared pattern projector of the Kinect suffers from high absorption in water, measurements at depths of up to 40cm were achieved.The sensor was mounted parallel to the water surface and the distance between the water plane and the sensor was measured.The point clouds captured by the Kinect were then corrected based on a ray-based refraction model.
The work of Palomer et al. (2017) does not concern itself with scanning in the water with a sensor in air, but the refraction correction necessary for their underwater laser scanner is similar.Since a mirror galvanometer-based laser projector is employed to project different laser lines in the underwater environment, the incident angles of the laser rays differ from each other.Instead of using a ray-based correction model Palomer et al. (2017) model the deformed laser plane with a pyramid cone.To reconstruct the laser points, the camera rays are intersected with the calibrated model of the deformed planes.
Some of the published literature also addresses inaccuracies introduced by waves, which are very difficult to model exactly.In early work Okamoto (1982) studied the influence of waves on aerial imaging of shallow waters.Fryer and Kniest (1985) examined the resulting errors for the application of stereo vision for creating digital elevation maps of shallow waters from an aerial platform.To achieve structured light imaging through a wavy water interface Sarafraz and Haus (2016) use a calibration target for online estimation of the water surface.In their setup a calibrated projector-camera setup is employed with the projector in air and the camera placed in the water.The scene is illuminated with a dot pattern.Underwater the camera observes the shift of the projected pattern on a checkerboard target.From this observation a model of the water surface is estimated.Objects placed on the same plane as the calibration target can then be reconstructed by computing the refraction for all observed projector rays.The restriction of this method is that it can be applied only to a controlled environment because the water surface is assumed to be modeled by a combination of sinusoidal functions.

METHODOLOGY
In this work we employ a triangulation based line laser scanner.
To create a precise point cloud of a semi-submerged scene we first scan the scene and create a 3D reconstruction without considering the refraction at the water interface.Since the employed structured light scanner captures only a profile, we need to move the scanner to create a complete 3D scan of the scene.We capture and record the 6-DOF movement of the scanner system externally.Then, from the sensor data we extract the water surface and correct all 3D point measurements inside the water body using a ray-based approach.In this section we describe the employed scanner hardware, how we calibrate the scanner, the 3D reconstruction approach and how the refraction at the water interface is modeled and corrected.

Underwater Laser Scanner
The  The bottom image in Fig. 3 shows the components of the laser housing.The laser projector is constructed from Powell laser line optics (labeled (a)), beam correction prisms (labeled (b)) and the laser diodes (labeled (c)) The lasers are two 1 W green diode lasers with a wavelength of 525 nm, which are controlled by power and control electronics (labeled (d)) and connected to an embedded PC (labeled (e)).A second IMU (labeled (f)) is installed for orientation determination.The two laser lines project a laser cross consisting of two perpendicular lines in the scene.
The lasers can be fired synchronized to the camera shutter using trigger signals.

Calibration
We approximate the camera projection function based on the standard pinhole model with distortion.The point X = (X, Y, Z) T in world coordinates is projected on the image plane according to where x = (x, y) T are the image coordinates of the projection, p = (px, py) T is the principal point and fx, fy are the respective focal lengths.We use separate focal lengths instead of a single principal distance in order to absorb small modeling inaccuracies.However, in our particular case the difference between the estimates for fx and fy is small.
Using the normalized pinhole projection we include radial and tangential distortion defined as follows where (k1, k2, k5) are the radial and (k3, k4) are the tangential distortion parameters.Here, x = (x, ỹ) are the real (distorted) normalized point coordinates and We calibrate the camera using Zhang's method (Zhang, 2000) with a 3D calibration fixture with AprilTags (Olson, 2011) as fiducial markers.The advantage here is that calibration points are extracted automatically even if only part of the structure is visible in the image.After performing laser line extraction we undistort all image coordinates of the detected line points.Therefore, we do not have to consider the distortion parameters during the 3D reconstruction step, which simplifies the equations presented in the following sections.
For calibrating the laser projector it is necessary to determine the laser plane parameters relative to the origin of the camera coordinate system.To do this, a L-shaped 3D calibration pattern, depicted in Fig. 4, consisting of two planes with AprilTags is used.This enables determining the laser plane equation from a single image.
First, we detect the L-shaped pattern in the camera image and compute the pose relative to the calibrated camera.From this we can compute the parameters of the two individual planes of the calibration target.Then, we detect laser points lying on the two calibration planes and reconstruct their 3D position by intersecting the camera ray with the respective calibration plane.The plane parameters of the laser plane can then be found by fitting a plane to the the reconstructed 3D laser point positions.We compute the best fitting plane based on a robust fit using RANSAC over multiple calibration images to improve the final solution.

3D Reconstruction
Once we have calibrated the system and know the camera model and laser plane parameters 3D reconstruction is performed using light section by intersecting the camera rays with the laser plane.We describe the laser plane πi using the general form where (ai, bi, ci) are the plane parameters and X = (X, Y, Z) T is a point in world coordinates.Using the perspective camera model described in Eq. 1 this is expressed as where x = (x, y) T are the image coordinates of the projection of X on the image plane, p = (px, py) T is the principal point and fx, fy are the respective focal lengths.
The coordinates of a 3D object point X = (X, Y, Z) T on the plane from its projection on the image plane x = (x, y) T can be computed by intersecting the camera ray with the laser plane: (6)

Estimation of the Water Surface
In this work we assume a flat water surface, which can be automatically detected and estimated using only the detected laser points.As depicted in the top two images in Fig. 5 we can see a reflection of the laser line on the water surface.If we set a low threshold for the laser line extraction of the structured light scanner, these reflections can be extracted as laser line points and reconstructed.The resulting point cloud is shown in the bottom left image of Fig. 5.It can be seen that the water plane is visible in the 3D point cloud.We can estimate the dominant plane of these measurements using a robust fit, thus estimating the plane The reflections are later purged to create the final 3D scan of the scene.We verify the estimated water plane parameters in the experiments by comparing them with external measurements.

Refractive Correction
To create accurate scans of the semi-submerged structures the refraction at the air-water interface needs to be considered.After performing 3D reconstruction based on light section as described in Sec.3.3 and applying the 6-DOF scanner trajectory, we split the resulting point cloud into above-the-water and underwater points based on the estimated water plane.Only the points that lie below the water plane are then corrected based on a ray-based approach similar to the work of (Klopfer et al., 2017).Since in our case the laser ray and camera ray do not follow the same optical path, we need to account for the refraction of the camera rays as well as the laser rays at the water surface.
The principles of the ray-based correction approach are depicted in Fig. 6.Using the 6-DOF trajectory of the scanner and the water surface estimated as described in Sec.3.4 we establish for each individual line scan of the structured light scanner the position of the camera center K, the position of the laser projection center L and the water plane in a common coordinate system.For each point p of the structured light scan located below the water surface we apply a ray-based approach to find the corrected point position P .
First, we compute the intersection points of the camera ray and laser ray with the water plane.SK is the intersection point of the line from the camera center K to the point p, which is visualized in red in Fig. 6.SL is the intersection point of the line from the laser projection center L to the point p, which is visualized in green in Fig. 6.
At these intersection points SK and SL we then need to account for the refraction effects.The incident angle of a ray r is where nW is the normal of the water plane.The refraction angle δ2 is then computed via Snell's law Figure 6.Principles of the ray-based correction approach.The point p is the initial reconstructed point using light section without accounting for refraction and the point P is the point after refractive correction.K is the camera and L the laser projection center.K ′ and L ′ are the projections of the camera and laser center along the water plane normal on the water plane.SK and SL are the respective intersections between the camera and laser ray with the water plane.
where n1 and n2 are the refractive indices of air and water.
In oder to compute the refracted ray r ′ we need to rotate the incident ray by the angle difference △ δ = δ2 − δ1.Since Snell's law in scalar form assumes that the incident ray and refracted ray are in the same plane we need transform the incident ray to apply the angle difference △ δ to a rotation.To do this we form a new orthonormal basis B, where the z-axis points in the direction of the water plane normal, the y-axis points in the direction of the projection of the camera center K ′ respectively laser center L ′ on the water plane.The refracted ray r ′ can then be computed by a rotation of the incident ray r around the x-axis of the new Basis B: where R △ δ is a rotation matrix describing a rotation of △ δ = δ2 − δ1 around the x-axis and T A B is a transformation matrix, which describes the base change from the original basis A to the constructed basis B.
We need to compute this for the camera ray as well as the laser ray to yield the refracted camera and laser rays.In principle the corrected point P results from intersecting the refracted camera and laser ray.However, the experiments showed that this does not yield a robust solution since errors of the estimated refracted rays result in large errors of the computed corrected point P .In practice some parameters, such as the position of the center of projection of the laser plane, are difficult to calibrate precisely, which results in errors of the incident laser ray.Therefore, instead of computing the intersection of the two refracted rays we compute the intersection between the refracted camera ray and and a plane constructed from the refracted laser ray and the intersection line between the laser and water plane.This constrains the solution to lie inside the constructed plane, which limits the effect of inaccurately estimated incident rays.

EXPERIMENTS
To validate the proposed approach we perform experiments with the scanner mounted to an industrial manipulator.This allows us to move the scanner with repeatable different angle of incidence for the laser plane with respect to the water surface.
The setup is depicted in Fig. 7. Three different scenes are placed in a half-filled water tank with an edge length of 1 m.The first scanned object is a wooden Euro-pallet, as shown in Fig. 1.The second scene, depicted in Fig. 7, is more complex and contains a dwarf figure, a coffee pot, a plastic pipe and two chessboards.The last scene only contains a large chessboard.The objects are placed in front of the scanner at a distance in the order of 1 m.

Experimental Setup
To ensure that we can move the scanner with a repeatable trajectory, we mount it to a KUKA KR-16 industrial manipulator.We move the scanner along a linear track with the laser projector pointing straight down, such that the laser plane is orthogonal to the water plane.We record multiple scans with the same motion and gradually rotate the scanner in 5 • steps up to an angle of 20 • .This way, we can observe the influence of different angles of incidence of the laser projector rays.We do not consider larger angles than 20 • for experimental reasons and constraints on the possible scanner movements without collision with the water tank.
To record the trajectory of the scanner we use an an external Op-tiTrack V120:Trio 6-DOF tracking system.Two markers, which are highlighted with red arrows in the left image of Fig. 9, were mounted rigidly to the scanner to ensure that always at least one marker is visible in the field of view of the optical tracker.
For calibrating the transformation between the scanner coordinate system and the coordinate system of the tracker, we use a AprilTag calibration target and a handheld probe with a tracking marker, as can be seen in the middle image in Fig. 9.With the calibrated camera of the structured light scanner we can determine the positions of the corners of the AprilTags in the scanner coordinate system.By touching the corners manually with the handheld probe we can also measure the same points in the tracker coordinate system.Based on many point correspondences we can find a least squares solution of the 6-DOF transformation between the tracking coordinate system and the scanner coordinate system.Since we can measure the 6-DOF poses of the tracking marker at the same time, we can compute the relative transformations between the maker and the scanner coordinate system.Using this information we can compute the transformation between tracker and scanner coordinate system based on the 6-DOF pose measurements of one of the two tracking markers attached to the scanner.Based on the scanner trajectory, measured by the tracking system, we can then combine all single line scans to an complete 3D-scan of the entire scene.
Additionally, the handheld probe was also used to measure the water plane in the tracker coordinate system, as depicted in Fig. 9.While this is not necessary for applying the proposed method, this was done to verify the water plane parameter estimates.

Verification of the Water Plane Detection
As described in Sec.3.4 it is possible to detect the water plane in the 3D scan.To validate the estimated plane parameters, we compare them to a plane fitted to the water plane points measured using the handheld probe with the tracking system.The angular difference between the normals of the planes determined from the scan and from the tracking system is δ = 0.21 • .Additionally, we transform the plane equations to the Hesse normal form and calculate the distance d between the planes and the origin of the coordinate system.The computed deviation of the distance is Based on the small angle difference of 0.21 • and the distance deviation of 1 cm, we consider the estimation of the water plane from the scanner data a suitable approach.The remaining error is expected to be mainly a result of accumulated calibration errors.

3D-Scan Results
The resulting point clouds of three different scenes are visualized in Fig.  scan.In this visualization the points above the water are colored in red and the points below the water surface are colored in blue.With increasing scanner angle the underwater points become more compressed in the vertical direction and the bend visible in the wooden plank becomes stronger.At an scanner angle of 20 • there is a strong deviation between the surface of the plank in the underwater and the above-the-water scan.However, after correction these errors are not visible anymore, and the points of the plank surface form a straight line.This is true for all scans independent of the scanner angle.The image in Fig. 10 bottom right shows a cross-section of the combined point cloud of all corrected scans.
To investigate this in more detail we perform measurements using the chessboard in the scene shown in the bottom images of Fig. 8.In this scene, the chessboard is positioned half below and half above the water table.Since we can assume that the chessboard is nearly perfectly planar, we compare the plane normals fitted to the underwater and above-the-water part.
For each angle of incidence we consider the submerged part and the part above water as separate planes.Subsequently, we calculate the angle between the normals of these two planes.For the uncorrected point clouds the angular error results for the different scanner angles are listed in Tab. 1.We also report the root mean squared error (RMSE) of the plane fit for the underwater and above-the-water points.
These measurements confirm the observed effect of the compression.While the angular error for the vertically aligned scanner is 5.56 • , it rises up to 14.53 • for a scanner rotation angle of 20 • .The steeper the angle of incidence, the faster the angular error increases.
After the correction of each scan, we computed the angular error again, to verify our method.The results are reported in Tab. 2.
For the vertical scanner alignment we were able to reduce the error from 5.56 • to 0.19 • .The remaining error for a scanner alignment of 5 The fact that the angular error is significantly reduced after applying the refractive correction supports the visual observation that the large errors introduced due to refraction are almost completely removed.
The remaining errors are expected to be largely caused by calibration inaccuracies.For example, the position of the laser projection center was not calibrated precisely for the experiments.
It was only estimated based on the calibrated laser plane and the rough position of the laser line optics.

CONCLUSIONS
In this paper we presented an approach for 3D scanning semisubmerged objects using a triangulation based structured light scanner.We derived a geometric model of the refraction for the employed laser line scanner and showed how the acquired point clouds can be corrected.In lab experiments using an industrial manipulator we demonstrated that the proposed method is effective to correct for refraction effects at the air-water transition, and that a consistent 3D scan of the semi-submerged structure can be acquired.Moreover, we showed that this is possible without any external markers or targets since the water surface can be directly estimated from the structured light sensor data.Further work is necessary to investigate the accuracy of the corrected scans, using more detailed reference objects.In this work we assume the water surface to be completely flat.This restricts the applicability of the proposed refractive correction method to calm waters since waves would introduce additional errors in the point cloud data.In future work we plan to apply the method to further scans captured in the field, investigate how water turbidity affects the accuracy of the proposed reconstruction method and how it can be extended for moving water.

Figure 1 .
Figure 1.Prototype underwater line laser scanner mounted on an industrial manipulator for scanning a semi-submerged scene.

Figure 2 .
Figure 2. Prototype structured light underwater scanner projecting a laser cross in a water tank.
Fig.3shows a cross-section of the camera housing and the laser housing.The camera housing, which is depicted in the top image in Fig.3, includes the lens with a focal length of 12.5 mm (labeled (a)).The camera is a FLIR Blackfly 2.3 Megapixel color

Figure 4 .
Figure 4. Calibration of the laser plane parameters using a L-shaped calibration target.camera (labeled (b)) with a 1/1.2"Sony Pregius IMX249 CMOS sensor.The image resolution is 1920 × 1200 pixels with 5.86 µm pixel size and a maximum framerate of 41 fps.An IMU (labeled (c)) is centered above the camera.For image processing an embedded PC (labeled (d)) is included in the housing.

Figure 5 .
Figure 5. Reflections of the laser line projector on the water surface can be exploited to estimate the water plane.parameters of the water surface.The bottom right image of Fig. 5 visualizes the fitted water plane and plane normal in white color.The reflections are later purged to create the final 3D scan of the scene.We verify the estimated water plane parameters in the experiments by comparing them with external measurements.

Figure 7 .
Figure 7. Top: Scanning a semi-submerged scene with the scanner mounted to an industrial manipulator.Bottom: Detail view of the captured scene with a laser line projected by the scanner.
Figure 8. Point clouds of the initial uncorrected reconstruction on the left side and the corrected result on the right side.The point clouds are colored by height.The error as an result of the refraction at the water surface is clearly visible in the uncorrected scans.For example, note that in the middle left scan the point measurements of the floor are not consistent.In the corrected scan, shown in the middle right image, the bottom of the tank is a consistent surface.

Figure 9 .
Figure 9. Left: Two markers attached rigidly to the structured light scanner for trajectory tracking.Middle: Measurement of reference points for the co-calibration of the tracker and scanner coordinate system.Right: Measurement of the water surface for the verification of the estimated laser plane parameters.

Figure 10 .
Figure 10.Cross-sections through 3D scans of a wooden pallet.Points below the water are colored in blue and points above the water in red.Starting from the top left with the scanner vertically positioned (0 • ), scans with different rotation angles of the structured light scanner are depicted.The last figure, bottom right, shows the cross-section of the reconstruction result after applying the refractive correction method.In the corrected scan the surface of the wooden pallet lines up well.
• , 10 • and 15 • is similar, ranging from around 0.34 • up to 0.37 • .The largest remaining error of 0.53 • is observed for a scanner rotation of 20 • .

Table 1 .
Angular errors between the plane normals above and below water for the initial uncorrected reconstruction result.Root mean square error (RMSE) of the plane fit.

Table 2 .
Angular errors between the plane normals above and below water for the corrected reconstruction result.Root mean square error (RMSE) of the plane fit.