TARGETLESS PHOTOGRAMMETRY NETWORK SIMULATION FOR INSPECTION PLANNING IN OIL AND GAS INDUSTRY

The oil and gas offshore industry demands regular inspections of components and structures that are subjected to extreme operational and environmental conditions. In this context, risers are pipelines that transport mainly oil, gas, water, and cables between submarine structures and the surface offshore platform, in the portion not touching the ocean floor. The emerged part of these risers is typically inspected by industrial climbing, which is a very time-consuming activity, has high operational costs, is dangerous and has a strong dependence on inspector skills. Remotely Piloted Aircraft Systems (RPAS) have been recently used for visual inspection of risers, however, no quantitative or geometrical evaluation has been conducted using this kind of image acquisition yet. An image-based measurement technique, such as close-range photogrammetry, can provide a 3D reconstruction using images, but a series of requisites is mandatory to achieve good results as image acquisition sequence, overlap, camera positioning network, spatial resolution and object texture in non-prepared and targetless scenes. The analysis of different image acquisition strategies using a real RPAS is too difficult because it demands a lot of time, good weather, daylight, and a scene similar to where risers are installed. An alternative is to use simulation. In this paper a ROS/Gazebo simulation is described and used to create a realistic textured 3D virtual environment of the platform, risers and RPA, providing a fast and low-cost solution to simulate different RPA trajectories for photogrammetry image acquisition in targetless scenes. These trajectories are evaluated by comparing the measured risers through photogrammetry to its CAD/simulated model. Since the scene is not prepared, the RPA position/orientation or a stereo vision setup can be used to set scale to the measurement result. The best trajectory found during simulations was also evaluated in a real experiment.


INTRODUCTION
In recent years, the use of Remotely Piloted Aircraft Systems (RPAS) has been growing quickly across many areas, such as military, security, civil engineering, archaeological, agronomy, forestry, geomatics, and telecommunications (Shakhatreh et al., 2019). Some characteristic applications in these areas are the aerial mapping, rescue operations, geophysics explorations, traffic monitoring, and finally, industrial inspection (Kridsada et al., 2016) (Jordan et al., 2018).
Typically, industrial inspections are focused on the electric installation, transmission/distribution lines, bridges, buildings, wind turbines (Jordan et al., 2018) and in-service flare inspection (Marinho et al., 2012).
Since 2014, the International Association of Oil and Gas Producers (OGP) is interested in the use of RPAS in three application fields: Heal Safety Environment (HSE), security (monitoring of facilities) and asset integrity (Mercuri et al., 2017). In the last application field are the pipelines connecting oil and gas offshore platforms to subsea structures. They perform different functions, as oil, gas, water, and chemistry products transportation and cables ducting (Marinho et al., 2006) (Wang et al., 2017). The portion of these pipelines that do not touch the ocean floor are called riser. A special type of riser is the flexible riser, also known as flexible pipeline, which are complex structures composed of multiple layers of metals and polymers. * Corresponding author -tiago.pinto@ufsc.br Industrial climbers are, commonly, the responsible for the inspection of the emerged portion of risers ( Figure 1). These climbers perform manual measurements of riser geometry, take photos and signalize possible problems/defects. This activity is expensive, dangerous, high time-consuming with approximately only one riser being inspected per day, besides being very laborious and strongly operator dependent (Marinho et al., 2006) (Mercuri et al., 2017). Typical defects in risers include diameter increase or reduction, ovalizations and ripples, which may be related to internal structure rupture and deformation. External damages can also occur as abrasion during the own riser installation and even by contact of boats illegally moored to the platform for fishing. These defects can vary in size, typical ripples can be detected along meters while the geometry change may be up to 10 mm wave-ripple amplitude. Typical abrasions may have a 10 mm to 500 mm millimeters size and submillimeter to 10 mm depth.
In that context, RPAS have been recently used to perform visual inspection, however, the image acquisition procedures were not planned for photogrammetric 3D reconstruction and dimensional inspection. If certain procedures are followed, images of the risers taken by RPAs can be used to perform its geometrical evaluation using photogrammetry.
In order to obtain a good measurement result using photogrammetry a set of requirements must be fulfilled, as image acquisition sequence, image overlap, camera positioning network, spatial resolution and object texture (Luhman et al., 2011). A good acquisition procedure with RPAs can be obtained by a study of trajectories and camera parameters. However, to perform this task using a real RPA is too time-consuming and expensive, requires the use of different hardware combinations such as different camera resolution and focal lengths, battery flight time, weather and requires a location at least similar to riser and platform configuration (Galkin et al., 2019). To overcome those problems, a simulation that comprises the RPA, camera, lenses, and components in the inspection scenario, like risers, offshore platform and environment/weather is desirable.
Simulations allows to test and tune the system and to perform different situations, e.g. hardware failures, trajectories, embedded sensor parameterization or different environment configurations (Mendonça et al., 2013). Moreover, simulation tools enable the creation of 3D scenarios with advanced rendering capabilities, flexibility and seamless integration with the robot control system. This paper 1 presents the design and implementation of a virtual environment based on ROS (Robot Operating System) (Quigley et al., 2009) and Gazebo (Koenig and Howard, 2004), which enables to quickly evaluate different photogrammetric acquisition strategies, hardware configuration, as camera resolution and focal length, and compare the results using geometric evaluation of the resulting 3D mesh. RPA flight with a planned route and image acquisition is performed in ROS/Gazebo virtual environment. These images are processed in a photogrammetric software, which results in a 3D textured mesh. Finally, a geometric evaluation is performed, comparing the 3D result to its ground truth (GT), which is the CAD model inserted in the simulation, allowing to evaluate the influence of the simulation parameters. The comparison can be displayed as a colored deviation map, that indicates 3D points to CAD errors. The process is presented as a flow chart in Figure 2. A real experiment was conducted to evaluate the best trajectory obtained during simulations. In the simulation environment, it was also simulated a stereo vision system.

VIRTUAL ENVIRONMENT SETUP
Simulation allows a prior verification of the operation of the systems, procedures, and parameters to be used in real environment applications. Currently, one of the most widely used RPA simulation environments is the integration between ROS and Gazebo.
ROS is an Open Source framework, which provides standard services such as hardware abstraction, low-level device control, message-passing between process through publisher-subscriber mechanism and commonly used functionalities. The same code used during simulation can be used in the real system, since Gazebo simulates all the environment and real hardware system with its inputs and outputs (Mendonça et al., 2013) (Quigley et al., 2009).
Gazebo is a free and robust physics engine for realistic robot simulations that provides a simulation of CAD (computer-aided design) models; sensors, like cameras; robots and rotors; highquality textures; different environmental conditions, like wind, water movement, and gravity. Three different acquisition trajectories were tested through Gazebo plugins to simulate the RPA's movement in the environment. The scene and risers creation are described in the fallowing.

Scene geometry
In this work, a scenario that reproduces a real offshore environment is presented. The initial simulated scenario consisted of an FPSO (Floating Production Storage and Offloading), riser balcony, sea surface and illumination. It is also possible to add other factors as image noise, mist and wind variations.
Due to the great complexity of the scene and high computational cost, it was modified to a simplified version, which allows faster experiments. The new scenario was reduced to a riser balcony containing six risers. Real risers and offshore platform photos were used to create the textures. The simulated scene is shown in Figure 3.

Images acquisition
Photogrammetric process

Simulation environment
RotorS Figure 3. Simulated scenario with six risers and high-quality textures from actual in-field photos.

Risers geometry
Risers were modeled using CAD, in which some artificial defects/artifacts were added. Therefore, during the measurement to CAD comparison it is possible to verify how well they were detected.
Once the CAD is ready, it is exported to a STL file and loaded in Blender to apply a texture (Dere et al., 2010) based on real risers photos. The same exported file is used as reference, or ground truth, to perform a deviation comparison of the measurement result.

RPA AND SUBSYSTEMS MODELING
The representation of the RPA and its subsystems can be made through models that describe its behavior and physical structure. The RPA payload enables the implementation of subsystems adding new functionality and generating data that assist in the inspection of risers. A gimbal control, camera and stereo vision system were modeled to integrate into the RPAS model. The communication between RPA and subsystems are made by Mavlink protocol and commands to control and define a set of trajectory (DJI, 2019a).

RPA
The RPA was modeled considering the physical characteristics of the M210, which is widely used in industrial applications, including offshore inspections. The RotorS Micro Aerial Vehicle (MAV) simulator Framework (Furrer et al., 2016) was used to model the RPA. RotorS is a modular Gazebo-based MAV simulator, which includes a position controller, state estimator, IMU (inertial measurement unit), generic odometry sensor and the VI-sensor (Furrer et al., 2016).
RotorS provides several MAV multirotor models. In this paper, the RPA is based on IRIS+ | 3DR, which had its mass, dimensions, rotors positions and control parameters modified to match the M210 created model. The final model in the simulation environment is shown in Figure 4. RPA's kinematic was not considered, what should be done in future work.

Gimbal
A 3-axis generic gimbal was modeled based on the specifications of DJI Zenmuse X5S (DJI, 2019b) camera gimbal. The 3D model of the gimbal generated using URDF (Universal Robotic Description Format) is mounted on the bottom of the RPA. The real and modeled gimbals are shown in Figure 5.

Cameras
A DJI X5S equipped with a 45 mm focal length lens, which is a DJI "plug and play" camera, were simulated for single-camera photogrammetry acquisitions. For the stereo setup, two FLIR Blackfly equipped with a 50 mm focal length were used with a 1 m baseline. The camera's main parameters are described in Cameras were configured in Gazebo through an SDF (Simulation Description Format) file, in XML format. In this file, it is possible to set parameters like angle of view, image resolution, noise, lens distortion and image format. For the described experiments, the image noise and lens distortion were set to none and the image/file format were set to R8G8B8/PNG.

IMAGE ACQUISITION PROCEDURES
In order to obtain good results using photogrammetry, some important factors mentioned in the literature must be followed. The image acquisition network, i.e. the position and orientation of the camera relative to the scene for each acquisition, must be carefully planned. For better points correspondence between images, about 80% image overlap and sequence acquisitions are recommended (Luhman et al., 2011) (Agisoft, 2020) (Atkinson, 1996) (Marcellino et al., 2019).
A good network design provides a better geometry intersection between lines from image points to correspondent object points. Consequently, there is a smaller uncertainty region around reconstructed points, as seen in Figure 6. The trajectory performed by the RPA, as well as the use of a gimbal for camera movement, exerts directly influence in geometry intersection, i.e. image acquisition network. To evaluate the RPA trajectory and gimbal movement that leads to better photogrammetry results, simulations in the ROS and Gazebo environment were performed. In each image acquisition, a command using the Mavlink protocol gets the RPA position. This position may be used to give scale to the measurement result and to reduce the photogrammetric processing time. A commercial photogrammetry software 2 was used to process the images.
In this work three different image acquisition networks were evaluated: "A" Gimbal pitch with static RPA; "B" Vertical RPA trajectory with static gimbal; and "C" Serpentine RPA trajectory with vertical and horizontal RPA movement and yaw gimbal rotation to keep riser inside the camera's FoV. Trajectories "A" and "B" are commonly used for visual inspection companies since they are easier to perform. Trajectory "C" is a proposed example, expected to be ideal for photogrammetric acquisition. The RPA and gimbal displacement and rotations that lead to an overlap of 80% were evaluated before acquisition. The RPA to riser distance was set to a minimum of 5 m, this complies with the value generally used for security reasons for in-field acquisitions. During simulation, the RPA speed was limited to 3 m/s due to computer limitations. Although, this does not affect the images acquisition and simulation results. A text file containing image name, camera position and orientation for each acquisition is generated by a developed ROS plugin, and the values are used during the photogrammetric processing. The evaluation of different trajectories in the simulation environment seeks to obtain optimized procedures, resulting in better measurement results and reduced flight time. Each trajectory is described below for a single camera setup (X5S camera). The last trajectory type was also performed for the stereo setup (two Blackfly cameras).

Gimbal pitch with static RPA
This procedure consists of image acquisitions with gimbal rotations in pitch axis while the RPA remains almost static. The rotation angle was evaluated in order to obtain 80% image overlap.

Vertical RPA trajectory
This acquisition trajectory is performed by moving the RPA only vertically, as indicated in Figure 7. A picture is acquired every time the RPA changes its height by a delta (Δℎ) of 20% of the FoV's height, allowing 80% overlap between acquired images. This trajectory is commonly used for visual inspection in the field due to its easy operation.

Serpentine RPA trajectory
The serpentine acquisition trajectory consists of a sequential combination of vertical (major) and horizontal displacements as illustrated in Figure 8. The RPA movement is defined by a set of waypoints in the horizontal plane. An image is acquired every time the RPA changes its height, as in the previous trajectory and also when transitioning for the next angular or lateral position. Yaw rotation is also applied in order to maintain the riser in the camera's FoV, similar to Figure 6. This is one example of the proposed acquisition procedure that is better suited for photogrammetric reconstruction, due to an improved image network.

Serpentine RPA trajectory -Stereo acquisition
The serpentine trajectory was also performed using a stereo setup. This setup has the main advantage of using the cameras' baseline as scene scale. This eliminates the dependency of using scales in the scene which are hard to use in field conditions. Also, the RPA position may be used only as initial point for the photogrammetric process and not used to set the scene scale, since this data may have considerable errors. The disadvantages are the extra weight and size and the need for a custom setup, since this kind of system is not typically available by the RPA manufacturers, so the mechanical structure and RPA integration, electronics, communication and software need to be developed.

SIMULATION RESULTS
The acquired images for each tested trajectory were processed using a photogrammetry software. The result is a textured 3D mesh of the measured riser and background scene.

Single camera -trajectory evaluation
The parameters of the three evaluated trajectories are listed in Table 2. An acquired image sample is shown in Figure 9 and the obtained 3D reconstructions are shown in Figure 10.  A B C Figure 10. Photogrammetric textured 3D dense cloud for the three different trajectories. For "A" the software was unable to reconstruct the riser.

Item \ Trajectory
In configuration "A" (gimbal pitch only), it was not possible to reconstruct a dense point cloud, so no geometric evaluation was performed. In configuration "B" (vertical only), a dense point cloud was obtained, but there is too much noise and deformations.
The main reason for the bad results of the two first acquisition procedures, A and B, is the poor acquisition network design, although the image overlap and spatial resolution were the same as the ones used for the serpentine trajectory. The lack of images in different positions resulted in a bad acquisition geometry, as shown in Figure 6.
For the Serpentine "C" trajectory, a detailed 3D reconstruction was obtained since all fundamental requisites were fulfilled.
The 3D mesh delivered by the photogrammetric software was evaluated by comparison with the original CAD model used in simulation and considered as ground truth (GT). The deviation maps of trajectories "B" and "C" are shown in Figure 11. With this kind of analysis, it is possible to evaluate the impact of variations in the acquisition network and camera/lens configuration in the quality of the reconstructed object. In a real inspection, the deviation map can be used to evaluate the pipe defects comparing the measurement result from a reference pipe obtained using an original CAD and/or a mathematical fitted cylinder. B C Figure 11. Deviation maps of the reconstructed riser to GT for trajectories "B" and "C".

Stereo vision (trajectory C)
For the stereo vision setup, only the trajectory "C" was performed. The simulation and the 3D result are shown in Figures  12 and 13, respectively.
Although the stereo cameras' lower resolution, if compared to X5S, simulation shows that good results can be achieved with the advantage of the cameras' distance (baseline) defining the scene scale, which can be obtained through separated system calibration. Another advantage is the acquisition of more images in shorter flight time. The calibration and fixed cameras' distance photogrammetric process will be performed in future work. Figure 12. RPA equipped with a stereo vision system performing trajectory "C" (serpentine) for photogrammetric measurement. Figure 13. 3D textured mesh from stereo setup simulation.

PRACTICAL RESULTS
Trajectory "C" was performed using M210 RTK v2 (DJI, 2019c) RPA equipped with a DJI X5S camera (DJI, 2019b) and a 45 mm lens. The acquisitions were performed in an open environment and about 5 m distance from the pipe, as shown in Figure 14. A PVC pipe was used as simulacrum of riser. The 3D reconstruction was successful and the obtained error standard deviation was of 0.58 mm. The full reconstructed scene and pipe analysis can be seen in Figures 15 and 16, respectively.

CONCLUSION
This paper presents the design and implementation of a virtual environment with realistic texture based on ROS/Gazebo for 3D optical inspection planning of risers using an RPA and photogrammetry. The simulation enables the evaluation of different images acquisition strategies and hardware configuration, as camera resolution and focal length, in nonprepared scenes using only scene texture, i.e. targetless and without scale bars. The simulated acquisitions are processed in a photogrammetric software, resulting in a 3D textured mesh. A geometrical analysis is performed to evaluate the influence of the simulation parameters such as RPA trajectory for image acquisition. An RPA was modeled based in IRIS+ | 3DR RPA provided in RotorS simulator. In addition, a functional gimbal was added for the DJI X5S simulated camera. The experimental results show that serpentine trajectory, a combination of vertical and horizontal displacements and yaw rotation to maintain the riser in the camera's FoV, was the best evaluated trajectory for images acquisition. A practical real word experiment using the best simulated trajectory was performed resulting in a successful 3D reconstruction. Future works include simulations for different acquisitions network and hardware configuration; further development of the stereo vision system simulations; in-depth comparison between simulations and real acquisitions; and external influences as wind, GNSS errors and image noise. An evolution of an inspection system like this may be used in order to detect micro torsions along the riser length, making possible to estimate if there is any broken metallic wire in the tensile armors. More studies must be carried in this aim.