EXTRACTION OF ROAD-CROSSING POWER AND COMMUNICATION LINES FROM MOBILE MAPPING DATA

In residential areas, maintenance of power and communication lines is an important issue. In recent years, the frequency and power of typhoons and storms have significantly increased. If utility poles incline due to strong winds, slack cables may contact with large vehicles. For automatically detecting loose road-crossing cables in wide areas, the MMS is very promising. However, when roadcrossing cables are measured using the MMS, large portions of points on cables may be lost, because the directions of laser beams are nearly parallel to the directions of road-crossing cables, and therefore, the laser beams cross road-crossing cables only a small number of times. In this paper, we propose a new method for reconstructing cables crossing roads. In our method, road-crossing cables are reconstructed using both point clouds and camera images. While point clouds of road-crossing cables may be partly missing, their camera images can be captured with little occlusion. Missing portions are recovered using lines extracted from camera images. First, points of each cable are extracted from a point cloud, and the 3D vertical plane on which the cable exists are calculated. From camera images, 2D line segments are extracted as candidates of cable lines. 2D line segments are projected onto the 3D vertical plane using the pinhole camera model. Finally, 3D cable lines are reconstructed on the 3D vertical plane from the merged points. In our experiments, road-crossing cables could be sufficiently extracted using our method.


INTRODUCTION
In residential areas, maintenance of power and communication lines is an important issue. Power lines and communication lines are connected to each house through utility poles or underground pipes. In Japan, utility poles are used in most areas, and the areas where cables are buried underground is very limited. The utility pole is usually shared by power lines and the communication lines, and the upper layer of cables is used for power lines and the upper layer is used for communication lines. In this paper, both power lines and communication lines are referred to simply as cables.
In recent years, abnormal weather has been often reported around the world. In Japan, the frequency and power of typhoons and storms have significantly increased to record the highest, and the inclination of utility poles due to strong winds has become a serious problem for infrastructure maintenance because it occurs simultaneously over a wide area. When the utility pole is inclined, the distances of loose cables from the ground decrease. For cables crossing the road, loose cables may contact with large vehicles. In order to avoid contact accidents, it is necessary to investigate whether the height of the cable meets the regulations.
Mobile laser scanning is useful for efficiently detecting loose cables in a wide area. So far, many methods have been studied for automatically detecting power lines from point clouds acquired by laser scanners. Many researches are based on point clouds captured using the airborne laser scanning (ALS) system. In their methods, cables were detected using the RANSAC method (Chen et al., 2012, Guo et al. 2016, the Hough transform method (Liu et al., 2009, Guan et al. 2016, or the machine learning method (Kim et al., 2012).
However, ALS is not suitable for precisely measuring the distances of cables from the ground, because most ALS systems have accuracies of only a few tens of cm at most. So far, ALSbased methods have been used for detecting thick and long cables stretched between transmission towers due to large point-to-point distances of point clouds. ALS has been a powerful approach for detecting such cables, because it can capture point clouds even in places with no road such as in the mountains.
However, in our survey, cables used for utility poles in urban areas had diameters of 1.3 cm for power lines and smaller diameters for communication lines. Therefore, point clouds obtained from a distance using ALS are too sparse to detect cables with small diameters in urban areas. In addition, since multiple cables are stretched in parallel between utility poles, lower cables cannot be measured from above using ALS.
To detect cables in urban areas, the vehicle-based mobile mapping system (MMS) is more suitable, because the MMS (a) Mobile Mapping System (b) Close-up Figure 1. Mobile Mapping System measures cables from the ground. The MMS can capture much more dense point-clouds than the ALS system, and it can also capture all parallel cables between utility poles from the ground.
As shown in Figure 1, the MMS is a vehicle on which a laser scanner, digital cameras, GPSs, and an IMU are mounted. It can capture images as well as point clouds while running.
Several methods have been proposed for extracting cables from MMS data. In existing methods, cables were reconstructed using Hough transform and Euclidean distance clustering (Guan et al., 2014), the voxel-based hierarchical method (Cheng et al., 2014), and the region growing (Zuang et al., 2016), and so on. The method for reconstructing missing portions on power lines is also proposed by (Yadav et al. 2017). However, the existing methods mainly aimed at detecting cables parallel to the road, and did not deal with cables crossing the road. The MMS can capture dense point clouds from cables parallel to the MMS trajectory. However, when the MMS measures roadcrossing cables, large potions on cables may be missing.
In Figure 2(a), there are cables parallel to the road and cables crossing the road. Figure 2(b) shows point clouds of cables. In this figure, while dense points could be obtained from cables parallel to the road, large portions are missing in cables crossing the road. Therefore, in conventional methods, only cables parallel to the road are detected, and most road-crossing cables are missing. In our knowledge, no method has been proposed for detecting road-crossing cables from point clouds captured using the MMS.
It is very important for maintenance work to measure roadcrossing cables, because interference between cables and vehicles has to be avoided. Since the minimum height of roadcrossing cables is specified by law, cable heights have to be periodically inspected. If the MMS can automatically detect road-crossing cables, maintenance work for cables can be performed more efficiently. Figure 3 shows the reason why many points on road-crossing cables are missing. The MMS runs while emitting laser beams, as shown in Figure 3(a). Since the laser beams cross the cable parallel to the road many times, as shown in Figure 3 (b), dense points can be obtained from the cable. On the other hand, the laser beams cross the road-crossing cable only a small number of times, as shown in Figure 3 (c), because the direction of the scanning pattern is almost parallel to the direction of the cable. Therefore, points on the road-crossing cable become sparse and a large portion of points may be missing. In such cases, point acquisition is unstable and missing portions cannot be predicted. In conventional methods, it is difficult to reconstruct cables if a cable crosses the road and a large portion of points are missing.
In this paper, we propose a new method for reconstructing cables crossing roads from MMS data. Missing points are recovered using camera images, and cables are reconstructed using points extracted both point clouds and camera images. As shown in Figure 1, the laser scanner and cameras are fixed on the vehicle, and their relative positions can be measured. In our method, the 3D projection plane is constructed using cable points extracted from a point cloud, and 2D cable lines extracted from images are projected on the 3D plane. 3D cable lines are reconstructed from points merged on the 3D plane.
The next section gives an overview of our method. Section 3 describes projection of point clouds onto images for relating points and pixels. Section 4 describes the extraction of cable points from point clouds, and Section 5 describes the extraction of cable lines from images. Section 6 describes the method for reconstructing 3D cable lines by combining points extracted from  Figure 3. Points scanned from cables point clouds and images. Next, we describe experimental results, and finally conclude this paper.

OVERVIEW
In our method, road-crossing cables are reconstructed using point clouds, camera images, and MMS trajectory acquired by the MMS. In the MMS data, point clouds of road-crossing cables may be partly missing. On the other hand, images of roadcrossing cables can be captured with little occlusion in most cases because they are captured from the front of the MMS.
Since images do not have 3D information explicitly, point clouds and images are considered to be complementary. In this paper, road-crossing cables are reconstructed by using both point clouds and images. Figure 4 shows the process of extracting and reconstructing road-crossing cables. Our method composed of three phases: point cloud processing, image processing, and cable reconstruction.
In the point cloud processing phase, points that cannot be cables are removed from point clouds. Then the remaining points are projected onto the horizontal plane. Since each cable is a downward convex curve, it can be detected as a straight line on the horizontal plane. Points projected on the straight line are referred to as 3D cable points. Then, the vertical plane is calculated using points projected on each straight line. In this paper, the vertical plane is referred to as the cable plane. The cable plane is used as the projection plane for 2D lines extracted from images.
In the image processing phase, images whose GPS times are close to the GPS time of 3D cable points are selected as corresponding images. Then line segments are extracted from each image. First, the region where cables may exist is selected using an object recognition technique. Line segments are extracted from the selected region using Hough transformation. Then, 2D cable lines are extracted from pixels extracted as line segments.
In the cable reconstruction phase, cable curves are reconstructed by integrating 3D cable points and 2D cable lines. First, 3D cable points are projected onto an image using the pinhole camera model. Then, the correspondences between the projected 3D points and the 2D cable lines are calculated on the image. When a 2D cable line and 3D points are corresponding, they exist on the same plane calculated as the cable plane. Therefore, the 2D cable line is projected onto the cable plane, and the 2D cable line and the 3D points are integrated on the cable plane. Finally, the 3D cable curve is reconstructed from the integrated points.

Mobile Mapping Data
In our method, point clouds, camera images, and MMS trajectory are used for reconstructing cables. Point clouds and images are captured using a laser scanner and digital cameras on the MMS. The MMS trajectory is the trajectory of the origin of the MMS, and it is represented as a sequence of 3D coordinates with GPS time. We assume that the GPS time is given to each point, each image, and each trajectory point. The GPS time is obtained from satellites and indicates when the data was captured.
The MMS used in this research was the Mitsubishi MMS-X (Mitsubishi Electric, 2018) , as shown in Figure 1. The laser scanner on the MMS is RIEGL VQ 250. The rotational frequency and the scan rate are 100Hz and 300,000 measurements, respectively.

Corresponding Images
The MMS captures points and images during driving. Figure 5 shows a point cloud and images, which contain the same scene. In this figure, T represents the GPS time. Since the point cloud and the images were acquired at the nearest time, they include the same objects. Candidates of corresponding images are selected by comparing the GPS time.
If points in a point cloud can be projected onto an image, objects in the point cloud are also included in the image. In such a case, the point cloud and the image are regarded to be corresponding.

Relationship between Points and Image Pixels
When the relative position between a laser scanner and a camera is given, 3D points captured using the laser scanner can be projected onto camera images. Figure 6 shows a pin-hole camera model for projecting points onto an image. In this model, the pixel position (#, #) can be calculated as the intersection between the normalized image plane and a straight line through ( , , ) and the optical center.
However, the pixel position on the actual image is deviated from the intersection point ( #, #) due to lens distortion and so on. Therefore, the pixel position (#, #) is corrected to ( , ) using the following equation.

EXTRACTION OF CABLES FROM POINT-CLOUDS
First, candidate points for road-crossing cables are extracted from point clouds using the following procedures.

Filtering Point Clouds
Before detecting cable lines, points that cannot be road-crossing cables are removed from point clouds. In our method, points near the road, planar objects, and cylindrical objects are detected and removed.
Points near the road can be easily extracted if the trajectory of the MMS is given because the height of the road is about 2m below the MMS. Since the minimum height of cables is determined by regulations, points are removed if the height from the road is smaller than a threshold. In this research, we specified the threshold as 3 m.
For detecting planar objects, a point cloud is projected onto the 2D grid using the method proposed by (Kohira et al, 2017). In this method, each point in a point cloud is projected on the pixel position (I, J), as shown in Figure 7. The phase number I is the sequential number of measurements in each rotation, and the rotation number J indicates how many times the laser beam has rotated since the start of measurement. Figure 8 shows an example of projection on the 2D grid. Planes are extracted on the 2D grid. By using adjacency relationships on the 2D grid, planes are detected using the region growing method. Since cables have very small areas on the 2D grid, planar regions with large areas are removed.
Cylindrical objects are detected and removed using the method proposed by (Fukano et al., 2015). By removing cylindrical objects, utility poles are removed from a point cloud. Figure 9 shows a process for detecting cylindrical objects. In this method, a wireframe model is generated by connecting adjacent points on the 2D grid, and the wireframe is sliced using horizontal planes. Then, circles are fitted to the section points, and cylindrical objects are extracted as vertically aligned circles. Figure 10 shows the remaining points after removing lower points, planar objects, and cylindrical objects. In this example, poles that support cables were removed as cylindrical objects.

Extraction of Cable Candidates
The cable shape is a downward convex curve in 3D space due to gravity. Therefore, when points on a cable are projected on a horizontal plane, it becomes a straight line. To detect cables, the remaining points in Figure 10 are projected onto a horizontal plane, and straight lines are detected from the projected 2D points. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume V-2-2020, 2020 XXIV ISPRS Congress (2020 edition) In our method, straight lines are detected using the RANSAC method by randomly selecting two points.
In this research, cables crossing the road are detected for avoiding interference with vehicles. Therefore, straight lines with small angles to the MMS trajectory are excluded as cables parallel to the road. In our experiments, we specified the angle as 45 degree. Figure 11 shows straight lines detected from the projected points.
Then a plane is generated using points on each straight line, as shown in Figure 12. This plane is referred to as the cable plane, because all points on the cable are constrained on this plane.

Extraction of Line Segments
In our method, missing points on road-crossing cables are compensated using 2D lines extracted from camera images. Figure 13 shows the process for extracting lines from each image. Each procedure in Figure 13 is explained as follows.
The detection of linear objects from images is one of fundamental problems in image processing. Since a variety of methods have been proposed so far, various solutions are possible for detecting cables from images. In this paper, we combine several imageprocessing operations to stably detect cables. In our experimental data, whole road-crossing cables were present only in images captured from a distance, because wide viewing angles were required for road-crossing cables. Since high-resolution cable images could not be obtained, we implemented several methods and selected the best one, which could most robustly detect cables from low resolution images. Our solution was selected by experiment, but we note that simpler methods might be possible for detecting cables if higher resolution images could be obtained.
In our method, first, the image is binarized using adaptive thresholding, in which a threshold for each pixel is determined using pixels in a small region. In our method, the threshold is calculated as the average of 15 × 15 pixels minus 5 (Jain, 1986). Figure 13 (b) shows the binary image of Figure 13 (a).
Since the binary image is noisy, morphological transformation is applied to remove noise. This operation alternately erodes and dilates white areas in the binary image. We used a 3 x 3 kernel for morphological transformation (Serra, 1983). Figure 13 (c) shows the transformed image.
Then, straight lines are extracted from the image using the Hough transform. Each cable is not entirely a straight line, but it can be locally approximated as a straight-line segment, because the curvature of each cable curve is small. Therefore, in our method, straight-line segments are extracted from each image, and they are connected to construct a smooth curved cable. For extracting straight-line segments, we use the probabilistic Hough transform proposed by (Kiryati et al., 1991) to improve efficiency. In the Hough transform, it is necessary to specify the minimum line length for rejecting short lines. For detecting cable lines, we specified the threshold as 30. Figure 13 (d) shows the output of the probabilistic Hough transform.
Next, vertical lines are removed from extracted line segments because the image of a road-crossing cable are nearly parallel to the horizontal plane. In our implementation, the line segment is removed as not a road-crossing cable if the angle from the vertical line is less than 30 degrees. Figure 13 (e) shows the image from which vertical lines are removed.
Finally, pixels extracted as straight lines are thinned using the method proposed by (Zhang et al., 1984). Figure 13 (f) shows thinned lines.

Selection of Cable Regions
Straight lines can be extracted from objects other than cables. Therefore, we restrict regions for line detection to reduce In typical camera positions, cables exist only in the upper part of each image. Therefore, the lower regions in each image can be excluded from cable regions. Suppose H is the height of each image. In our method, regions are excluded if their positions are less than kH, where k is a constant value in the range 0 to 1. Since the value k depends on the camera setting, it is specified by the user according to captured images. In this research, we specified k as 0.5.
In the second step, regions are further eliminated using a convolutional neural network (CNN). Since local operators for line detection are sensitive to noise, they produce a lot of false positives. In this study, a CNN is used for selecting regions where cables exist. CNNs are useful to reduce false positives using features in multiple layers.
We use VGG16 (Simonyan et al., 2014) trained using ImageNet (Krizhevsky et al., 2012) as a CNN classifier. VGG16 is finetuned using the two classes "Cable" and "Other". For fine-tuning VGG16, we selected 260 images in which road-crossing cables exist. We divided the top half of each image into 50 rectangle regions as shown in Figure 14 (a), and manually selected regions that contain cables. Each rectangle image was input to VGG16 with the label "Cable" or "Other". The number of training images were 13,000. Figure 14 (b) shows regions selected as "Cable". In this figure, regions excluded as "Other" are shown in gray, and lines extracted only from the regions selected using VGG16 are shown in magenta.

Detection of Cable Lines
In line extraction, pixels of thinned lines are obtained, as shown in Figure 14 (b). Since the pixels contain multiple cables, pixels are divided into cable lines. In our method, pixels on lines are grouped if they are adjacent in the eight neighbours. Since some cables are crossing, corner pixels are detected and removed before finding cables. First, line pixels are smoothed using Gaussian smoothing with the 3 x 3 kernel size. Then corner points are detected using the corner detection method proposed by (Harris et al., 1988).
Detected corners are removed from the line pixels. Then, the remaining line pixels are grouped by investigating the eight neighbours, as shown in Figure 15. Since each group may still contain multiple cable lines, they are further subdivided by detecting line segments using the RANSAC method. In the RANSAC method, two pixels are randomly selected from each group, the line equation is calculated, and the number of line pixels on the straight line are counted. This process is repeated many times, and the line segment with the most pixels is selected. The subdivided lines are maintained as 2D cable lines.

Merging Lines Extracted from Point-Cloud and Images
From a point cloud, 3D cable points and cable planes are extracted for road-crossing cables. From an image, 2D cable lines are extracted. Then the correspondences between the 3D cable plane and the 2D cable lines are calculated.
3D cable points are projected onto the image using the pinhole camera model shown in Figure 6. Figure 16 shows projected points in black. For each projected point, the nearest 2D cable  -2-2020, 2020XXIV ISPRS Congress (2020 line is searched, and the 2D cable line and the 3D cable plane are associated if most points on the 3D plane are projected on the 2D cable line. Then, pixels on each 2D cable line are projected onto the associated 3D cable plane. Suppose pixel ( ' , ( ) on the normalized image plane is projected to ( ' , ' , ' ) on the cable plane. Point ( ' , ' , ' ) can be calculated as the intersection between the normalized image plane and a straight line from the optical center, as shown in Figure 17. However, since pixel ( ' , ( ) is corrected using Equation 1, the pixel ( #, #), which is the position before correction, has to be calculated. In order to obtain the pixel position before correction, the inverse calculation of Equation 1 is required. Here, we rewrite Equation 1 as follows.
Then ( # ' , # ' ) can be calculated using the following optimization function.
We solve this function using the Quasi-Newton method with the initial values A ' , ( B.

Reconstruction of Cable Lines in 3D Space
Points from a point cloud and an image are unified on the cable plane. However, points from the image may deviate from 3D cable points due to calibration errors. Therefore, 3D points from the image are aligned to the points from the point cloud.
First, points from the image are approximated to a catenary curve defined as: where , , = parameters for a catenary curve. For fitting a catenary curve to points, first, the parabola is fitted to points as: and the initial values ) , ) , ) for , , are calculated as: Then the catenary curve is calculated using the non-linear leastsquares method, such as the Gauss-Newton method.
The catenary curve is moved to fit the maximum number of 3D cable points on the cable plane. In our method, an angle resolution and a positional resolution are specified, and the catenary curve is rotated and shifted ( , ) around the center position, where , , and are integers. By substitute various integers for , , and within the specified range, the optimal transformation matrix can be obtained for the catenary curve.
Finally, points on the catenary curve are transformed using the optimal transformation matrix. In Figure 18, 2D cable lines extracted from an image were merged to 3D cable points on the cable plane. 3D cable points are shown in red, and 2D cable lines are shown in black. While there are missing portions in the initial 3D cable points, the merged points become continuous cable lines.

EXPERIMENTS
We evaluated our method using actual MMS data. Point clouds and images were measured in Fukuoka and Tokyo in Japan. By using the proposed method, we extracted and reconstructed roadcrossing cables. As corresponding images, we used images taken within 5 seconds from the GPS time of a point cloud. Figure 19 shows some of experimental results. On the left of the figure, the image corresponding to the point cloud is shown. On the right, the point cloud and the reconstructed road-crossing cables are shown. Reconstructed cables are shown in magenta. The lowest road-crossing cables are shown in red arrows in point clouds.
In these examples, all of the lowest road-crossing cables could be successfully reconstructed. This is because the distance between the cables and the MMS was relatively small and dense point clouds could be obtained from the lowest cables. However, some cables could not be reconstructed due to too sparse points or too far distance.
We displayed all the point clouds in the evaluation data on the screen and visually recognized 57 road-crossing cables. Our method could reconstruct 40 road-crossing cables from 57 ones. The success rate was 70%. Figure 19. Reconstructed road-crossing cables Our method could sufficiently detect the lowest cables closest to vehicles, but there are limitations to reconstruct all road-crossing cables. There were the following reasons why the cable could not be reconstructed: (1) Very few points could be captured from some cables because cables were far from the road or very thin.
(2) 2D cable lines could not be detected from the image due to poor contrast or too thin cable lines. (3) Catenary curves could not be fitted due to inadequate 2D line extraction.
The experimental results show that our approach is effective for reconstructing road-crossing cables, which cannot be reconstructed using existing methods. In order to further improve the success rate of our method, it would be required to develop more robust methods for detecting cables from a very sparse point cloud and extracting cable lines from images captured under various conditions.

CONCLUSION
Extraction of road-crossing cables are important to avoid interference between vehicles and cables. However, when roadcrossing cables are measured by the MMS, large portions of points on cables may be lost. In this paper, we proposed a new method for reconstructing road-crossing cables using point clouds and camera images. In our method, points on cables were extracted from a point cloud and the vertical cable plane was calculated as the projection plane for 2D lines on the image. From images, line segments were extracted and continuous cable lines were detected. Then correspondences between 3D cable points and 2D cable lines were detected, and 2D cable lines were projected on the cable plane. 3D points and 2D cable lines were merged on the cable plane, and they were fitted to a catenary curve. In our experiments, our method could sufficiently reconstruct road-crossing cables.
In future work, we would like to develop a robust method for detecting cables from a very sparse point cloud. We would also like to investigate machine learning techniques for robustly extracting cable lines from images.