EXTRINSIC CALIBRATION OF ROTATING 2D LASER RANGE FINDER AND CAMERA USING PHOTOGRAMMETRIC TEST FIELD AND PING PONG BALLS

: In this article, a method for the extrinsic calibration of a 2D laser range finder and a camera is presented. This technique produces a 3D point cloud from the test field by connecting a laser range finder to a servomotor. In this study, ping balls and standard photogrammetric targets were employed as a test field. Ping pong balls are used because they can be easily recognized in data from laser range finder and camera. To calculate extrinsic calibration parameters between a camera and laser range finder, these balls are employed as control points in the data. The extrinsic calibration of the Laser range finder and camera is carried out using the point cloud created from the test field and the photos captured from the test field. In this method, a sphere is fitted to each ping pong ball's points in the 3D point cloud, and the coordinates of that sphere's center are taken to be the coordinates of that ball. By measuring the distances between various targets in the test field, the scale can be resolved. This approach was compared with another state-of-the-art method. The proposed method is more accurate and stable than the alternative way, taking into account the average inaccuracy of check points.


INTRODUCTION
High-resolution cameras and 2D laser LRFs (LRFs) are commonly used in mobile robots and mobile mapping systems, and integrating these two sensors is a common difficulty in some applications.To give 3D geometry and color texture information in some systems, these two sensors must be correctly calibrated (Zhu et al. 2020), (Zhang and Singh 2014).To create a 3D LRF system, the LRF may be attached to a servo motor, and estimating the poses of the LRF using visual odometry is required to have a mobile 3D laser scanner (Zhang and Singh 2015) (Xi et al. 2019) (Bi et al. 2021).Acquiring a 3D point cloud from an indoor (J.Li, He, and Li 2015) or outdoor environment (L.Wei 2013), detection of pedestrians (P.Wei et al. 2018) construction of semantic maps (Iocchi and Pellegrini 2007), object distance estimation (Kumar et al. 2020) or lane detection (Yenıaydin and Schmidt 2019) for self-driving vehicles are only a few applications for using this kind of configuration.To effectively employ information from both types of sensors, an accurate estimate of their relative position, or extrinsic calibration, is required.Due to the difficulty of precisely installing and aligning the camera and LRF, data misregistration is a common problem.
The following features should be present in any extrinsic calibration method for calibrating a 2D LRF and a camera for non-experts: 1) The calibration target should be simple to build and detect, 2) The capability of the system to determine transformation values from a single sample, and 3) the system must be capable of coping with circumstances in which the camera and laser are not pointed in the same general direction (Palmer et al. 2020) 4) In situations where extrinsic calibration is used, such as in autonomous cars, it should be possible to update or modify it.Previous techniques (Ahmad Yousef et al. 2017) (Hillemann and Jutzi 2017) (Vasconcelos, Barreto, and Nunes 2012) (Gao et al. 2003) depended on identifying lines in laser readings and checkerboard patterns in images.However, it is challenging to use similar techniques to detect displacement in the Z-axis.Trihedron-based techniques for figuring out the height at which the laser meets the target have been created as a result (Gomez-Ojeda et al. 2015) (Hu et al. 2016).These targets, however, suffer from rotating problems, which is why techniques that made use of a v-shaped test field (Palmer et al. 2020) were created.Additionally, it has the benefit of allowing calibration from a single sample.Only (Palmer et al. 2020) employed ping-pong balls in any of the earlier techniques to calibrate the laser and camera, which had vastly different perspectives of the target.However, their method was not compared with the trihedron-based methods and The advantage of employing photogrammetric bundle adjustment (Fraser 1997) (Cronk, Fraser, and Hanley 2006) (Abbas et al. 2014) (Hosseininaveh et al. 2014) for precisely predicting the center of the ping pong balls were neglected because they used just one image for calibration.This paper suggests a method for calibrating a camera and a 2D LRF using a photogrammetric test field to address the shortcomings and restrictions of earlier methods.Trihedronbased approaches (Zhu et al. 2020) (Fan et al. 2019) and spherical target-based methods (Palmer et al. 2020) can both use the test field.In addition to employing the room's corner and ping-pong balls, bundle adjustment was employed as a contribution of this study to the related work (Zhu et al. 2020) (Palmer et al. 2020) (Fan et al. 2019) as an accurate approach for 3D measurements of targets (the center of the ping-pong balls) and the posture of the camera.Additionally, this work compares spherical target-based approaches with trihedronbased methods, which have not before been done in the literature.A current state-of-the-art approach (Fan et al. 2019) in the category of trihedron-based methods is implemented, and a different version of this method with various benefits is shown and compared with the original method.Figure 1 displays a test field with some black targets and ping pong balls.The room's walls and floor are covered in targets that are equally spaced.No prior knowledge of the coordinates for the control point is necessary for the operations outlined here.Due to the contrast between black and white, the control points in the photos are visible.In addition to being simple to recognize in photographs, ping pong balls are also simple to recognize in LRF data due to their spherical shape.The approach described in this work can be used to perform calibration operations using targets and ping pong balls placed in any room corner.

Figure 1. The photogrammetric test field
The remainder of this paper is structured as follows.In the section that follows, an overview of related literature is given.The third section discusses the proposed method for extrinsic calibration between 2D LRF and camera.Experiments and results are described in the fourth section, which goes into great detail about how to use a designed system to apply the method.The fifth section evaluates the method.The conclusion is the sixth section.

RELATED WORKS
The literature can be classified into two categories based on the types of range finders employed with the intention of extrinsic calibration between the camera and the rangefinder: beams that can be seen and unseen (Kim and Ha 2020).Regarding the calibration methods, (Khurana and Nagla 2021) offered an exhaustive review of all current calibration techniques for calibrating LRF and camera extrinsically.Both sensors simultaneously examined either natural sceneries (targetless) or predetermined targets (target-based) in the approaches.Based on the human or automatic extraction of feature correspondences, these two groups were then categorized (point-to-plane, point-to-line, point-to-point, line-to-plane).Target-based and manual calibration has been the subject of long-term research, whereas targetless and automation calibration have recently drawn greater interest.Contrarily, target-based methods rely on prior knowledge of the target, which increases the accuracy of calibration results but decreases the system's usability because it calls for the design and construction of the target and isn't always appropriate for runtime adjustments in the event of de-calibration.Target-based approaches frequently employ artificial calibration objects that are simple to identify by both sensor modalities, either manually or automatically.Examples include checkerboards, customdesigned targets, and polygonal boards.More than one view of the calibration board is typically needed to extract feature correspondences between the 2D LRF and camera to establish geometric constraints between these two sensors to estimate relative transformation.Researchers (28,29) developed a calibration method utilizing a chessboard and the point-on-plane constraint.This method requires at least 20 images of diverse postures to provide an initial closed-form solution for the extrinsic parameters.Sometimes it takes more than 20 shots to get good initials.Researchers have presented a calibration method based on trihedrons.The employment of specially designed boards, such as the circular, v-shaped (31,32) (Yang, Liu, and Patras 2012) (Itami and Yamazaki 2019), right-angled triangulation (G.Li et al. 2007), and cubic boards (Chen et al. 2012), was another strategy.
The above-mentioned strategies have two significant shortcomings.Despite taking numerous pictures, a few awkward photographs could trick the solution or make it unstable.They have many solutions or converge to a local minimum early, which could produce inaccurate calibration findings.Hu et al. suggested an extrinsic calibration method that only needed one shot at the target and produced a distinctive outcome (Hu et al. 2016).They used a tri-rectangular trihedron as their calibration pattern and estimated the camera and LRF positions using perspective-three-line (P3L) and perspectivethree-point (P3P), respectively.To accomplish a trustworthy and precise calibration between the camera and the LRF in a single shot, Fan et al. used a photogrammetric test field.The calibration of each sensor independently using the shared reference was used to perform the extrinsic calibration of the two sensors.By using the test field's room corner to solve a condensed P3P problem, they calibrated the LRF concerning the test field.The camera's extrinsic parameters concerning the test field were then obtained using a large number of control points (Fan et al. 2019).Although they used the technique in both simulation and actual indoor and outdoor scenarios, they needed to know the camera's lens distortion parameters beforehand to correct the one image they had taken from the test field.
In addition, this approach makes the unrealistic assumption that the walls and floor of the room are flat, devoid of curvature, and perpendicular to one another in pairs.This method also has the drawback of requiring a test field, the coordinates of which must be determined and made known beforehand.Despite these issues, their approach has the benefit of only requiring one image and one shot of the LRF data, as well as the ability to calibrate a specific corner of the room.This method is used in this study as the most recent state-of-the-art for calibrating LRF cameras, and the outcomes of the methods suggested in this article are contrasted with those of this method.Ping-pong balls were utilized to estimate the extrinsic calibration parameters between the laser and camera with dramatically diverse viewpoints of the target (Palmer et al. 2020) with the same objective of simplifying the calibration procedure for nonexperts.
In addition to the widely used techniques, such as Levenberg-Marquardt optimization (Moré 1978) and EPnP (Lepetit, Moreno-Noguer, and Fua 2009), a unique technique was developed for parameter estimation.Even though their method was demonstrated to be more accurate than others, because only one image was utilized for calibration, it was not possible to employ bundle adjustment as a reliable method for determining the poses of cameras and ping pong ball centers.
To sum up, the most recent methods for extrinsic camera and LRF calibration are trihedron-based methods and sphere targetbased methods.The research question arises, based on the literature, which of these two categories of methods has better accuracy.the following section describes the presented method based on this research question.

PROPOSED METHOD
As illustrated in Figure 2, four coordinate systems were defined for the 2D LRF, 3D LRF (which combines a 2D LRF with a servo motor), test field, and photogrammetric model.The coordinate system for the 2D LRF was written as ( ). center of the 2D LRF was defined as the origin of the 3D LRF coordinate system and was designated as ( ). the axis is located along the servomotor shaft.
axis is perpendicular to and is along the top of the servo motor.The coordinate system is also right-handed due to the definition of axis. the photogrammetric coordinate system ( ) was thought to have its origin in the optical center of the camera.The optical axis of this camera was the y-axis of this coordinate system, and the image plane of the camera was parallel to the plane.( ) is used to represent the coordinate system for the test field.The test field coordinate system was constructed with the vertex of a room corner as its origin and its edges as its axes ( ).Several photogrammetric retro-reflective targets and ping pong balls that were mounted on the room's corner walls made up the test field.Using a digital caliper, the distances between some of the coded targets were precisely measured.

Figure 2. Configuration of the coordinate systems
The approach was put out to determine the camera's relative orientation to the LRF.The technique is based on a 3D point cloud generated by a 3D LRF while a servo motor rotates a 2D LRF.For every point P, the coordinates of this point are = ( , , ) in the 3D LRF coordinate = ( , ) in the 2D LRF coordinate system, =( , , ) in the photogrammetric coordinate system, and = ( , , ) in the test field coordinate system.The rotation matrix and translation vector are abbreviated and , serve as the extrinsic calibration parameters between the photogrammetric coordinate system and the 3D LRF coordinate system.Any desired point P is used to build equation ( 1): (1) Where: rotation matrix between photogrammetric coordinate system and 3D LRF coordinate system = 3D coordinates of P translation vector between photogrammetric coordinate system and 3D LRF coordinate system.

Calibration using 3D point cloud
We need to locate the coordinates of some objects (the center of the ping pong balls) in two different 3D coordinate systems to establish the calibration parameters between the LRF and the camera.The optical center of the camera used in the main station (NOTE: the main station is the station where 2D LRF and cameras collect data) serves as the origin of the photogrammetric coordinate system, whereas the center of the 2D LRF serves as the origin of the 3D LRF coordinate system, as was previously mentioned.The 2D LRF data are registered to another in the first phase of this approach to create a 3D point cloud by spinning the LRF with a servomotor and using the encoder data of the servomotor.Each point associated with a ping pong ball in the point cloud is fitted with a sphere.The fitted spheres' centers and the ping-pong balls' centers share the same coordinates.The test field is photographed at the main station and numerous other stations in the following step.bundle adjustment is used to calculate the positions of the balls in the photogrammetric coordinate system.The third and final stage involves using the center of the ping pong balls in two distinct coordinate systems to determine the calibration parameters using the least square.Each step is thoroughly explained in the sections that follow.

Producing a 3D point cloud with rotating LRF
As shown in Figure 3, to generate a 3D point cloud, information from the servomotor and 2D LRF must be considered simultaneously.For each intended point P that the 3D LRF system successfully captures, three observations are obtained.equations (2) are used to determine each point's coordinates in a 3D LRF coordinate system, such as point P. (2) Where r = the distance between the target point P and the LRF center α = the distance between the vector r and the axis β = the angle between and axis measured by the servomotor encoder equation ( 2) and 2D LRF and servomotor data may be used to create a 3D point cloud from the test field.The ping pong balls are spherical, thus it is easy to identify their corresponding points.One way to accomplish this is to separate the points on each ball and then fit a sphere to each of them.The common locations discernible in both the rangefinder data and the camera pictures are thought to be the center of these spheres, which are the center of ping pong balls.

Estimating the control points coordinates using photogrammetry
At the same station (main station) where the 3D point cloud was formed in the previous stage, a camera is used to take a picture of the test field.If the camera is positioned differently and additional images are captured from the test field the bundle adjustment can be used to estimate the locations of the targets and the centers of ping pong balls as well as the exterior and interior camera parameters in each station (Triggs et al. 1999).
During the bundle Adjustment procedure, the center of ping pong balls and retro targets were taken into account as common locations in the photographs.To resolve the scale issue, the distances between various targets were measured and incorporated as a fixed constraint in the bundle adjustment procedure.The origin of the photogrammetric coordinate system was thought to be the optical center of the camera at the main station.

Computing extrinsic calibration parameters between the LRF and the camera
As was previously said, the optical center of the camera in the main station was the origin of the photogrammetric coordinate system and the origin of the 3D LRF coordinate system is the center of the 2D LRF in the main station.the rotation matrix and the translation vector between the 3D LRF coordinate system and the photogrammetric coordinate system will be the extrinsic calibration parameters.It should be noted that the three shared points (ping pong balls) in the two coordinate systems can be used to determine the six unknown parameters of the translation vector and rotation matrix.On the other hand, these three balls shouldn't be on the same line.
The relation between the photogrammetric coordinate system and the 3D LRF coordinate system is expressed by equation ( 1), which can also be represented as equation ( 3): (3) Taylor's method of linearization can be used to obtain the parameters of equation ( 3), which is referred to as 3D conformal.

EXPERIMENTS AND RESULTS
As shown in Figure 4, the designed gimbal was used to implement the proposed method.A laser range finder was mounted to the gimbal.The LRF is a Hokuyo URG-04LX-UG01 that is connected to a Dynamixel MX-28T Servo Motor through the designed gimbal.The Hokuyo LRF had a 240-degree field of view and a 0.352-degree angular resolution and an accuracy of 30 mm for ranges under one meter and 3% of the range for ranges over one meter, this LRF could measure objects out to a maximum of 5.6 meters.The MYNT EYE D1000 stereo camera had a 2.45 mm focal length and a 1280 x 720-pixel image resolution.To an accuracy of 0.088 degrees, the encoder of the Dynamixel motor could measure the shaft rotation.The entire setup was powered by a power bank and the processor was a Jetson Xavier AGX Developer kit.The images from the left and right cameras, as well as the LRF data, were recorded in the main station to implement approaches.Additionally, some convergent stereo images were taken around the test field from various locations.To estimate the interior and exterior orientation of the camera as well as the coordinates of the targets and ping-pong balls in the photogrammetric test field, Australis ("Australis -Photometrix Photogrammetry Software" n.d.), a well-known photogrammetry package, was used to perform the bundle adjustment.The camera manual was used to choose the initial values needed to bundle adjustment in each experiment using Australis.The pixel size was 3.75 nanometers, and the focal length was 2.45 mm.Both our proposed method and the method proposed by Jia Fan (Fan et al. 2019) are implemented.

Calibration using 3D point cloud
The system was positioned in front of the test field so that all of the targets were dispersed throughout the picture space in the main station to execute the calibration.The 2D LRF was spun by the servomotor at the main station to produce an extremely high-density 3D point cloud, as explained in section 3-1-1.The ping pong ball points are distinguished from other points easily.
A sphere was fitted to each of the ping pong ball's points once they were chosen in the GOM inspect software ("GOM Inspect" n.d.). Figure 6 shows the point cloud and fitted spheres.At the main station, the image from the test field was acquired with the left and right cameras of the stereo camera.In addition, several images from the test field were captured with the stereo camera in other stations.According to Figure 6, the targets and the ping-pong balls were treated as corresponding points in the images.The bundle adjustment operation was accomplished in Australis software.according to Figure 6 to tackle the scale problem, the distances between the S 0-b, c-d, e-f, g-f, and h-g targets were measured and they were treated as constraints in the bundle adjustment.The origin of the photogrammetric coordinate system was regarded to be the optical center of the left camera at the main station.This method was used to determine the targets' and ping-pong balls' coordinates in the photogrammetric coordinate system.The center of ping pong balls in the two coordinate systems of the 3D LRF and the photogrammetric model was used to determine the calibration parameters, as explained in section 3-1-3.The positions of the centers of the ping pong balls marked in Figure 6 with yellow numerals were used to determine the extrinsic calibration parameters for the LRF and camera (the numbers 1, 2, 3, 5, 6, 7, 8, 10, 13, 14, and 16).The locations of the ping pong ball centers, indicated with white numerals (the numbers 4, 9, 11, 12, and 15), served as the check points and were used to evaluate the extrinsic calibration accuracy.
4.2 Extrinsic calibration between a camera and a 2D LRF using a photogrammetric control field (Fan et al. 2019) 16 photos from the test field were used to alter the bundle to determine the targets' coordinates.The single image was rectified by removing the effect of lens distortions with the known camera calibration parameters estimated in bundle adjustment in agisoft metashape software.Following that, the DLT coefficients were determined using 56 control points with known coordinates that were approximated using a bundle adjustment.The DLT coefficients were then calculated using 56 control points with known coordinates estimated in bundle adjustment.The rotation matrix and translation vector of the single image concerning the test field were obtained using the DLT coefficients.On the other hand, the position vector and the rotation matrix of the 2D LRF relative to the test field were also obtained by the P3P method and using the LRF data captured from the corner of the room.Finally, using the rotation matrices and translation vectors of the camera and 2D LRF concerning the field test, the calibration parameters of the LRF relative to the camera were obtained.

EVALUATIONS
A 3D point cloud from the test field was used in sections 4-1 to determine the coordinates of the centers of the ping pong balls in the 3D LRF coordinate system.The bundle adjustment approach used in Australis was also used to get the coordinates of the ping-pong balls in the photogrammetric coordinate system.Table 2 displays the 3D coordinates of the check points that were acquired using the mentioned approach for both photogrammetric and 3D LRF coordinate systems.
By translating the check points' coordinates from the 3D LRF coordinate systems to the photogrammetric coordinate system using the extrinsic calibration parameters obtained with each of the methods presented, the estimated 3D coordinates of the check points in the photogrammetric coordinate system were calculated.Table 3 shows the estimated 3D coordinates of the check points obtained using the aforementioned procedure for each of the two approaches.using the estimated 3D coordinates of the check points, the difference in coordinates between the estimated values and their coordinates in the photogrammetric coordinate system attained in Australis was evaluated for all techniques (DX, DY, and DZ in Table 4).Using the following equation, the resultant error for the check points was determined: (4) Where DX, DY, DZ = differences between true and estimated coordinates values of P Figure 7 shows the error values for the check points in the implemented calibration techniques.The average of the errors for all check points is displayed in the final row of Table 4.

CONCLUSIONS
In this study, a new test field was used to propose a trustworthy and practical way for the extrinsic calibration of a camera and a 2D LRF.The presented method was then contrasted with a current state-of-the-art method.There were 16 ping pong balls on the test field in addition to the photogrammetric targets.In camera photos and LRF data, the ping pong balls were recognized due to their characteristic geometric shape.In this technique, a 3D point cloud was produced using servomotor data in addition to the range and camera data for the extrinsic calibration of the LRF and camera.The calibration parameters were calculated using the ping pong balls as common targets.In the (Fan et al. 2019) technique, the extrinsic calibration was carried out in the room's corner, and the P3P problem was solved using the three points acquired from the fitted lines on the LRF data on the walls and floor.The camera's position concerning the test field was then established.Finally, the extrinsic calibration parameters between the LRF and the camera were determined utilizing the external parameters of the LRF and the camera concerning the test field.The results of the trials showed how trustworthy the suggested approach is.

Figure 3 .
Figure 3.The structure of the 3D LRF system includes a 2D LRF and a servomotor connected by a gimbal.

Figure 4 .
Figure 4.The integrated sensor is composed of a 2D LRF, a stereo camera, and a servomotor

Figure 5 .
Figure 5.The test field provided for the calibration methods.The balls labeled with yellow numbers are control points used for estimating the calibration parameters and the check points are white.

Figure 6 .
Figure 6.The 3D point cloud of the test field is generated with the proposed system and the fitted sphere to the points of the ping pong balls.

Figure 7
Figure7shows the error values for the check points in the implemented calibration techniques.The average of the errors for all check points is displayed in the final row of Table4.Figure7illustrates the most accurate calibration technique, which makes use of the 3D point cloud (10.4 mm average error).The average error for Jia Fan's method(Fan et al. 2019)   was 15.5 mm.

Figure 7 .
Figure 7.The error values of the check points for the calibration methods.

Table 1 .
The estimated extrinsic calibration parameters of the LRF and camera using the presented methods.

Table 2 .
The 3D coordinates of the check points (mm) in both photogrammetry and 3D LRF coordinate systems.

Table 3 .
Coordinates of check points in two different coordinate systems

Table 4 .
The Errors in the Calibration Methods