CREATING IMMERSIVE AND INTERACTIVE SURVEYING LABORATORIES IN VIRTUAL REALITY: A DIFFERENTIAL LEVELING EXAMPLE

Surveying engineering education includes several outdoor laboratories that complement and enhance theoretical concepts taught in class. In addition, outdoor laboratories develop student skills with instruments and surveying techniques. These laboratories are often affected by weather, leading to cancelled laboratories, which reduce the time students spend with instruments and disrupt/delay the academic plan. Furthermore, terrain characteristics are important in surveying, as each terrain and project introduce unique surveying challenges. However, training often takes places in one location, thus, limiting student comprehension and experience on how to use the same instrument and techniques in different terrain conditions. Virtual reality constantly gains ground in education, as it overcomes restrictions of physical laboratories and enhances student learning. This study discusses the development of a leveling laboratory in immersive and interactive virtual reality, as well as the challenges encountered. We have replicated a part of the Penn State Wilkes-Barre campus, where students conduct many of their physical laboratories, in virtual reality with geometric and photorealistic fidelity using remote sensing and photogrammetric methods. Dense point clouds derived from terrestrial laser scanning and small unmanned aerial surveys are used for terrain and man-made object modeling. In addition, we have developed software that simulates surveying instruments, their properties, and user/student interaction with the instrument (e.g., moving the tripod, leveling the level instrument and leveling rod, etc.). This paper demonstrates that by utilizing cuttingedge remote sensing and virtual reality technologies, we can create realistic laboratories that can supplement physical outdoor laboratories and improve/enhance undergraduate instruction of surveying students.


INTRODUCTION
Surveying and Geomatics engineering education (and related disciplines) involves several outdoor laboratories that develop student skills associated with using surveying equipment and collecting spatial data. The laboratories further enhance and connect theoretical concepts learned in class with physical practical exercises and applications. A major challenge with outdoor laboratories is that they are weather dependent. Rain, snow, and low temperatures are common in Pennsylvania and in the northern United States. In addition, the recent coronavirus pandemic crisis forced many universities to switch to remote learning. Surveying / Geomatics programs faced a real challenge as they had to cancel their outdoor labs. Cancelled laboratories are therefore a reality during semesters, which can reduce the time students spend with equipment. This reduces their opportunity to get accustomed with instruments and surveying techniques and develop the necessary field skills. First year surveying students rely on such laboratories to develop their surveying skills (Bolkas, Chiampi, 2019). Furthermore, cancelled laboratories disrupt the educational process and course plans, as laboratories become misaligned with lectures. Another issue we identify is that most outdoor activities take place in the same location. Surveying engineering requires data collection in many different environments such as urban versus rural areas, flat versus mountainous areas, highway and building construction sites, etc. Each project presents unique terrain challenges. It is very difficult for surveying programs to conduct laboratories in * Corresponding author different situations and in off-campus locations due to constraints related to transportation costs (of students and equipment), accessibility to such sites, as well as safety issues (liability). This limits students' comprehension on how to use techniques and instruments in real applications and results in them being underprepared for the job market.
With the advent of low-cost head mounted displays (HMD), virtual reality started gaining ground in education. Virtual reality has been used to address issues with (Freina, Ott, 2015): (i) travelling in time (past or future), (ii) physical inaccessibility, (iii) limits due to a dangerous situation, and (iv) ethic problems such as in surgery. In engineering disciplines, including surveying engineering, items (ii) and (iii) are encountered frequently as discussed above. Virtual reality in education has been found to improve students' academic performance and motivation, students' engagement, social and collaborative skills, psychomotor and cognitive skills (for a review see Martín-Gutiérrez et al., 2017). In addition, game-based environments, which can be created through virtual reality software, have been found to increase student motivation and engagement (Coller et al., 2011;Coller et al., 2014). Virtual reality and interaction of users with virtual environments encourages students to be active learners, permits autonomous exploration, promotes decision-making, allows for more interaction than conventional learning methods and in general promotes a constructivist approach of learning (Martín-Gutiérrez et al., 2017). Bolkas, Chiampi, (2019) discussed the challenges of physical laboratories in first year surveying engineering students and suggested the use of immersive and interactive virtual reality to supplement physical laboratories.
The main objective of this study was to create a leveling laboratory in immersive virtual reality. This objective is divided into two main smaller objectives: (i) create a virtual environment, (ii) create software to handle the leveling instrument in virtual reality. This paper discusses the technical part of the leveling virtual reality laboratory. First, we start with a description of how we used terrestrial and aerial remote sensing methods to create a virtual environment; then we discuss the main features of the software that handles the leveling laboratory; we continue with practical and implementation considerations of the virtual reality labs. Finally, we conclude with important remarks and points for future work.

VIRTUAL ENVIRONMENT
An integral part of the virtual reality laboratory is the virtual environment. We replicated part of the Penn State Wilkes-Barre campus with geometric and photographic fidelity in virtual reality.

TLS and sUAS Datasets
Remote sensing and photogrammetric technologies were essential in order to derive accurate and reliable geometric information. We use and combine dense point clouds derived from terrestrial laser scanning (TLS) and small unmanned aerial systems (sUASs). The two point clouds are combined to deal with advantages and disadvantages of each sensor. For instance, TLS is a line of sight instrument and based on the scanner setups there might be data gaps and obstructions (see Figure 1b and 1d). On the other hand, sUAS surveys provide imagery with bird's eye view when acquiring nadir or near-nadir images (Figure 1a and 1c). Oblique imagery can provide the means for complete 3D reconstruction of objects; however, this was not done in this study because most of the building information comes from the TLS information. Therefore, the sUAS dataset is used to fill gaps existing in the TLS dataset. The two datasets use the same control network, which was established with Global Navigation Satellite System (GNSS) static observations. They are geo-referenced with respect to North American Datum 19832011[NAD 83 (2011], State Plane Coordinate System, Pennsylvania North Zone. Comparison and accuracy assessment of the two datasets showed a sufficient agreement of 1-2 cm in parking lot and asphalt areas, 3-5 cm in field and grass areas, and 4-8 cm in building areas Bolkas, 2019).

Terrain and 3D modeling
The combined point cloud from TLS and sUAS is used to create the terrain and model buildings and other man-made structures. The TLS and sUAS point-clouds were merged using a semi-automated algorithm (Bolkas et al., 2020). The fused point cloud was classified into terrain and non-terrain objects using the dense point cloud classification tools in Agisoft Metashape. The two classification classes were then manually refined. The terrain class was then used to create a combined terrain mesh (Figure 2b). The terrain mesh initial contained more than 4 million faces. It was then smoothed and reduced to about 50 thousand. This was done to reduce complexity of the terrain, as complex terrains can interfere with the virtual tripod and user experience by reducing the frames per second in virtual reality. The terrain mesh is then brought into Unity Game Engine and converted to Unity Terrain using the tools developed by Telksnys, (2017). Man-made structures were manually modeled from the merged point cloud using the software Autodesk 3DS Max ( Figure 2a). Autodesk 3DS Max allows importing dense clouds and has several tools for 3D modeling and texturing. To model buildings we fitted standard primitives such as square and irregular shaped boxes ( Figure 2a). Independent comparison using checkpoints collected with total stations and compared with corresponding points in their modeled versions showed standard deviations that do not exceed the 10 cm, thus confirming high geometric accuracy of the virtual environment.

Texturing
Texturing of the 3D objects is important to provide a sense of photorealism to the environment. Because the environment we created is based on a real-world location, it is important to create custom textures that mimic the actual scene. TLS point clouds can be accompanied with RGB values when capturing point cloud data, however, these photographs are often of low resolution and therefore insufficient for texturing. In addition, collecting imagery considerably delays data acquisition. sUAS imagery also has insufficient detail for texturing in virtual reality, as they are captured from high altitude. Users can freely navigate in the environment and they can come very close to modeled objects, hence they will be able to see the individual pixels when using images from TLS and sUAS means, this affects the overall immersive experience. Custom textures were created using close-up pictures of objects such as buildings, asphalt, grass, etc. These were edited in Adobe Photoshop, which also offers tools for creating normal maps for the textures. Normal maps are important to provide a depth illusion, they provide a 3D depth to 2D textures. The 3D objects and textures are imported into the Unity Game Engine, and the textures are applied by creating materials for each object. Figure 3a shows examples of these textures as they are applied in the 3D models and terrain in Unity. For instance, Figure 3a shows the front side of the Bell Center for Technology building, where we can see the asphalt, curb, and brick-wall textures. Visual comparison of Figure 3a with the point clouds in Figure 1 confirm that the physical environment is replicated with high fidelity. In addition, Figure 3b shows the smoking area that is located at the side of the Bell Center for Technology building (see also Figure  1a). Some 3D models and textures were ready-to-use from the Unity store, such as the trees and shrubs in Figure 1a, the wood texture and cars in Figure 3b. Ready models and textures were used in cases where creating our own textures and models might be time consuming, in cases where we think that replicating the physical model and texture is not necessary, and in cases where using a ready-to-use model or texture does not reduce the sense of being virtually at the campus (e.g., replicating trees can be time consuming and unnecessary).

SURVEYING LAB SOFTWARE
In addition to the virtual environment, we have developed software that simulates surveying level instruments, their properties, and user/student interaction with the instrument (e.g., moving the tripod, leveling the level instrument and leveling rod, etc.).

Surveying Equipment in Virtual Reality
The level instrument, shown in figure 4a, was modeled in Blender and based on a Topcon automatic level instrument. The dimensions were approximately measured using a tape, ruler, and calipers. Level instruments consist of several individual and moving parts such as the telescope, focus knob, tribrach screws, etc. These were modeled as standalone objects allowing us to programmatically add functionality to them. Note that instrument viewing magnification and field of few were faithfully replicated from the physical level instrument by comparing instrument documentation to real world and virtual calibration tests. In addition, taking the difference between the upper and lower crosshair and multiplying by 100, will yield the distance, as with actual level instruments. Tripod legs were also made standalone objects allowing us to add functionality that raises and lowers each leg.

User Interaction with Joysticks
The hardware selected for this project was the Oculus Rift, which provides two basic user interactions with the virtual world, namely these are selecting objects and grabbing objects. This limits user interaction with the world, especially when trying to replicate a surveying laboratory in virtual reality. Consider that surveyors are required to perform several complicated motions when they handle equipment and instruments. For instance, surveyors grab and rotate several instrument parts such as tripod legs, tribrach screws, and knobs on the instruments. In our leveling software implementation users can grab the objects like the tripod and instrument and the leveling rod and move them to different locations. Movement takes place by using a joystick and pushing in the desired direction.

User Interaction Using a Virtual Tablet
Due to current hardware limitations hand rotation and dexterity is an interaction that currently cannot be implemented with high accuracy; therefore, to deal with this shortcoming, we have developed a virtual tablet similar to an iPad. As seen in figure 5 this tablet appears in the user's virtual hand and allows for a consistent place to display menus. This system allows a user to interact with the equipment with a high degree of accuracy. For example, if the user wants to raise or lower one leg of the tripod, then the user selects the leg and the corresponding menu appears on the tablet screen with appropriate user interface controls which allows for a tripod leg to be raised or lowered. The other legs can similarly be raised or lowered individually. In a similar way we deal with most surveying functions such as rotating the tribrach screws, rotating the instrument horizontally, and turning the focus knob. For the rotation of tribrach screws and instrument we provide two option for the user, either grab the sliding bar, which provides a fast but coarse rotation, or use the fine movement arrows, which provide a slower but finer rotation (see Figure 6). Note that the tribrach screws are color coded, thus, the user can identify the slider and screw correspondence. In the current version users can select up to two screws, mimicking that surveying techniques, where a surveyor moves two screws first and then the third screw. Feedback from students indicates that they would like to have all three screws available for them in the virtual tablet because selecting and deselecting screws was considered annoying. The user can lean towards the instrument and see through the telescope, however, observing through the telescope can be difficult. For ease of use a screen appears on the tablet, which projects the same view from the virtual telescope.

Virtual Leveling Rod
With regards to the leveling rod, the user can grab the rod and place it above a surveying mark, raise the rod, rotate in three directions to level it (i.e., with respect to three rotations based on a Cartesian system). The user can aim towards the rod and take an observation. The existing lab requires the user to position both the instrument and the rod. Future developments will allow for multiple students to co-exist in the same virtual environment. The current user limitation can be considered an educational advantage because the user will experience the roles of both handling the instrument and rod. This gives the opportunity for students to understand all steps necessary with differential leveling field procedures and develop both skills (of instrument operator and rodman) at the same time.  Figure 6. (a) virtual differential level instrument; (b) leveling menu for the differential level instrument. Note that tribrach screws are color-coded for ease of use.

Virtual field book
The ability to write in a field book is another function that cannot be replicated in virtual reality. Taking off a virtual reality headset and writing in a real-world field book breaks the immersive experience and causes the user to feel disorientated. To deal with this challenge we developed a virtual field book. This is a six-column cell-based field book that appears on the tablet screen. The user can select a cell and use the select trigger function and a virtual keyboard to record measurements. The field book is empty by default (as in real life), thus the user can prepare the field book for several leveling tasks (e.g., leveling using the middle crosshair versus three wire leveling).

Virtual Pedometer
Finally, students have access to a virtual pedometer in order to conduct pacing and balance their backsight and foresight distances. Since the virtual environment does not track feet placement is was necessary to develop a pacing analog. In addition, students can use their pacing distance measurements to estimate theoretical survey misclosures. The pedometer is using a user entered value for the length of one pace (e.g., 3 feet per pace). Another option for students is to use 3-wire leveling to obtain a distance between the instrument and rod.

INSTRUCTIONAL FEEDBACK
An important advantage of the virtual lab is that we can derive important instructional feedback, that cannot be derived in physical implementations. It is a common issue of students not being able to achieve closure requirements and not being able where was the mistake. This often leads to frustration and negative lab experiences. In the virtual software developed here, after the completion of the lab a PDF document is generated. The PDF contains a page each time a student enters a value in the field book. The instructor can use this information to detect student mistakes during the virtual leveling laboratory. Such mistakes can be forgetting to level the instrument or rod, imprecise leveling of the instrument and rod, not balancing the distances between backsight and foresight measurements, or incorrect readings of the rod. Figure 7 show the parameters that are recorded every time. For example, we have information about the event date and time, rod mis-leveling, leveling instrument mis-leveling, distance from the instrument to the rod, actual rod measurement (in the example of Figure 7 the actual measurement is 1.374 m and the observed value is 1.371 m), elevation difference between instrument and rod, and focus percentage. This is information that students and instructors do not have during traditional physical laboratories. This information provides important feedback and is an advantage of virtual reality surveying laboratories. The instructor can understand how well students follow surveying techniques and use this information to help students understand their mistakes. Furthermore, recordings of the entire leveling lab allow instructors to assess student critical thinking in terms of deciding the leveling path, turning points, etc., as well as overall comprehension of theoretical concepts and leveling measurement procedures that were taught in class.

VIRTUAL REALITY LABS
The software described above allows us to conduct several leveling laboratories with modifications in each exercise. The developers have the ability to establish survey markers with known coordinates and elevations at any place in the virtual environment. Therefore, we can create an infinite number of different leveling lines. In addition, the developed software allows students to drop a temporary point (turning point) to transfer elevations, as they would do in a real case.
In addition, we have options to change instrument and rod parameters, thereby changing the conditions of the laboratory. Such parameters include: (i) introducing a collimation error to the level instrument, thus the step of balancing the backsight and foresight distances becomes important; (ii) changing the leveling sensitivity for the level instrument and rod, thus making it easier or harder for students to accurately level the instrument and rod. These are great features in order to demonstrate to students the influence of improper field procedures on the accuracy of a leveling survey.

IMPLEMENTATION
The laboratories can be used to prepare students for the actual laboratory or supplement physical laboratories by changing conditions as described in the previous sections. The laboratories were implemented for the first time in a freshman surveying course, where leveling is first introduced. In the first implementation students conducted a closed leveling loop. The main goal of this implementation was to receive instructional and technical feedback. Figure 8a shows a top view of the virtual environment and the benchmark location of the leveling loop implementation. Physical labs in the discussed class take place in the large field areas at the right side of Figure 8a. The virtual implementation took place in the left side of Figure 8a. This area was selected as it contains more abrupt elevation changes, and we placed obstacles such as cars and signs in certain locations to make surveying more challenging (Figure 8b). Surveying from BM1 to BM2 did not have any challenges, as this was the first setup and we wanted to give students the time to familiarize themselves with the virtual setting and controls. In the line of sight between BM2 and BM3 there is a sign close to BM2 (Figure 8c), therefore, if students are not careful the rod will be behind the sign and students will not be able to read the rod, thus having to change the location of the instrument. Then from BM3 to BM1 there is an elevation change, plus there are trees, cars, and light poles (Figure 8b). Students have to consider all these obstacles to identify a suitable setup location. If students identify these challenges, they can complete the leveling loop with only three setups. showing the benchmark locations, the differential level instrument, and leveling rod; (c) screenshot showing the sign next to benchmark 2.

Physical Space and Side Effects
The implementation took place in a dedicated classroom, where there are 6 stationary workstations with associated software and hardware. Students had about a 5 foot by 5 foot space where they could freely move without tripping. Configuration of the Oculus headset includes virtual marking of the area where they can freely move. When students come close to that area a cyan-colored net appears on their headset. When students physically cross this net, the color turns to red indicating that they need to step back. This allows the students to stay confined to a safe area while immersed in the environment.
In addition, there are concerns about the side effects of using immersive virtual reality with students. Students unfamiliar with immersive virtual reality have reported nausea, headaches, and eyestrain (Regan, 1995). However, as users become familiar with the technology reports of symptoms drop (Regan, 1995;Kennedy et al., 2000). Administration of over the counter motion-sickness medication helps in reducing the nausea symptoms (Kennedy et al., 2000;Stanney et al., 2002). In addition, the exposure time is another factor that can affect the presence of motionsickness, headache and eyestrain symptoms. The aforementioned studies recommend immersion periods of about 20 to 30 minutes. Therefore, the leveling loop laboratory was designed to take 10 to 15 minutes for experienced users and 20 to 30 minutes for inexperienced users. We also allowed students to take breaks to reduce motion-sickness effects if required.

Student Feedback
Participation of students was voluntary, and students had to complete anonymous surveys to provide feedback. Students replied questions related to (1) general background, (2) general and surveying pedagogy, (3) technical feedback and side effects. From category (2)  The average scores show potential of virtual reality labs to enhance surveying engineering education. Keep in mind that only one out of seven students who participated in this study had used immersive and interactive virtual reality before. Therefore, almost all students were inexperienced with virtual reality. The latter has a connection with side effects, as inexperienced users will most likely suffer from motionsickness. Note that five out of seven participants felt nauseas, with three reporting a moderate severity and one high severity. Nevertheless, most students reported that they enjoyed the experience despite the nausea symptoms.
In the future, we will progressively add and test new laboratories or variations of the leveling laboratory (e.g., conducing a leveling laboratory in different terrain conditions, using different parameters, etc.).

CONCLUSIONS
This paper dealt with the technical aspects of developing an immersive and interactive virtual reality laboratory for surveying engineering education. The laboratory that was developed was based on leveling tasks. The virtual reality laboratories will be used to supplement physical surveying laboratories. In this paper we have discussed and demonstrated how TLS and sUAS remote sensing methods can be used to create virtual reality environments suitable for immersive virtual reality environments. The two sensors complement each other, in terms of data gaps, and they can be combined to create a complete point-cloud dataset. The combined point-cloud was used to create the virtual terrain and geometrically model man-made structures such as buildings. The developed virtual environment replicates the physical one with sufficient geometric accuracy. Custommade and open source textures provide photorealism and connection between the physical and virtual environments. These demonstrate the important role of photogrammetric technologies on building virtual environments.
In addition, we have discussed the main functionalities of the developed software that controls the leveling instrument and other related functions (e.g., taking measurements, note keeping, user interface, etc.) in virtual reality. To overcome limitations in virtual reality hardware technology we have developed a virtual tablet, where the user can control the leveling instrument and rod, and have access to the virtual field book for recording measurements. Instructors have access to important information related to the virtual reality lab such as actual location and heights of benchmarks and turning points, off-level conditions of the instrument and rod, and distance between instrument and backsight and foresight observations. These can greatly assist instructional activities to detect mistakes such as wrong measurements, imprecise leveling of the instrument and rod, critical thinking on selecting leveling path and placing turning points. Furthermore, the developed software allows us to change surveying conditions and test several surveying scenarios such as different collimation error, different leveling sensitivity, different locations for beginning and ending benchmarks.
Student feedback of our first implementation showed promising results as average scores were about 3.8-4.0 out of 5.0 in questions related to the use of virtual reality to help them understand surveying methods, operate surveying instruments, and prepare them for the real lab. Despite restricting the lab to less than 30 minutes, most students felt motion-sickness, which is an important limiting factor. We expect that these symptoms will get reduced with more frequent use of virtual reality and improvements to the virtual environment.
Future work will focus on testing more virtual labs (with emphasis on first-year surveying students) to increase the student sample, assess their effectiveness in enhancing surveying engineering education, and receive feedback and perform changes in the environment and software. We plan to model other instruments such as total stations and global navigation satellite system receivers, which will allow us to create a plethora of virtual labs. In addition, we plan to include more virtual environments that can provide different surveying conditions (e.g., city environments, construction sites).