WI-FI RSS FINGERPRINTING FOR INDOOR LOCALIZATION USING AUGMENTED REALITY

Indoor localization has attracted the attention of researchers for wide applications in areas like construction, facility management, industries, logistics, and health. The Received Signal Strength (RSS) based fingerprinting method is widely adopted because it has a lower cost over other methods. RSS is a measurement of the power present in the received radio signal. While this fingerprinting method is very popular, there is a significant amount of effort required for collecting fingerprints for indoor space. In this paper, we propose an RSS fingerprinting method using Augmented Reality (AR) that does not rely on an external sensor resulting in ease of use and maintenance. This method uses spatial mapping techniques to help align the floor plan of existing buildings; then, after the alignment, we map local device coordinates to global coordinates. After this process, we partition the space in equally distanced reference points for RSS fingerprint collection. We developed an application for Microsoft HoloLens to align the floor plan and collect fingerprints on reference points. Then we tested collected fingerprints with existing RSS based indoor localization methods for its accuracy and performance.


INTRODUCTION
Indoor localization has gained popularity in recent years due to the availability of mobile devices and increased demand for a solution for positioning and navigation for indoor spaces. Global Navigation Satellite System (GNSS) has been very dominant and successful for outdoor uses. However, since GNSS requires Line-of-Sight (LOS) transmission paths, it does not work best in indoor spaces. Wi-Fi signals can also be used for indoor localization (Liu et al., 2007). Nowadays, Wi-Fi technologies are found everywhere in various forms; it gives the advantage to this technology to provide indoor location-based service.
The RSS fingerprinting based localization systems are usually implemented in the 802.11 wireless local area network to determine user location by measuring frames sent from different access points (APs). Localization/positioning techniques that use Wi-Fi RSS measurement have two phases. In the first phase, the system has to build a database of fingerprints, where each fingerprint is the RSS information collected by measuring received signal strength at reference point from access points. In the localization phase, RSS signals measurements from APs are compared from the fingerprint database using mathematical modeling methods to estimate user location (Zegeye et al., 2016).
Most of the currently available methods to build a fingerprint database are tedious and labor-intensive (Zhuang et al., 2016, Wang et al., 2016. The non-stationarity of Wi-Fi signal distribution creates another challenge due to the radio signal propagation effect induced by changes in the environment (Ali et al., 2010), which requires the fingerprint database to be updated periodically to train the system for better location estimation.
To build a Wi-Fi fingerprint database, usually, it requires to survey every reference point and record fingerprint at each point * Corresponding author (Cheng et al., 2014). This method provides an improvement in the reliability of a fingerprint database by averaging the RSS at each reference point and also provides a coarse of orientation (Bahl andPadmanabhan, 2000, Xiang et al., 2004). However, this method is very time-consuming and labor-intensive if reference points need to cover a big area, and surveyors are to label reference points manually (Fuller, 2009). This process can take several hours to complete even for small spaces and is prone to human error (Bolliger, 2008). (Yang et al., 2012) proposed LiFS, it assumes that human motion connects discrete Wi-Fi RSS under certain semantics. It suggests the idea what human participation should be minimized during data collection process. Another work tries to solve this problem by adopting a crowd-sourced fingerprint collection method to decease collection time (Niu et al., 2015). Many methods utilize external sensors or input. These external sensors are very expensive, hard to set up and maintain.
In this paper, we propose an approach to collect fingerprints for localization which utilizes an already deployed infrastructure with no knowledge of access points placement. We developed a system in AR that minimizes the effort for measuring the distance between the reference points and trying to figuring out their labels for fingerprint data using a semi-automated process. In addition, our user interface allows to align a floor plan with an indoor structure and generate equally sized reference points for automatic labeling while collecting fingerprints. We test our method on two different Wi-Fi based localization techniques to estimate user location in an indoor space: 1) kNN with Path Evalulation and Retroactive Adjustment and 2) Multiple RSSI with Predicated Locations Input, Multiple Location Output (P-MIMO) Long short-term memory (LSTM) nerual network, and compared our AR-based method to a manual approach.
This section gives an overview of existing research. Section 2 discusses importance of data collection in different scenarios. Section 3 introduces and discusses the method for fingerprint collection using AR, structuring files and databases. Section 4 details information about experiments. It describes methodologies to test accuracy improvement. And finally, comparison and discussion between fingerprint data collected manually and AR assisted.

BACKGROUND
Modeling Urban Environments As more and more cities support initiatives toward sustainability, there have been projects that aim to measure better, analyze, and optimize urban settings. Creating a digital twin (Ruohomäki et al., 2018) of an environment enables this by a virtual scene of buildings, objects, people, and information gathered from the real world reflecting reality. 3D Town (Corral-Soto et al., 2012) demonstrates it in a campus setup, a static collection of 3D building models placed on a scene is populated by virtual pedestrian and vehicle avatars as detected and located from a fixed camera. These systems become a platform for analysis and data-driven applications. Figure 1 shows how our application fits in this overarching theme.
Modeling With the ease of data collection, modeling, and computing power, an exponential increase of level-of-detail in BIM and GIS has given a more exhaustive representation of the real world, giving way to new applications like asset/facilities management and indoor/outdoor navigation. More recently, developments in accommodating representations of combined BIM and GIS (Hor et al., 2018), will only continue to inject more data as we encounter more IoT-equipped infrastructures.
Urban Analytics At the advent of IoT, more and more sensors are installed in buildings to monitor usage in addition to improving the capabilities of cameras. Along with hardware improvements, the rapid advancement of computer vision techniques enables pedestrian detection and tracking. Data derived from these algorithms can help understand crowd behavior and comfort (Yang et al., 2019) in an urban environment, all of which can help improve accessibility, operations, and security.
Augmented Reality Mobile AR has opened possibilities in bridging virtual and real environments (Ramos et al., 2018). From in-situ creation of models and interior layout design to creating immersive applications such as indoor navigation and facilities management. In this work, we utilize AR for the rapid collection of data for large scale modeling. All the related urban applications requiring modeling and navigation also benefit from a fast deployment and maintenance.

METHOD
For this paper, we propose an approach to solve the localization problem. To achieve this goal, we created a user interface to manually align floor plan with real-world space with the help of AR, and then using spatial tracking techniques; we can track all reference points on the same floor. Figure 2 shows the framework for this method. Figure 3 show the user interfaces developed for this framework.

Floor Plan Alignment
In this phase, a floor plan is selected and displayed on the floor by utilizing spatial understanding function in the AR framework. After that, AR application presents user controls to align the floor plan with an indoor environment manually. This tool gives the ability to scale, rotate, and move floor plan. Once manual alignment is finalized, the floor plan can be tracked using spatial tracking function in the AR framework.

Spatial Tracking Function using the Hololens
Spatial Tracking Function is based on the parallel tracking and mapping method (Klein and Murray, 2007). In this method, first off, an initial map is generated based on a known stereobased technique, which is based on the estimation of essential matrix parameters using the five-point algorithm and the Random sample consensus (RANSAC). In the meanwhile, using the image motion techniques, initial poses of the camera are estimated. By having this pose, map points can be projected into the images. Following this, the camera poses are updated using a few coarse-scale features. The final poses are then estimated for each frame according to a large number of corre-spondences coming from the previous stage. By having accurate poses for the camera, the floor plan can be simply tracked by the HoloLens.

Space Partitioning
In this phase, the user is provided with an option to select the size of a reference point for generating a grid on the floor plan. A grid is generated and visualized on AR display with labels for each reference point. Figure 5 demonstrates space partitioning and labels of reference points with reference to a floor plan.
Rn,m is grid of reference points generated based on input from ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume V-4-2020, 2020 XXIV ISPRS Congress (2020 edition) ri,j = RSSI1 RSSI2 · · · RSSI k (2) ri,j is reference point which contains k samples of RSSI readings, where k = 1, ..., K.
These reference points are tracked for the same indoor environment during the fingerprint collection process.

Fingerprint Collection Tool
After generating the grid on the floor, the user can easily navigate between reference points. The tool provides a user interface to start the fingerprint collection process. Information of visible APs is collected for each reference point. This information includes the AP name, SSID, and time of reading. This data is collected and then stored on a JSON file on the file system.

Storage
The fingerprints for reference points are stored on JSON files. Each reference point has one text which contains fingerprints collected at that location. The format of the JSON file is shown in Listing 1.
In the offline phase, these JSON files are read, and a database is created. A simplified information structure is generated in order to speed up initialization during the training or testing phase, as shown in Table 1. Filtering also occurs for real-time readings during localization as well as test readings used for performance evaluation. The case occurs on a fingerprint-by-fingerprint basis; a fingerprint may be removed or merged with the neighboring fingerprint depending on the number of APs representing that fingerprint and/or the timestamps of RSSI readings contained in it. If fingerprints read by the required for localization algorithms are less than expected; then, the remaining expected RSSI readings will be added from the following fingerprint with timestamps a millisecond later than the previous fingerprint.

Filtering
In this way, two fingerprints can be combined into one for preparing the fingerprint database for the localization estimation algorithm. In some rare cases, if the fingerprint is incomplete and is followed by another incomplete fingerprint, that fingerprint is discarded as it can create unreliability. The main criterion for creating a fingerprint is a threshold for the minimum expected RSSI readings for any fingerprint attained.

Fingerprint Collection with Microsoft HoloLens
The experiments were set up in an indoor space of Petrie Science Building at York University. Figure 4 shows the floor plan of the test site. For these experiments, the existing infrastructure of the building was used, no additional access points were deployed. The data collection was done during working hours.
We developed an application for Microsoft HoloLens using Mixed Reality Toolkit (MRTK) 1 and Unity3D 2 . The application offers functionality to select a floor plan and allow the user to manually align it to the real-world space. Once aligned, it can divide the floor plan into an equally spaced grid of reference points. The application can collect fingerprints at the reference points and store them in JSON files.
The fingerprint collection site used for these experiments has a dimension of 56 × 56 m. It has four corridors which has equal in length. Each corridor is 45.5 m long and 2 m wide. After dividing the floor plan into an equally spaced grid of reference points, we collected fingerprints on reference points that were in corridors. Total 648 APs were detected on whole floor. This includes 2.4 G and 5 G frequencies broadcast by one AP and these APs also have capability of broadcasting virtual APs. This training data is then used with two different indoor location estimation methods to perform evaluations.
For comparison of location estimation accuracy, we collected two different fingerprint datasets. One dataset is collected arbitrary while moving in one direction, collecting point after certain intervals and manually labeling them. The other dataset is calculated with proposed method in this paper. The AR application is deployed on Microsoft HoloLens 1 for this experiment. The fingerprint datasets were collected on Microsoft HoloLens with AR assistance and without AR assistance. We implemented two different localization algorithms for training and testing with these datasets and compared the results: 1. kNN with Path Evaluation and Retroactive Adjustment (Gulo et al., 2018) 2. RNN for RSSI Localization (Hoang et al., 2019)

kNN with Path Evaluation and Retroactive Adjustment (PERA)
This location estimation method constructs a database from JSON files generated during the fingerprints collection process. The database is set up as reference points in real-world space that respond to a particular location in the positioning area. The mean signal strength is calculated and associated with their corresponding identifiers. Figure 8 illustrates process of localization for this method.
This method is based on the k-Nearest Neighbours algorithm (a commonly used machine-learning-based algorithm for pattern recognition). This algorithm searches and compares a set of reference points for k-nearest points of that set to a query point. Euclidean or Manhattan distance is used as a proximity metric. The reference points found are used assigned a class or value to that query point. For this experiment we used value k = 1. This algorithm also evaluates the path of the subject over the last five epochs. It is assessed based on these criteria: 1) kNN proximity, 2) Short-term movement regularity, and 3) Longterm movement regularity. Based on these criteria, a final score is calculated. The highest-ranked path then identifies discrete locations for the subject's most likely path for the epochs considered.  Figure 10 shows the comparison result of both datasets. kNN + PERA with AR assisted fingerprint database has an accuracy of 1.29 m with 80% errors compared to kNN + PERA with manually collected fingerprint database with 1.52 m accuracy. Figure 9 shows 174 reference points where fingerprint data was collected. It demonstrates that for AR assisted dataset the number of correctly identified reference points are 149 compared to 141 reference point using manually collected dataset which is 4% improvement. Also AR assisted collection process takes less time for collection.

RNN for RSSI Localization
This location estimation algorithm uses a recurrent neural network to train and estimate user location. For this experiment, we are using the P-MIMO LSTM model for training and testing. This method works in two phases: 1) Offline phase 2) Online phase. This process is demonstrated in Figure 11.
In the offline phase, it reads the fingerprint database and filter data. Then it calculates Euclidean distance between each point with every other point in the database. After this step, a probability map is generated to represent probability on user location in a trajectory. These trajectories are then fed to the neural network for training.
In the online phase, the user's test location is input to the trained RNN network for getting an estimated location. The sliding window averaging technique is used to average the error of predicted location to increase localization accuracy. We trained this model with both datasets collected for the previous experiment and tested the accuracy of this method. In this experiment, we used a laptop computer equipped with i7-7820HK Four-Core CPU and an Nvidia GTX 1080 GPU. The running time is approximately 3 s per epoch. Therefore, the training time is approximately 3 × 1000 = 3000s (≈ 33 minutes). Figure 13 shows the results produced from experiments with both datasets. It shows the RNN with an AR assisted database produced a slightly better accuracy 1.24 m with 80% of errors compared to 1.47 m with manually collected fingerprint database.   Figure 12 shows map for correctly and incorrectly identified reference points. We can see that the results from the AR assisted fingerprints dataset has more correctly identified reference points. This fact indicates that with the help of AR, we can divide the indoor space more uniformly (i.e., uniform space portion), which results in improved accuracy of indoor localization techniques. This observation suggests that an RSSI field generated from uniformly partitioned space can minimize a negative effect of spatially biased representation of the signal field and thus increase localization accuracy. Our experimental results show that with AR assisted dataset from 174 reference points, the number of correctly identified reference points are 153 compared to 144 reference point using manually collected data set which is 5% improvement.
One of the possible reasons why more incorrectly identified reference points can be found around the corner is there are fewer Wi-Fi hotspots available and also because corners also have doors to the stairs. Changes to the environment and obstruction cause a multi-path effect for Wi-Fi radio signals, which in turn causes some errors at those reference points. This negative corner-effect can be found in both results obtained from ARbased and non-AR-based fingerprint datasets.

CONCLUSION
This paper presents a novel process for manually aligning a floor plan with the indoor environment with the help of AR and then generates a grid of equally spaced reference point cells of an easy collection of Wi-Fi RSSI fingerprints. This method simplifies the process of fingerprint collection by eliminating the need for any extra and complicated setup of external hardware or sensors. It also generates labels for fingerprints collected at reference points. Since this method simplifies the process of collecting fingerprints, it is less labor-intensive and easy to use, which in turn helps in collecting more samples for more improved accuracy. The experiment results demonstrate an improvement in the accuracy of existing indoor localization techniques. In future research, we plan to automate the process of aligning the floor plan with the indoor environment without the help of an AR user interface. Also, we will verify the efficiency and accuracy of the proposed AR system in a large and diverse environment.

ACKNOWLEDGMENTS
The project, entitled "Intelligent Systems for Sustainable Urban Mobility (ISSUM)", has been supported by the Ontario Re-search Fund (ORF) Research Excellence Program, the Natural Sciences and Engineering Research Council (NSERC) and Data Analytics and Visualization (DAV) CREATE Progam.