AUTOMATED VOLUME MEASUREMENT OF A WASTE BIN WITH A SMARTPHONE APP BASED ON PHOTOGRAMMETRIC METHODS

In order to get accurate information about the volume of a partly filled bin of packaging waste a dedicated smartphone app was developed. The main functions of the app are to collect sets of images, which are sent to an external server (cloud) for automatic photogrammetric image processing. The results, basically the filling volume is transferred back to the user. The photogrammetric workflow includes an orientation and a volume calculation phase. The workflow was extensively tested for its functionality, performance, reliability and possible sources of error. Thereby several tests were conducted to estimate the optimum number of images, the best image configuration and the influence of different filling levels. The results demonstrate, that the filling volume may be automatically and accurately determined, if the configuration of the images is appropriate. Best results were obtained with many images taken nearly vertically.


INTRODUCTION
Sometimes people make the experience that waste bins are already full before waste collection and at other times, the bottom of waste bin is almost visible, when the trash is picked up by the truck. Increasing efficiency in waste management and smart cities is about optimizing waste collection. This in turn leads to lower exhaust and noise emissions. At the same time, fuel consumption can be significantly reduced by optimizing waste collection, Omara et al. (2018). In addition, flexible emptying intervals can be implemented that react directly to the degree of filling of the bin. This not only prevents overfull bins, but also conserves resources by reducing unnecessary collection trips (empty bins). In order to be able to use the advantages of optimised collection routes, it is necessary to record the degree of filling of the residential waste bins. Various methods for determining the degree of filling are described in the literature, e.g. Hannan et al. (2015) whereby, depending on the application, the mass of the waste, the filling level or the filling volume of the bin must be determined. For most processes, sensors are used which are permanently installed in the bin. Single sensor systems, such as infrared diodes, ultra-sonic sensors or weight cells may only determine one fill level value per waste bin and they work best in combination with a waste compactor.
However, waste can consist of larger elements with cavities (e.g. waste paper or packaging waste), which makes the waste bin not filled evenly. The filling height therefore varies greatly over the barrel cross-section and is no representative measure for the filling degree as long as no waste compactor is used. If the system should reliably record a filling level without a waste press, the filling volume has to be determined, which also takes uneven surfaces into account. Therefore, a single image approach will not work, thus the three dimensional surface of the waste surface requires a photogrammetric multi-image approach.
Therefore, photogrammetric methods shall be used which allow for a three-dimensional reconstruction the surface of the waste.
In order to keep the costs low on the hardware side, the approach pursued in this article is a mobile method based on photogrammetric image processing for determining the filling volume of waste bins from the reconstruction of the waste surface. The presented approach is divided into the following specific steps, which will be presented in this paper in larger detail:  Development of an app to use a smartphone as a costeffective measuring instrument.  Automated calculation of the filling volume from the reconstructed waste surface by photogrammetric image processing.  Linking the app with photogrammetric image processing in a bi-directional way.
Additionally, a quality assessment of the results will be given. The app is primarily intended for waste disposal companies who want to get an exact overview of the actual fill level of the bins to be collected in order to optimize their disposal trips. Of course, anyone can use the app to determine how much waste is produced, for example, daily or weekly.

MOBILE PHOTOGRAMMETRY
Photogrammetric methods are capable to create 3D reconstructions of an object based on images. Due to the increasing use of mobile devices and the technical improvements of the cameras and additional sensors, these devices are also used for photogrammetric purposes. Compared to tablets, the use of smartphones is advantageous due to is ubiquity and mobile use, due to the small mass and size of smartphones, and the comparably low-cost acquisition with increasingly better hardware in terms of camera and computing power.
For photogrammetric procedures, the 3D depth sensors already installed in some smartphones are most suitable due to dual cameras. This allows 3D reconstruction using a stereo camera approach e.g. Masiero et al. (2018). However, photogrammetric methods are already used by utilizing the built-in camera of commercially available smartphones, Barbero-García et al. (2018). These procedures are the subject of research in the many different areas, e.g. medical purposes, e.g. Barbero-García et al. (2019), for architecture, Pan et al. (2011) or on changes in the environment, Dabove et al. (2018).
There are three possible approaches for processing the data in near real time, whereby image acquisition is always realized on the smartphone:  Image acquisition and processing are separated (smartphone for image acquisition, server for photogrammetric processing)  Shared processing on the smartphone and the server  Image acquisition and processing on the smartphone.
If the image acquisition is separated from the photogrammetric data processing, the images can be processed using existing image processing software, either commercial or open software. If individual processing steps are shifted to the smartphone, a client-server model is necessary to increase performance. Fritsch & Syll (2015) for example use algorithms for camera calibration that are executed directly on the smartphone. To increase computing power, the images are transferred to a server on which they are processed. The 3D reconstruction is performed by Structure-from-Motion, i.e. by using several images of the object from different camera positions. Nocerino et al (2017) shift even more processing steps to the smartphone. In doing so, the calculations of the smartphone and those of the cloud are combined.
If all calculation steps are completely shifted to the smartphone, the sensors built into the smartphone, such as the gyro and acceleration sensors, can also be used as additional information. For example, Kolev et al. (2014) uses video sequences of an object. Muratov et al. (2016) achieves computing times of less than one minute when reconstructing textured 3D mesh on a tablet computer by parallelizing the computing processes. Since smartphone technology becomes more and more powerful and thus processing on the smartphone is possible in real time, most of the current research work has a focus on to the latter approach. Nevertheless, depending on the type of application, there are always pros and cons of processing on the smartphone versus processing on a server, see Table 1 for a summary.
When using photogrammetric techniques, camera calibration is a necessary element to extract 3D information from the images. The interior orientation of the smartphone camera can be achieved by self-calibration either directly on the smartphone (Fritsch & Syll (2015)), or by using image processing software on a server, Barbero-García et al. (2018).
For the latter option, Barbero-García et al. (2018) determined the influence of different parameters. They test the influence of the number and distribution of images, use of control points, different smartphone models and camera calibrations on a known spherical object. As a result, the distribution and number of images is particularly important for fully automated exterior orientation. For this purpose, the use of coded control points is useful because they improve the exterior orientation and additionally the accuracy of the reconstructed surface. The choice of the smartphone model (highest vs. medium price and quality level) reveals better results for the higher class model in terms of accuracy and orientation, but this can be compensated by the number of images and the use of more control points.

THE PHOTOWASTE-APP
A smartphone app was developed for the detection of the waste bin volume, the source code can be found at https://github.com/Pempelchen/TonneApp, Figure 1. It was realized with a tool called Expo (expo.io (2017)) using the libraries of the Expo SDK (version 32.0.0). The Expo SDK provides easy access to the native user interfaces of the iOS and Android operating systems as well as the smartphone's functionalities, such as the camera and local memory. Programming is done by using JavaScript only. Expo enables the development of native apps for iOS and Android. All tests were conducted with an Android smartphone.  After opening the app, the user has to choose to either start a new project, view already calculated volumes or read the app's instructions. When starting a new project, the steps as shown in Figure 2 are run through and finally the selected images are sent to the node server. The results of the calculations are transmitted by the node server back to the smartphone, to be accessed by the user. The implementation of the individual functions is described below. Several steps are necessary to upload the waste bin images with the app. The unique barcode, which is attached to every waste bin, acts as an identification key throughout the whole processing. Waste management companies generally provide waste bins with a unique barcode that contains information about the type of bin and the address of the owner / household and can therefore be used as a unique identification key. The general workflow of the App consists of four steps: 1. The barcode of the waste bin must be scanned, as the barcode is the key to link the images and the results to the respective waste bin. 2. Images of the waste bin are recorded from different directions with the smartphone camera using the flashlight. The images form a gallery, in which the captured images can be displayed, selected and transferred to the node server, Figure  3. In addition, calculated filling volumes of already, measured bins can be displayed in an additional screen. 3. After image collection the selected images from a waste bin and the meta data are transferred to a node server. The node server is monitored by a file monitoring software. This tool constantly checks for new files. As soon as it detects a new set of image files and the associated metadata, a batch script starts the photogrammetric image processing automatically. 4. The results, such as the filling level are transmitted back to the node server and finally to the smartphone App, where the results are displayed.

Automatic photogrammetric data processing chain
After the images import the automatic photogrammetric image processing follows (Figure 4). In the first step the elements of the interior (IO) and exterior orientation (EO) are defined. The IO of the camera may be determined and/or optimized by simultaneous calibration. However, experiments revealed that simultaneous calibration does not work stable during automatic processing due to various reasons. For the automated calculation, the parameters of the IO are therefore loaded from a file. For this purpose, a camera-specific orientation must be carried out before the first use of the app. Before the image alignment, the coded targets are recognized automatically, since these act as additional tie points and thus improve the orientation of the block. The coded targets also define the coordinate system of the bin. During the tests the reference positions of the coded targets were measured with a total station. For practical use a flappable mobile control point frame shall be used, Figure 11. The coded targets are automatically detected by PhotoScan with the detectMarkers function. For the alignment of images, the following settings are made: medium accuracy (matchPhotos), generic_preselection activated and reference_preselection deactivated. To speed up the calculations and to capture the waste bin without additional, surrounding objects adjacent to the waste bin, a perimeter for the calculations is defined. The perimeter is defined by the four control points in the upper corners of the bin, therefore it is tested beforehand whether these coded targets are already recognized. If the points are not recognized, the maximum perimeter calculated by PhotoScan is used.
The georeferencing of the coded targets is done in local coordinates. For this purpose, the measured coordinates of the control points are loaded. Afterwards the camera alignment is optimized within the scope of a bundle block alignment. This completes the orientation and the part of the volume calculation follows. Using the depth information of the images the dense point cloud is calculated. The settings are chosen as follows: medium quality and aggressive depth filter. The DEM (digital elevation model) is calculated from the dense point cloud, with the option "interpolation activated". The last processing step is the calculation of the fill volume. For this purpose, the base plane of the DEM is defined as the upper barrel plane, which is imported as a polygon in .shp format. The free waste binvolume is calculated from the height difference between base plane and DEM and is calculated as a function of the class vector data. The difference between the manufacturer's specification for the volume of the ton and the calculated free volume results in the fill volume of the bin. The fill volume determined in this way is written to the file volumen.csv, which is located in the project folder. In addition, the calculated fill volume is stored in JSON format in the file volumen.json on the node server to make it available to the app.

RESULTS
The system was extensively tested for its functionality, performance, reliability and possible sources of error. Thereby several tests were conducted to estimate the optimum number of images, the best image configuration and the influence of different filling levels.
From a photogrammetric perspective the first step is the definition of a reference frame. For the initial tests, this was determined with coded targets. Due to the elongated shape of the waste bin, the intersection angles of the rays from different perspectives become worse in the lower part of the bin. As the lower part of the bin is generally filled with waste, it is more important to distribute the GCP's in the upper section of the bin, see Figure 5. In order to get a first idea of the expected accuracies, the volume of an empty waste bin was determined and compared to the data sheet and a terrestrial laser scanner (TLS), see Table 2 and Figure  6.
In the point clouds, the upper wall and the bottom of the bin are reconstructed without gaps in both methods. The wall near the bottom, however, shows gaps in both methods. The height of the barrel corresponds to the lowest point of the DEM, because the barrel opening is defined by the xy-plane and the z-axis points from the barrel bottom to the barrel opening.
In comparison to the manufacturer's data, volume deviations up to 7% were obtained for the empty waste bin with the photogrammetric approach. The reason for these rather large deviations are mainly due to unfavourable viewing and intersection conditions at the sides of the barrel shaped waste bin. Reference measurements with a terrestrial laser scanner revealed slightly lower deviations of 4% to the manufacturer data. The results of the experiments showed that the system works.
With an optimal filling volume, where masking of control points by waste and barrel edges are minimized, an accurate volume estimation could be calculated even with a small number of images. By calculating the filling volume over the reconstructed waste surface, the results show a variation of the median values of the individual distributions at camera positions of up to 15 litres, due to the different visible image sections of the image sets.

Optimum number of images per waste bin
In order to determine the optimum number of images per scan 20 tests each with a different number of converging images were conducted on a ¾ full waste bin, Figure 7 and Figure 8.  Independently of the number of images, it is possible to compute a valid filling volume with the photogrammetric method. However, the calculated filling volumes differ, with the median values ranging between 172 litres and 186 litres. Figure 8 shows the results for the different number of the camera positions of the three-quarter filled yellow bin.

Reliability and accuracy
In order to determine the reliability of volume measurements at different filling levels (empty, half-full, ¾ full and full) 20 tests each were carried out for each filling level, using two configurations with 16 and 25 images respectively. The calculated volumes for different filling levels of the barrel and the distribution of the camera positions with 16 and 25 images are shown in Figure 9.  For the half-filled bin (Figure 9, top right), the variation of the computed volume is very small when using 25 images. In contrast, the volume values for the use of 16 images are very scattered. The median values for the calculated filling volumes are 105 litres (16 images) and 97 litres (25 images). The median values for the half-filled bin differ more than for the empty ton, with a difference of 8 litres.
The three-quarter filled bin (Figure 9, bottom left) reveals an even greater difference of 186 litres (16 pictures) vs. 172 litres (25 pictures). The variation between the tests was lower than the half-full bin.
Tests with the bin filled to just below the edge (Figure 9, bottom right), show a strong scattering and a skewed distribution of the values when using 16 images. Although the average values of the calculated fill volumes are within the desired range (240 liters), some calculations have not aligned enough images, leading towards false results. Tests with 25-image configuration always yielded reliable results.
To conclude, more images are always better, especially images from a nearly vertical position. A smaller number of images (16) may yield bad results under the following scenarios: 1. If the bin is only slightly filled and a large part of the waste is covered by the upper plane of the bin, which means that the fluctuation is large.
2. If the bin is filled to just below the rim, the uppermost control points become very important. If they are not recognized, the images cannot be aligned and therefore no volume may be calculated.

CONCLUSIONS AND OUTLOOK
The demonstrated approach yields good and reliable results, more or less independently of the type of material. The app works fine, thus providing accurate results with sufficient images. The photogrammetric workflow of the smartphone images using agisoft Photoscan works fully automatic, including volume determination, import and export of the data. However, the determination of the filling volume via the waste surface cannot provide results accurate to the litre, as cavities in the surface cannot be clearly assigned to the filled or free volume. Generally speaking, a compact and continuous waste surface in the bin will yield more accurate results than a spiky inhomogeneous waste surface.
For future commercial use, additional factors influencing the reliability and the performance must be investigated in further detail. These factors include the lighting conditions under which the images are taken, the necessary quality of the smartphone camera, the composition of waste and other bins. The last two points relate to the properties of the materials (colour, transparency) and how they can be photogrammetrically recorded. Furthermore, the app needs some improvements in a number of points with regard to its functionality, including the subsequent upload of image sets if no Internet is available. Furthermore, calculations of the fill level are not necessary for a full and an empty bin, for both cases the user should be able to manually confirm an empty or full waste bin with either one or none images. In addition, the extension of the system to other waste bins (other sizes, manufacturer, and colour) has to be realised and tested. For this purpose, the app requires selection options for the ton type, which are transferred to the subsequent elements of the system.
For a practical use in a commercial environment, the individual system elements must be adapted to the existing IT infrastructure. In addition, the mobile provision of control points must be implemented for the acquisition of local 3D coordinates. For this purpose, a mobile flappable reference frame with coded control points is necessary, which is placed into the waste container prior to the photogrammetric measurement, see Figure 11.