AN ULTRASONIC SENSOR FOR HUMAN PRESENCE DETECTION TO ASSIST RESCUE WORK IN LARGE BUILDINGS

When the fire brigade arrives at a burning building, it is of vital importance that people who are still inside can quickly be found. Smart buildings should be able to expose this location data to the fire brigade working in a smart city. In this paper the feasibility is researched of using ultrasonic sound sensors for human presence detection in smoke-filled spaces. This type of sensor could assist the fire brigade when evacuating a large building by directing them to the places where their help is most needed. The advantage of ultrasonic sound over other sensors or cameras is that its signal is able to pierce through smoke, does not require badges or other wearable devices and introduces little privacy and security risks. In addition, ultrasonic sensors are very inexpensive making it possible to equip every room of a building with an ultrasonic presence detector. In this research both a preliminary ultrasound measuring device and signal processing algorithm have been designed. Testing results show that the walking movement of a person in an indoor area can be detected with the combination of the sensor and the algorithms. In addition, tests of the signal strength in smoke have shown that ultrasound is capable of “looking through” the smoke. The algorithm based on a particle filter allows for more information to be extracted from the relatively simple sensor signal by detecting human walking movement specifically and opens up the way for an ultrasound based indoor positioning system that can be used in emergency situations.


INTRODUCTION
In large building fires, an important factor that determines the strategy of the fire brigade is the presence of people inside the building.Choices like where to start extinguishing and whether to risk going into the building to search for people depend on this knowledge.Usually, the only source of information there is comes from incomplete sources such as statements from the people who fled the building or lists of employees.The largest portion of fatalities in fire emergency situations is not caused by a direct result of the heat, but rather poisoning by the smoke generated by the fire (Stec, 2017).People suffering from the results of smoke are often unconscious, but could still be saved if the firemen can reach them in time.For instance, carbon monoxide, the most common poisonous gas in smoke from an indoor fire, may incapacitate a person within 1 or 2 minutes at an air concentration of 6400 ppm, but actual death from carbon monoxide may not occur for up to 20 minutes (Goldstein, 2008).During this time, quickly finding people is of vital importance.If an overview of the building, indicating where people are, could be given so that rescue actions can be quickly and efficiently coordinated then this could potentially save lives.
The digital layer of a smart city would provide all kinds of information that the fire brigade may need for disaster management.Location data is intrinsically part of this information.In the data of a smart city, there is no difference between inside or outside location.This is a continuous environment with one grid reference.However, the underlying sensor technology will be different.The options for indoor positioning are quite varied.RFID badges (Ortakci et al., 2015), cameras (Sun et al., 2016) or even Bluetooth (Lee et al., 2017) can all provide positioning information.However, these sensors either do not work when there is smoke or require people to wear something, making them prone to failure when their information is most required and possibly privacy sensitive.In this paper a new sensor based on ultrasound is proposed that may be able to solve the indoor positioning problems, provide the smart city with the required positioning data in ordinary circumstances and in addition give the fire brigade an edge in disaster management when a fire occurs.
An important quality of ultrasound is that it is not obstructed by smoke, retaining its localisation ability for a longer time during a fire.It has been shown that it is possible to create a selfconfiguring network of ultrasonic sensors that can track the location of a target with an average error of less than 18 cm (Basaran et al., 2014).Since the sensors work on the basis of echolocation, the people inside the building do not need to do anything in order to be tracked.It has been shown that it is also possible to use mobile devices for ultrasonic localisation (Filonenko et al., 2013), however, in an emergency situation it is a great advantage that the localisation responsibility can be transferred from the person to the smart building itself.In addition, this type of echolocation has a smaller impact on peoples privacy because they never need to provide their identity to the system and no images of any person could ever obtained by the ultrasonic sensor.
The fact that people are not required to identify themselves in any way before the system can track them poses the challenge to prevent tracking non-human objects such as doors, chairs and possibly animals.This can be done by analysing the specific type of movement in more detail.Any moving object has a character-ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume IV-4/W7, 2018 3rd International Conference on Smart Data and Smart Cities, 4-5 October 2018, Delft, The Netherlands istic movement signature in the ultrasonic signal.This signature is caused by the separately moving parts of the object and is often called the micro-Doppler signature (Kalgaonkar and Raj, 2007).
Recognising human movement from this feature is the research that has been performed in this paper.The scope is to show that enough data can be extracted from the signal of a single ultrasonic sensor to characterise human walking movement.To establish trust in the smoke penetrating capabilities of the ultrasonic sensor and in order to provide a starting point for future research, the proposed setup has also been tested in smoke conditions.

SONAR
Sonar is mostly used under water.Under water, sound carries much further and is therefore a good alternative for radar, which does not work well in water.In air, using sonar is still possible but its range is reduced to the order of 10 meters (Kalgaonkar and Raj, 2007).For indoor applications this is still quite usable.
In this research a continuous sound wave is used.A continuous wave does not allow distance measurement, but since this research is focussed on recognising movement of people, this is not a problem and the same hardware could be used to do a pulsed measurement in order to allow distance measurements as well.
An important advantage of sonar is that the hardware is very inexpensive since it basically consists of just a speaker and a microphone.The device that has been built for this research consists of an ultrasonic receiver and transmitter accounting for a retail price of approximately 6 euros each and two audio amplifiers of about 1 euro per amplifier.Combined with some resistors and capacitors, the total of all the components is less than 20 euros, excluding the hardware for signal processing, which is currently performed on a laptop.
The transmitter of the ultrasonic sensor sends out a sound of constant frequency and the receiver registers the echo of this sound from the environment.There are two types of information conveyed in this signal.First of all, any moving objects that reflect the sound wave will change the frequency of the reflected wave due to the Doppler effect.Secondly, the time it takes a pulse of sound to reach an object and return to the sensor can be used to calculate the distance to a certain object.It should be noted that the signal that an ultrasonic sensor provides is one dimensional (mono audio) and therefore the reflected signal of all objects in a room will be summed into this one dimensional signal.The signal processing therefore becomes a very important task.This research is focussed on creating an algorithm that is able to process all the little frequency changes in an ultrasonic signal and to use a model based pattern recognition approach to recognise human activity.The distance measurement is left out in this research but could be incorporated in further research to localise the activity.
In order to detect movement from the sonar signal, the Doppler effect is analysed in some more detail.The frequency of the echo of a walking person changes due to the Doppler effect.This happens at all of the separately moving body parts because the moving parts either elongate or compress a sound wave depending on the direction of their movement.This changes the frequency of the returned signal.The velocity v of a body part with respect to the sensor can be calculated from the returned frequency fr and the sent frequency fs as where v sound is the velocity of sound in air.Since fs and v sound are constants, the velocity of an object is a linear function of the frequency change it causes.
The most common way of analysing what frequencies are in a certain signal is by using the Fourier transform.The Fourier transform is a data transformation that takes a time -amplitude signal and converts it to a frequency -amplitude signal so that it can be seen how much each frequency component is present in the signal.By equation ( 1) the presence of frequencies in the signal is directly related to the velocities that were in the measured scene.Unfortunately the Fourier transformation loses the time component of the data so that it is not possible to deduce how the velocities in the scene change over time.A way to keep this dynamic information in the data is to apply the Fourier transform to small pieces of signal.This way frequency and therefore velocity information is made available for every interval of signal while it is still possible to see how this information changes over time by comparing the subsequent intervals.This method, often called the short time Fourier transform, results in a three dimensional graph with time, frequency and amplitude on the axes.This graph is referred to as the spectrogram of the signal.
An example of a spectrogram from the signal that was reflected off a walking person can be found in figure 1.The time and frequency are on the x and y axes and the amplitude is represented by the brightness level (brighter means higher amplitude).The linear relationship between frequency and velocity allows for the creation of a secondary y-axis giving the velocity corresponding to each frequency.Different velocities that correspond to moving body parts can be observed in the spectrogram.The line at zero velocity is always present due to reflections from the static surroundings.
Figure 1.This figure shows the spectrogram of a walking person.Brighter colours correspond to higher presence of the frequencies.
Together all the small movements and their associated frequency changes cause a so called Doppler signature in the spectrogram.Based on this it is possible to discriminate between types of movement based on the ultrasonic signal.Previously, this technique has been used for classifying gait (Kalgaonkar and Raj, 2007).In (Groot et al., 2013) a particle filter has been applied in combination with the Thalmann model (Boulic et al., 1990) that describes human walking for classifying Doppler signatures from radar data.In this paper the same approach will be used, but focussed on recognising human presence in an emergency situation.

THE MOVEMENT MODEL
In order to recognise a walking person from the signal, the human walking motion has to be characterised in terms of the body part velocities.Using statistical methods, the movement model can then be matched to the spectrogram of the measured signal.The model that was used for walking people is the Thalmann model (Boulic et al., 1990).This model has been built up empirically from measuring the body parts of many people while they were walking.It describes the angles that the joints make in relation to each other over time.The model takes the person height and walking velocity as parameters.Using the angles between the joints, it is possible to calculate the (x, y, z) coordinates of the body parts and then calculate their velocity relative to the sensor.
Figure 2. In this visualisation of the Thalmann model, the various body parts that are modelled can be seen, indicated by different colors.
Figure 3.The velocities of some (not all) body parts can be seen here.The mean velocity is about 4 km/h and that the ankles reach the largest velocities.

THE MEASUREMENT MODEL
Using the Thalmann movement model, the system that is observed (walking persons) can be characterised.In order to compare this to the signal that the ultrasonic sensor receives, the measurements themselves also have to be modelled.As shown in equation 1, the velocities that are calculated from the Thalmann model (such as shown in figure 3) are directly related to the frequencies that are present in the signal.Indeed, figure 3 is already bearing some resemblance to the spectrogram in figure 1.However, it is not quite the same as the spectrogram yet.In practice the velocities will be spread across neighbouring frequencies because of variations within a time frame of the spectrogram and because of the characteristics of the discrete Fourier transform itself (Harris, 1978).The velocities are therefore spread in the model using a Gaussian spreading function.
Secondly, the strength of the reflection by a person depends on the distance that this person is away from the sensor and on the reflecting surface area.Sound spreads spherically and therefore the sound intensity per square meter decreases with the same rate with which the surface of a sphere increases.This is proportional to the distance squared.In addition, the air also absorbs part of the energy adding an exponential decay factor to the equation.In figure 1 this effect can also be observed as the signal becomes weaker while the person is walking away from the sensor.The resulting sound intensity I(d) in watt per square meter that is received from an object with a reflective surface of A square meters at distance d meters is proportional to where I0 in watt is determined by the sound intensity at the sound source.
Finally, there will be noise in the measurements.The noise in the spectrogram can be described by a scaled non central chi square distribution (Muirhead, 1982).Using the probability distribution of the noise, it is possible to assert how likely it is that an observed difference between the model and the measurement is due to a model misfit or due to noise.

FILTERING
In order to make real time estimates of the number of people, it is necessary to process the signal as it arrives.A particle filter was chosen to do this.Particle filters are often used for tracking applications and their advantage is that it is easy to incorporate any model of choice describing the movement of the object that is being tracked (Doucet et al., 2000).That means that the model for walking that has been set up in this research can easily be extended to allow for more activities to be detected.
At each time step the particle filter predicts, using the movement model, what the next measurement will be.It does this in a stochastic way by randomly generating many variations (called particles in this context) on what the next measurement could be like.When the measurement then becomes available, the particles are assigned weights that correspond to how well they fit the measurement.Before the next prediction step, the particles are then resampled from the current set of particles with each particle having a chance of being drawn equal to the weight it was assigned in the previous step.By performing the resampling step, particles that fit the measurement very well will be multiplied and ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume IV-4/W7, 2018 3rd International Conference on Smart Data and Smart Cities, 4-5 October 2018, Delft, The Netherlands particles that were far off from the measurement will be extinct.
After the resampling step, the particles are again evolved to the next future measurement and the algorithm is continued in the same way.
The parameters that are the input to the model that generates the theoretical signal from the Thalmann movement model are: • Person height • Walking velocity • Distance to the sensor The walking phase is a number between 0 and 1 which describes how far a person in in a walking cycle.A walking cycle consists of two steps, one with the left and one with the right foot.After this the legs are again in the position in which they started.

RESULTS
In the first experiment, the sensor was placed in an office hallway were people walked by.The sensor was placed on shoulder hight and a continuous measurement was performed.The particle filter algorithm was then used to detect when there was a person walking through the hallway.
The second experiment was performed on on a training location of the fire brigade at the Spinel Safety Center in Dordrecht, The Netherlands.At this training facility, a house containing some common furniture like a couch and some closets and a small kitchen was filled with glycerine based smoke, so dense that it was not possible to distinguish objects from even half a meter distance.The sensor was placed inside the house next to its entrance.Figure 4 shows this room.
In both cases, a continuous ultrasound tone of 41KHz was emitted by the sensor and its echo was recorded with a sampling frequency of 96KHz.The recorded sound was saved as a wav file and processed afterwards using the algorithm described in the Figure 4.A picture of the inside of the testing house before it was filled with smoke.The picture is taken from the entrance.previous section.The results of the first and second experiment can be seen in figures 5 and 6, respectively.In these figures the upper graph shows the calculated likelihood of a person being present, the middle graph shows the measured spectrogram and the lower graph shows the simulated measurement that best fits the measured data.
The measurement that was recorded in a normal office environment can be found in figure 5.The characteristic Doppler signature of a walking person can be clearly distinguished in the measurement and it can also be seen that the simulated spectrogram follows the real signal quite well.The likelihood of a person being present is fairly accurate, being close to 1 when the signal shows that there is a person and 0 when there is only noise.It can be seen that the system has more difficulty recognising that there is a person when the person is walking towards the sensor (negative velocity) than when the person is walking away from the sensor (positive velocity).In the first case the person is detected at a 4 meter distance while in the second case, the person can be tracked up to 10 meters away from the sensor.This is because when a person walks towards the sensor from far away, the signal starts out very weakly and because the system cannot see what the signal will be like in the future, the very small signal can still be reasonably explained as being due to noise.When the person starts walking away while being close to the sensor, the system knows that there has been a person before and it is therefore reasonable to believe that this same person is still causing the signal.It is possible to increase the sensitivity of the system by increasing the a priori likelihood of a person being present, but this would inevitably increase the number of false positives.
During the smoke measurement, the signal strength in the room was measured with and without smoke when the room was empty.This measurement showed that the difference in signal strength was within the standard deviation of the signal, and therefore insignificant, showing that the sensor can measure equally well in the glycerine based smoke as in normal conditions.Since the sensor measurements themselves are not affected, the spectrograms measured in smoke and in normal visibility are in principle the same, as long as the scene is the same.It is therefore interesting to look at the behaviour of a person walking without any visibility in order to understand how the scene might change.When looking at the spectrogram that was measured on a person walking in the smoke filled room in figure 6, it is much harder to recognise the characteristic Thalmann Doppler signature.Starting from 12 seconds it can be found, but still at a much lower velocity level than in normal conditions.Since the sensor itself is not affected by the smoke, the only conclusion that can be drawn from this difference, is that the person himself is behaving differently.This conclusion was indeed confirmed by the person on whom the measurement was performed.When walking in the smoke filled room, the test subject was not able to see anything and therefore very withholding while walking, out of fear of bumping into something.This can especially be seen in the signal before 10 seconds where the person first walked towards the sensor.After this the person knew the room a little bit better and was able to walk back in his normal way, but still much more slowly than he would normally walk.
Looking at the presence likelihood it can be observed that despite the different movement signature, the person is still detected, although with less certainty.When looking at the simulated spectrogram it can be seen that the algorithm cannot follow the signal very well and therefore is not very similar to the measured signal.The reason for this is simply this kind of movement was not in the model and therefore cannot represent the signal in a good way.If a good movement model for blind path finding was available then it would be possible to also detect this kind of movement and the particle filter could even be programmed to detect whether someone is exhibiting ordinary walking movement or not.However, since the movement characteristic of a person walking through smoke (possibly in panic) is inherently less certain, it might be better to recognise human behaviour at times when the situation is still safe and after that keep tracking the person using an ultrasonic trilateration solution even when the movement signature of this person changes.

CONCLUSIONS
By combining several simple electrical components a sensor was built that can measure movement with enough accuracy to be able to distinguish the characteristic movement signature of a walking person when analysed using its spectrogram.Moreover, this sensor was tested in smoke conditions and was still able to generate the same quality of measurements, making it suitable for use in a disaster management system for the fire brigade if extended with a way to remember peoples locations after they have fainted.It was possible to build this sensor, excluding signal processing hardware, for under 20 euros, potentially making it a very cost effective solution.
Using a model based approach incorporating a particle filter, an algorithm was proposed that can automatically assert the likelihood that a given signal is due to a walking person.This likelihood was very reliable in an office environment, especially when people were within 4 meters from the sensor.At this distance the system was always able to detect a person.From larger distances the weaker signal and the uncertainty this introduced resulted in people being detected up to 10 meters when they had been detected close by before, but not when they walked towards the sensor from far away.
By testing the signal processing algorithm in smoke conditions as well, it has been found that even though the ultrasonic sensor signal itself is unaffected by the smoke, the changed behaviour due to the blinding effects of smoke causes people to move very differently.The system was still able to detect the person eventually, but gave much less certainty.
The fact that the sensor works as expected, even when there is no ordinary visibility due to smoke, combined with the fact that the proposed algorithm is able to detect people in the case for which it was currently designed shows that it will be a promising effort to improve the set up and extend the algorithms before productising the system.

FUTURE WORK
Even though the initial results show that a system based on ultrasonic sensing can recognise human motion and is able to make measurements in smoke, there is still a lot of work to be done in order to extend the current work into a full indoor positioning solution that can assist the fire brigade in rescuing people from a burning building.
First of all, there is the problem of people behaving differently when there is a lot of smoke in a room.In the experiments performed in this research, the behaviour changed only because of the lack of sight, but in a real emergency situation, the poisonous smoke is also likely to make people faint.A possible solution for this would be to track people even before an emergency situation.This way the system could recognise that the movement was due to a human being before, and then keep track of the location at which new movement occurs.When someone does not move any more, but did not move out of range either, it is very likely that the person is still in the same place and the fire brigade should be alerted.
In order to apply the above tracking logic to a system, it would be very helpful to apply distance measurement in combination with movement measurement.For this, the hardware does not have to be changed, only the signal would have to be switched from a continuous signal to a pulsed signal and the processing algorithm adjusted accordingly.When the distance measurement of multiple sensors in a room are combined, it would be possible to calculate the location at which movement occurred.Also, the current algorithm for the detection of human motion could be extended to be able to count the number of people moving.This would give the system more accuracy when there are multiple people walking close together and individual localisation is more difficult.
Using more sensors in a network would be necessary in any case in order to cover larger rooms and to keep track of a full building.
There is quite some logic involved in receiving all the information and processing it.Ideally the sensors themselves would process a part of the data on site so that the infrastructure is not required to transfer the raw sensor signal data.In order to process the data on site, the current algorithm would need to be improved in terms of speed.The Thalmann model that has been used in this research provides a very detailed movement description, but for the purpose it is used in it would also be sufficient to have a simpler approximation of the movement model that would be faster to evaluate.A possibility for easily building movement models for many types of movement could be using a 3D pointcloud sensor as described in (Murray et al., 2018).
Finally, the current sensor is merely a prototype to be able to test with.In order to make it suitable as a real product, it would require a more robust package that can withstand high temperatures and it should be easy to connect into the huge sensor network of a smart city.
Figure 5. Particle filter results on the signal of a person walking away from the sensor and then back again.

Figure 6 .
Figure 6.Particle filter results on the measurement of a person trying to find his way through glycerine based smoke, first away from the sensor and then back.