It is well known that almost all electronic devices unintentionally emit unique spurious RF signals when in operation. By using an SDR like an RTL-SDR to record the spectra from electronic devices, it's possible to build up a database of known emissions. We can then detect when an electronic device is active by comparing the live spectrum to spectra stored in the database.
In a previous post we covered Disney's EM sense which is an experimental smart watch that automatically detects what electronic device the wearer is touching. With EM Sense they use an RTL-SDR and a database of raw pre-recorded spectrum data. To detect what the wearer is touching the live signal from the RTL-SDR is correlated against the database, and the closest match is returned.
José's script does something very similar, however instead of correlating with raw spectrum data he instead uses the waterfall image that is generated by rtl_power and heatmap.py. The rtl_power program allows an RTL-SDR to scan the frequency spectrum over a wider bandwidth by rapidly scanning ~2.4 MHz chunks of bandwidth at different frequencies. Heatmap.py is a program that turns the scanned data from rtl_power into a heatmap image of the spectrum.
To add an entry to the database, the electronic device is placed 7-8 centimeters away from the RTL-SDR, and a heatmap image recorded between 24 - 921 MHz is saved to disk. This can be repeated for multiple electronic devices. Each image will record the spurious signals from the electronic device, resulting in a unique heatmap image per electronic device.
Once the database has been created, you can then place any of the devices found in the database next to the RTL-SDR, and record a heatmap for 20-30s. That heatmap will then be compared against the images in the database using imagemagick which is an image analysis and manipulation library. The electronic device associated with the closest matching image in the database will be returned.
In his experiments he tested various electronic devices like an iPhone and was able to successfully determine when it was nearby.
Back in November 2015 we posted about Disney Research’s EM-Sense which was an RTL-SDR based smart watch that was able to actually sense and detect the exact (electronic) object the wearer was touching. It worked by using the RTL-SDR to detect the specific electromagnetic emission signature given off by various different electronic devices.
The Disney research team have put forward the idea that a low cost SDR like the RTL-SDR can be used in place of RFID tags when they would have been used to identify electronic devices. The idea is that the SDR can be used to read the electromagnetic emissions of the electronic device, which can then be used to identify the item, thus eliminating the need for an RFID tag or barcode. Their abstract reads:
Radio Frequency Identification technology has greatly improved asset management and inventory tracking. However, for many applications RFID tags are considered too expensive compared to the alternative of a printed bar code, which has hampered widespread adoption of RFID technology.
To overcome this price barrier, our work leverages the unique electromagnetic emissions generated by nearly all electronic and electromechanical devices as a means to individually identify them. This tag-less method of radio frequency identification leverages previous work showing that it is possible to classify objects by type (i.e. phone vs. TV vs. kitchen appliance, etc). A core question is whether or not the electromagnetic emissions from a given model of device, is sufficiently unique to robustly distinguish it from its peers.
We present a low cost method for extracting the EM-ID from a device along with a new classification and ranking algorithm that is capable of identifying minute differences in the EM signatures. Results show that devices as divers as electronic toys, cellphones and laptops can all be individually identified with an accuracy between 72% and 100% depending on device type.
While not all electronics are unique enough for individual identifying, we present a probability estimation model that accurately predicts the performance of identifying a given device out of a population of both similar and dissimilar devices. Ultimately, EM-ID provides a zero cost method of uniquely identifying, potentially billions of electronic devices using their unique electromagnetic emissions.
In the paper we can see that the EM-ID hardware is essentially just a direct sampling modified RTL-SDR and antenna. The RTL-SDR is modified to use direct sampling as this allows it to receive 0 – 28 MHz, and thus 0 – 500 kHz where the most useful EM emissions exist. The system process is to basically scan the device using the antenna and RTL-SDR, extract features such as power peaks from the recorded EMI spectrum and then turn this data into a device signature which can then be used to compare against a database of previously recorded and known device signatures. (e.g. light bulb, iPhone).
The prototype watch does this by using the RTL-SDR to detect the electromagnetic (EM) noise emitted by particular objects and compare it against a stored database. They call this technology EM-Sense. In the paper the authors summarize:
Most everyday electrical and electromechanical objects emit small amounts of electromagnetic (EM) noise during regular operation. When a user makes physical contact with such an object, this EM signal propagates through the user, owing to the conductivity of the human body. By modifying a small, low-cost, software-defined radio, we can detect and classify these signals in real-time, enabling robust on-touch object detection. Unlike prior work, our approach requires no instrumentation of objects or the environment; our sensor is self-contained and can be worn unobtrusively on the body. We call our technique EM-Sense and built a proof-of concept smartwatch implementation. Our studies show that discrimination between dozens of objects is feasible, independent of wearer, time and local environment.
The frequencies required for EM detection are around 0 - 1 MHz which falls outside the range of the RTL-SDR's lowest frequency of 24 MHz. To get around this, they ran the RTL-SDR in direct sampling mode. The RTL-SDR is connected to the watch, but a Nexus 5 smartphone is used to handle the USB processing which streams the signal data over WiFi to a laptop that handles the signal processing and live classification. In the future they hope to use a more advanced SDR solution, but the RTL-SDR has given them the proof of concept needed at a very low cost.
An example use scenario of the watch that Disney suggests is as follows:
Home – At home, Julia wakes up and gets ready for another productive day at work. Her EM-Sense-capable smartwatch informs and augments her activities throughout the day. For instance, when Julia grabs her electric toothbrush, EMSense automatically starts a timer. When she steps on a scale, a scrollable history of her weight is displayed on her smartwatch automatically. Down in the kitchen, EM-Sense detects patterns of appliance touches, such as the refrigerator and the stove. From this and the time of day, EM-Sense infers that Julia is cooking breakfast and fetches the morning news, which can be played from her smartwatch.
Fixed Structures – When Julia arrives at the office, EMSense detects when she grasps the handle of her office door. She is then notified about imminent calendar events and waiting messages: "You have 12 messages and a meeting in 8 minutes". Julia then leaves a reminder – tagged to the door handle – to be played at the end of the day: “Don’t forget to pick up milk on the way home.”
Workshop – In the workshop, EM-Sense assists Julia in her fabrication project. First, Julia checks the remaining time of a 3D print by touching anywhere on the print bed – “five minutes left” – perfect timing to finish a complementary wood base. Next, Julia uses a Dremel to cut a piece of wood. EM Sense detects the tool and displays its rotatory speed on the smartwatch screen. If it knows the task, it can even recommend the ideal speed. Similarly, as Julia uses other tools in the workshop, a tutorial displayed on the smartwatch automatically advances. Finally, the 3D print is done and the finished pieces are fitted together.
Office – Back at her desk, Julia continues work on her laptop. By simply touching the trackpad, EM-Sense automatically authenticates Julia without needing a password. Later in the day, Julia meets with a colleague to work on a collaborative task. They use a large multitouch screen to brainstorm ideas. Their EM-Sense-capable smartwatches make it possible to know when each user makes contact with the screen. This information is then transmitted to the large touchscreen, allowing it to differentiate their touch inputs. With this, both Julia and her colleague can use distinct tools (e.g., pens with different colors); their smartwatches provide personal color selection, tools, and settings.
Transportation – At the end of the day, Julia closes her office door and the reminder she left earlier is played back: “Don’t forget to pick up milk on the way home.” In the parking lot, Julia starts her motorcycle. EM-Sense detects her mode of transportation automatically (e.g., bus, car, bicycle) and provides her with a route overview: “You are 10 minutes from home, with light traffic”.
EM-Sense: Touch Recognition of Uninstrumented Electrical and Electromechanical Objects