AERNets: Lightweight Event-Driven Networks of Biomimetic Image Sensors

Why emulate AER?

AER imagers are not easy to find, especially not in a commercial, off-the-shelf (COTS) manner. Therefore, to study AE imaging in a sensor network context, it is advantageous to employ COTS cameras to emulate AER imagers. Part of our research aims to create a detailed behavioral model for an AER imager architecture. Using this model we are in the process of developing a realistic AER emulator that will allow us to study different architectural alternatives before they are put into silicon. In order to promote the development of AER sensor specifications, and to solicit community feedback on AER architectures and applications, we plan to distribute our AER emulator to the community in two forms: 1) as a desktop tool and 2) as part of a wireless sensor node.  Validated design specifications are used to design and fabricate the CMOS AER imagers at E-Lab, led by Eugenio Culurciello at Yale.

Building an AER Emulator

The diagram below illustrates the data flow of the ENALAB AER Emulator software. An input from a COTS USB camera is filtered using traditional image processing techniques and transformed into a stream of events. This is then either saved to a text file, or used to reconstruct an image stream that can be displayed and saved as AVI.

Data flow diagram of AER emulation.

The program's interface allows customization of each step of the data flow above, as can be seen on the screenshot (below). The parameters for each stage is displayed in a column underneath the corresponding stage. Documentation will be available shortly.

Software screenshot.

Counting people with an AER motion imager

Detecting and counting people present in the field-of-view of a camera is a basic service that is necessary for a number of distinct applications, such as security, facility logging, assisted-living and home-care. As part of our research, we consider the problems person-detection and counting using motion-sensing AER imagers. For this, we developed a motion-histogram approach that detects moving objects within a given size range without the need to perform costly image operations, such segmentation. The advantages of this approach are:

• Fast, real-time operation &emdash; We have demonstrated our approach live on an Intel iMote2 sensor network (IPSN 2008), achieving a frame rate of 15fps with a 208MHz processor, all the while performing the typical sensor-networking tasks, such as clock synchronization and radio communication.
• Privacy preservation &emdash; Since the motion-histogram operates directly on pixel motion data, no pictures are ever used by our approach. This allows the use of motion-sensing AER cameras that are unable to take traditional pictures.
• Easy installation &emdash; Our approach requires no calibration, background modelling, or any learning period. Just place the nodes on the ceiling and you are done.

Each of our custom camera nodes (a) is placed on the ceiling, facing down (b). The nodes look for the locations where the motion patterns in the image best match the cuboid model of a person (c). For the image in (d), for example, the best match is shown in (e).	Left: The cuboids in the image map to bins in a histogram. People are detected at the histogram peaks. Right: Example histogram from a scene where two people have been detected.

Person-Identification in an AER network

In addition to finding the number of people in a scene and detecting their positions, many applications also require a history of all past detections of each person (i.e. tracking). However, obtaining coherent tracks based soly on the detected locations is an NP-hard problem, as ambiguities arise when people come close to one another. The solutions in the literature either make unrealistic assumptions regarding people's motion models, or utilize on-the-fly learning of visual models &emdash; requiring the person's image to be captured, which brings forth privacy concerns. In addition, these approaches fail in the presence of large gaps in camera coverage. In such cases, track histories can be recovered by only if each person is uniquely identified.

Rather than relying on visual or motion models, we propose tagging each person of interest with an accelerometer-carrying node of known ID. Then each tagged person is uniquely identified by matching their motion signature as captured by the accelerometer with that captured from the camera. This method has the advantage of maintaining our "no-pictures" privacy policy, as well respecting the anonymity of anyone not wearing an accelerometer node.

People carrying wearable nodes can be identified by matching motion or gait features from the wearable nodes with those detected with the camera network. The proposed identification method is capable to withstand gaps in coverage.	We detect motion and gait features from both cameras and accelerometers. Motion features are "turning" and "moving/stopping" while gait features are "heel-strike" and "midswing" instants of gait. A person in the video is identified when the track's motion signature (the timestamps of the triangle markers on the plot) matches that of some accelerometer in the scene.
The figure shows a box-plot of the expeted performance of the gait-matching person-identification algorithm as the number of people in the field-of-view increases from 2 to 10. For each n-person scenario, 50 simulations were performed and the mean is shown in red.	Performance of the person identification approach using experimentally-acquired data. 24 experimental traces were shuffled to produce scenarios from 2 to 10 people. The accuracy never goes below 83%, following closely our expectation from the previous figure.
Video of 3-person experiments. One person (with the red jacket) carries an accelerometer node, and two others do not. Through gait-matching, the system is able to correctly identify the accelerometer-carrying one over 87.5% of the times.