This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Efficient Feedforward Categorization of Objects and Human Postures with Address-Event Image Sensors
February 2012 (vol. 34 no. 2)
pp. 302-314
Shoushun Chen, Sch. of Electr. & Electron. Eng. (EEE), Nanyang Technol. Univ., Singapore, Singapore
P. Akselrod, Electr. Eng. Dept., Yale Univ., New Haven, CT, USA
Bo Zhao, Sch. of Electr. & Electron. Eng. (EEE), Nanyang Technol. Univ., Singapore, Singapore
J. A. Perez-Carrasco, Inst. Microelectron. Sevilla (IMSE), Sevilla, Spain
B. Linares-Barranco, Inst. Microelectron. Sevilla (IMSE), Sevilla, Spain
E. Culurciello, Electr. Eng. Dept., Yale Univ., New Haven, CT, USA
This paper proposes an algorithm for feedforward categorization of objects and, in particular, human postures in real-time video sequences from address-event temporal-difference image sensors. The system employs an innovative combination of event-based hardware and bio-inspired software architecture. An event-based temporal difference image sensor is used to provide input video sequences, while a software module extracts size and position invariant line features inspired by models of the primate visual cortex. The detected line features are organized into vectorial segments. After feature extraction, a modified line segment Hausdorff-distance classifier combined with on-the-fly cluster-based size and position invariant categorization. The system can achieve about 90 percent average success rate in the categorization of human postures, while using only a small number of training samples. Compared to state-of-the-art bio-inspired categorization methods, the proposed algorithm requires less hardware resource, reduces the computation complexity by at least five times, and is an ideal candidate for hardware implementation with event-based circuits.

[1] R. Van Rullen and S. Thorpe, "Rate Coding versus Temporal Order Coding: What the Retinal Ganglion Cells Tell the Visual Cortex," Neural Computation, vol. 13, pp. 1255-1283, 2001.
[2] S. Thorpe, A. Delorme, R. Van Rullen, and W. Paquier, "Reverse Engineering of the Visual System Using Networks of Spiking Neurons," Proc. IEEE Int'l Symp. Circuits and Systems, vol. 4, pp. 405-408, 2000.
[3] C. Koch, The Quest for Consciousness: A Neurobiological Approach. Roberts and Company Publishers, 2004.
[4] T. Serre, "Learning a Dictionary of Shape-Components in Visual Cortex: Comparison with Neurons, Humans and Machines," PhD dissertation, MIT, Apr. 2006.
[5] T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, and T. Poggio, "Robust Object Recognition with Cortex-Like Mechanisms," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp. 411-426, Mar. 2007.
[6] J.J. Peissig and M.J. Tarr, "Visual Object Recognition: Do We Know More Now than We Did 20 Years Ago?" Ann. Rev. Psychology, vol. 58, pp. 75-96, 2007.
[7] J.K. Tsotsos, A.J. Rodriguez-Sanchez, A.L. Rothenstein, and E. Simine, "Different Binding Strategies for the Different Stages of Visual Recognition," Proc. Second Int'l Conf. Advances in Brain, Vision and Artificial Intelligence, pp. 150-160, 2007.
[8] S. Chikkerur, T. Serre, C. Tan, and T. Poggio, "What and Where: A Bayesian Inference Theory of Attention," Vision Research, vol. 50, no. 22, pp. 2233-2247, 2010.
[9] R. Serrano-Gotarredona, T. Serrano-Gotarredona, A. Acosta-Jimenez, and B. Linares-Barranco, "A Neuromorphic Cortical-Layer Microchip for Spike-Based Event Processing Vision Systems," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 53, no. 12, pp. 2548-2566, Dec. 2006.
[10] L. Camunas-Mesa, A. Acosta-Jimenez, C. Zamarreno-Ramos, T. Serrano-Gotarredona, and B. Linares-Barranco, "A $32\times32$ Pixel Convolution Processor Chip for Address Event Vision Sensors with 155ns Event Latency and 20Meps Throughput," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 58, no. 4, pp. 777-790, Apr. 2011.
[11] T. Teixeira, A. Andreou, and E. Culurciello, "An Address-Event Image Sensor Network," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 4467-4470, May 2006.
[12] E. Culurciello and A. Savvides, "Address-Event Image Sensor Network," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 955-958, May 2006.
[13] E. Culurciello and A. Andreou, "CMOS Image Sensors for Sensor Networks," Analog Integrated Circuits and Signal Processing, vol. 49, no. 1, pp. 39-51, 2006.
[14] T. Teixeira, E. Culurciello, E. Park, D. Lymberopoulos, A. Barton-Sweeney, and A. Savvides, "Address-Event Imagers for Sensor Networks: Evaluation and Modeling," Proc. Information Processing in Sensor Networks, pp. 458-466, Apr. 2006.
[15] Z.M. Fu and E. Culurciello, "A 1.2mw CMOS Temporal-Difference Image Sensor for Sensor Networks," Proc. IEEE Int'l Symp. Circuits and Systems, May 2008.
[16] S. Pellegrini and L. Iocchi, "Human Posture Tracking and Classification through Stereo Vision and 3D Model Matching," J. Image and Video Processing, vol. 2008, pp. 7:1-7:12, 2008.
[17] C. Wu, H. Aghajan, and R. Kleihorst, "Real-Time Human Posture Reconstruction in Wireless Smart Camera Networks," Proc. Int'l Conf. Information Processing in Sensor Networks, pp. 321-331, 2008.
[18] R. Jafari, W. Li, R. Bajcsy, S. Glaser, and S. Sastry, "Physical Activity Monitoring for Assisted Living at Home," Proc. Int'l Workshop Wearable and Implantable Body Sensor Networks, Mar. 2007.
[19] Z. Fu and E. Culurciello, "Fall Detection Using an Address-Event Temporal Contrast Vision Sensor," Proc. IEEE Int'l Symp. Circuits and Systems, May 2008.
[20] G. Virone, M. Alwan, S. Dalal, S.W. Kell, B. Turner, J.A. Stankovic, and R. Felde, "Behavioral Patterns of Older Adults in Assisted Living," IEEE Trans. Information Technology in Biomedicine, vol. 12, no. 3, pp. 387-398, May 2008.
[21] H. Su and F.-G. Huang, "Human Gait Recognition Based on Motion Analysis," Proc. Int'l Conf. Machine Learning and Cybernetics, vol. 7, pp. 464-468, Aug. 2005.
[22] B. Boulay, F. Bremond, and M. Thonnat, "Posture Recognition with a 3D Human Model," Proc. IEEE Int'l Symp. Imaging for Crime Detection and Prevention, pp. 135-138, June 2005.
[23] J. Isaacs and S. Foo, "Hand Pose Estimation for American Sign Language Recognition," Proc. 36th Southeastern Symp. System Theory, pp. 132-136, 2004.
[24] K. Takahashi, T. Sakaguchi, and J. Ohya, "Remarks on Real-Time Human Posture Estimation from Silhouette Image Using Neural Network," Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, pp. 370-375, Oct. 2004.
[25] E. H-Jaraha, C. Urunuela, and J. Senar, "Detected Motion Classification with a Doublebackground and a Neighborhood-Based Difference," Pattern Recognition Letters, vol. 24, pp. 2079-2092, 2003.
[26] L.H.W. Aloysius, G. Dong, H. Zhiyong, and T. Tan, "Human Posture Recognition in Video Sequence Using Pseudo 2-D Hidden Markov Models," Proc. Eighth Control, Automation, Robotics, and Vision Conf., pp. 712-716, Dec. 2004.
[27] P. Spagnolo, M. Leo, A. Leone, G. Attolico, and A. Distante, "Posture Estimation in Visual Surveillance of Archaeological Sites," Proc. IEEE Conf. Advanced Video and Signal Based Surveillance, pp. 277-283, July 2003.
[28] J. Triesch and C. von der Malsburg, "A System for Person-Independent Hand Posture Recognition against Complex Backgrounds," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 12, pp. 1449-1453, Dec. 2001.
[29] M. Gottardi, N. Massari, and S. Jawed, "A 100µw $128 \times 64$ Pixels Contrast-Based Asynchronous Binary Vision Sensor for Sensor Networks Applications," IEEE J. Solid-State Circuits, vol. 44, no. 5, pp. 1582-1592, May 2009.
[30] E. Artyomov and O. Yadid-Pecht, "Adaptive Multiple-Resolution CMOS Active Pixel Sensor," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 53, no. 10, pp. 2178-2186, 2006.
[31] A. Teman, S. Fisher, L. Sudakov, A. Fish, and O. Yadid-Pecht, "Autonomous CMOS Image Sensor for Real Time Target Detection and Tracking," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 2138-2141, May 2008.
[32] A. Fish, L. Sudakov-Boresha, and O. Yadid-Pecht, "Low-Power Tracking Image Sensor Based on Biological Models of Attention," Int'l J. Information Theory and Applications, vol. 14, no. 2, pp. 103-114, 2006.
[33] S. Mizuno, K. Fujita, H. Yamamoto, N. Mukozaka, and H. Toyoda, "A $256 \times 256$ Compact Cmos Image Sensor with On-Chip Motion Detection Function," IEEE J. Solid-State Circuits, vol. 38, no. 6, pp. 1072-1075, June 2003.
[34] J. Choi, S.-W. Han, S.-J. Kim, S.-I. Chang, and E. Yoon, "A Spatial-Temporal Multiresolution CMOS Image Sensor with Adaptive Frame Rates for Tracking the Moving Objects in Region-of-Interest and Suppressing Motion Blur," IEEE J. Solid-State Circuits, vol. 42, no. 12, pp. 2978-2989, Dec. 2007.
[35] P. Lichtsteiner, C. Posch, and T. Delbruck, "A $128 \times 1280$ 120dB 15µs Latency Asynchronous Temporal Contrast Vision Sensor," IEEE J. Solid-State Circuits, vol. 43, no. 2, pp. 566-576, Feb. 2008.
[36] D. Kim, Z. Fu, and E. Culurciello, "A 1-mw CMOS Temporal-Difference AER Sensor for Wireless Sensor Networks," IEEE Trans. Electron Devices, vol. 56, no. 11, pp. 2586-2593, Nov. 2009.
[37] E-Lab, E. Culurciello's Laboratory at Yale Univ., http://www.eng. yale.eduelab/, 2011.
[38] C. Farabet, B. Martini, P. Akselrod, S. Talay, Y. LeCun, and E. Culurciello, "Hardware Accelerated Convolutional Neural Networks for Synthetic Vision Systems," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 257-260, 2010.
[39] C. Farabet, B. Martini, P. Akselrod, B. Corda, S. Talay, Y. LeCun, and E. Culurciello, "Bio-Inspired Vision Processor for Ultra-Fast Object Categorization," Proc. High Performance Embedded Computing Workshop, 2010.
[40] E. Culurciello, R. Etienne-Cummings, and K. Boahen, "Arbitrated Address Event Representation Digital Image Sensor," Proc. IEEE Int'l Solid-State Circuits Conf. Digest of Technical Papers, pp. 92-93, May 2001.
[41] J.A. Lenero-Bardallo, T. Serrano-Gotarredona, and B. Linares-Barranco, "A 3.6µs Asynchronous Frame-Free Event-Driven Dynamic-Vision-Sensor," IEEE J. Solid-State Circuits, vol. 46, no. 6, pp. 1443-1455, June 2011.
[42] L. Chen, G. Lu, and D. Zhang, "Effects of Different Gabor Filter Parameters on Image Retrieval by Texture," Proc. 10th Int'l Multimedia Modelling Conf., vol. 1, pp. 273-278, July 2004.
[43] D. Huttenlocher, G. Klanderman, and W. Rucklidge, "Comparing Images Using the Hausdorff Distance," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 9, pp. 850-863, Sept. 1993.
[44] D.-G. Sim, O.-K. Kwon, and R.-H. Park, "Object Matching Algorithm Using Robust Hausdorff Distance Measures," IEEE Trans. Image Processing, vol. 8, no. 3, pp. 425-429, Mar. 1999.
[45] J. Chen, M.K. Leung, and Y. Gao, "Noisy Logo Recognition Using Line Segment Hausdorff Distance," Pattern Recognition, vol. 36, no. 4, pp. 943-955, 2003.
[46] C.-H. T. Yang and S.-H. Lai, "Hybrid Image Matching Combining Hausdorff Distance with Normalized Gradient Matching," Pattern Recognition, vol. 40, no. 4, pp. 1173-1181, 2007.
[47] Y. Gao and M. Leung, "Face Recognition Using Line Edge Map," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 6, pp. 764-779, June 2002.
[48] M. Barnsley, Fractals Everywhere. Academic Press, 1988.
[49] M. Litzenberger, C. Posch, D. Bauer, A. Belbachir, B.K.P. Schon, and H. Garn, "Embedded Vision System for Real-Time Object Tracking Using an Asynchronous Transient Vision Sensor," Proc. 12th Signal Processing Education Workshop, pp. 173-178, Sept. 2006.
[50] T. Delbruck and P. Lichtsteiner, "Fast Sensory Motor Control Based on Event-Based Hybrid Neuromorphic-Procedural System," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 845-848, May 2007.
[51] R. Serrano-Gotarredona, M. Oster, P. Lichtsteiner, A. Linares-Barranco, R. Paz-Vicente, F. Gomez-Rodriguez, L. Camunas-Mesa, R. Berner, M. Rivas-Perez, T. Delbruck, S.-C. Liu, R. Douglas, P. Hafliger, G. Jimenez-Moreno, A. Ballcels, T. Serrano-Gotarredona, A. Acosta-Jimenez, and B. Linares-Barranco, "CAVIAR: A 45k Neuron, 5M Synapse, 12G Connects/s AER Hardware Sensory Processing Learning Actuating System for High-Speed Visual Object Recognition and Tracking," IEEE Trans. Neural Networks, vol. 20, no. 9, pp. 1417-1438, Sept. 2009.
[52] http://www.eng.yale.edu/elab/research/svision svision.html, 2011.
[53] http://www.eng.yale.edu/elab/research/postures postures. html, 2011.
[54] Java AER Open Source Project, http:/jaer.wiki.sourceforge. net/, 2011.
[55] Y. LeCun, F. Huang, and L. Bottou, "Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2004.
[56] F.J. Huang and Y. LeCun, "NORB Object Database," http://www.cs.nyu.edu/~ylclab/datanorb-v1.0-small /, 2009.
[57] M. Riesenhuber and T. Poggio, "Hierarchical Models of Object Recognition in Cortex," Nature Neuroscience, vol. 2, pp. 1019-1025, 1999.
[58] K. Takahashi and T. Uemura, "Real-Time Human Body Posture Estimation Using Neural Networks," JSME Int'l J. Series C Mechanical Systems, Machine Elements, and Manufacturing, vol. 44, no. 3, pp. 618-625, 2001.
[59] T. Masquelier and S.J. Thorpe, "Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity," PLoS Computational Biology, vol. 3, no. 2, p. e31, Feb. 2007.
[60] R. Vogelstein, U. Mallik, E. Culurciello, R. Etienne-Cummings, and G. Cauwenberghs, "Saliency-Driven Image Acuity Modulation on a Reconfigurable Silicon Array of Spiking Neurons," Proc. Advances in Neural Information Processing Systems, Dec. 2004.
[61] M. Oster, Y. Wang, R. Douglas, and S.-C. Liu, "Quantification of a Spike-Based Winner-Take-All VLSI Network," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 55, no. 10, pp. 3160-3169, Nov. 2008.
[62] A. Linares-Barranco, M. Oster, D. Cascado, G. Jimenez, A. Civit, and B. Linares-Barranco, "Inter-Spike-Intervals Analysis of AER Poisson Like Generator Hardware," Neurocomputing, vol. 70, no. 70, pp. 2692-2700, May 2007.
[63] B. Zhao and S. Chen, "Realtime Feature Extraction Using MAX-Like Convolutional Network for Human Posture Recognition," Proc. IEEE Int'l Symp. Circuits and Systems, May 2011.
[64] Z. Fu, T. Delbruck, P. Lichsteiner, and E. Culurciello, "An Address-Event Fall Detector for Assisted Living Applications," IEEE Trans. Biomedical Circuits and Systems, vol. 2, no. 2, pp. 88-96, June 2008.
[65] T. Teixeira, D. Jung, G. Dublon, and A. Savvides, "PEM-ID: Identifying People by Gait-Matching Using Cameras and Wearable Accelerometers," Proc. ACM/IEEE Int'l Conf. Distributed Smart Cameras, pp. 1-8, 2009.
[66] T. Teixeira and A. Savvides, "Lightweight People Counting and Localizing in Indoor Spaces Using Camera Sensor Nodes," Proc. ACM/IEEE Int'l Conf. Distributed Smart Cameras, pp. 36-43, 2007.

Index Terms:
visual perception,feature extraction,image sensors,learning (artificial intelligence),pattern classification,pattern clustering,software architecture,video signal processing,human postures,object feedforward categorization,real-time video sequence,address-event temporal-difference image sensors,event- based hardware,bio-inspired software architecture,primate visual cortex,line features,vectorial segments,feature extraction,line segment Hausdorff- distance classifier,on the fly cluster based size invariant categorization,on the fly cluster based position invariant categorization,training samples,state-of-the-art bio-inspired categorization methods,computation complexity,hardware implementation,event-based circuits,Feature extraction,Image segmentation,Image sensors,Libraries,Human factors,address-event image sensor.,Human posture categorization,bio-inspired categorization,event-based circuits
Citation:
Shoushun Chen, P. Akselrod, Bo Zhao, J. A. Perez-Carrasco, B. Linares-Barranco, E. Culurciello, "Efficient Feedforward Categorization of Objects and Human Postures with Address-Event Image Sensors," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 2, pp. 302-314, Feb. 2012, doi:10.1109/TPAMI.2011.120
Usage of this product signifies your acceptance of the Terms of Use.