The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - February (2012 vol.34)
pp: 302-314
Shoushun Chen , Sch. of Electr. & Electron. Eng. (EEE), Nanyang Technol. Univ., Singapore, Singapore
P. Akselrod , Electr. Eng. Dept., Yale Univ., New Haven, CT, USA
Bo Zhao , Sch. of Electr. & Electron. Eng. (EEE), Nanyang Technol. Univ., Singapore, Singapore
J. A. Perez-Carrasco , Inst. Microelectron. Sevilla (IMSE), Sevilla, Spain
B. Linares-Barranco , Inst. Microelectron. Sevilla (IMSE), Sevilla, Spain
E. Culurciello , Electr. Eng. Dept., Yale Univ., New Haven, CT, USA
ABSTRACT
This paper proposes an algorithm for feedforward categorization of objects and, in particular, human postures in real-time video sequences from address-event temporal-difference image sensors. The system employs an innovative combination of event-based hardware and bio-inspired software architecture. An event-based temporal difference image sensor is used to provide input video sequences, while a software module extracts size and position invariant line features inspired by models of the primate visual cortex. The detected line features are organized into vectorial segments. After feature extraction, a modified line segment Hausdorff-distance classifier combined with on-the-fly cluster-based size and position invariant categorization. The system can achieve about 90 percent average success rate in the categorization of human postures, while using only a small number of training samples. Compared to state-of-the-art bio-inspired categorization methods, the proposed algorithm requires less hardware resource, reduces the computation complexity by at least five times, and is an ideal candidate for hardware implementation with event-based circuits.
INDEX TERMS
visual perception, feature extraction, image sensors, learning (artificial intelligence), pattern classification, pattern clustering, software architecture, video signal processing, human postures, object feedforward categorization, real-time video sequence, address-event temporal-difference image sensors, event- based hardware, bio-inspired software architecture, primate visual cortex, line features, vectorial segments, feature extraction, line segment Hausdorff- distance classifier, on the fly cluster based size invariant categorization, on the fly cluster based position invariant categorization, training samples, state-of-the-art bio-inspired categorization methods, computation complexity, hardware implementation, event-based circuits, Feature extraction, Image segmentation, Image sensors, Libraries, Human factors, address-event image sensor., Human posture categorization, bio-inspired categorization, event-based circuits
CITATION
Shoushun Chen, P. Akselrod, Bo Zhao, J. A. Perez-Carrasco, B. Linares-Barranco, E. Culurciello, "Efficient Feedforward Categorization of Objects and Human Postures with Address-Event Image Sensors", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.34, no. 2, pp. 302-314, February 2012, doi:10.1109/TPAMI.2011.120
REFERENCES
[1] R. Van Rullen and S. Thorpe, "Rate Coding versus Temporal Order Coding: What the Retinal Ganglion Cells Tell the Visual Cortex," Neural Computation, vol. 13, pp. 1255-1283, 2001.
[2] S. Thorpe, A. Delorme, R. Van Rullen, and W. Paquier, "Reverse Engineering of the Visual System Using Networks of Spiking Neurons," Proc. IEEE Int'l Symp. Circuits and Systems, vol. 4, pp. 405-408, 2000.
[3] C. Koch, The Quest for Consciousness: A Neurobiological Approach. Roberts and Company Publishers, 2004.
[4] T. Serre, "Learning a Dictionary of Shape-Components in Visual Cortex: Comparison with Neurons, Humans and Machines," PhD dissertation, MIT, Apr. 2006.
[5] T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, and T. Poggio, "Robust Object Recognition with Cortex-Like Mechanisms," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp. 411-426, Mar. 2007.
[6] J.J. Peissig and M.J. Tarr, "Visual Object Recognition: Do We Know More Now than We Did 20 Years Ago?" Ann. Rev. Psychology, vol. 58, pp. 75-96, 2007.
[7] J.K. Tsotsos, A.J. Rodriguez-Sanchez, A.L. Rothenstein, and E. Simine, "Different Binding Strategies for the Different Stages of Visual Recognition," Proc. Second Int'l Conf. Advances in Brain, Vision and Artificial Intelligence, pp. 150-160, 2007.
[8] S. Chikkerur, T. Serre, C. Tan, and T. Poggio, "What and Where: A Bayesian Inference Theory of Attention," Vision Research, vol. 50, no. 22, pp. 2233-2247, 2010.
[9] R. Serrano-Gotarredona, T. Serrano-Gotarredona, A. Acosta-Jimenez, and B. Linares-Barranco, "A Neuromorphic Cortical-Layer Microchip for Spike-Based Event Processing Vision Systems," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 53, no. 12, pp. 2548-2566, Dec. 2006.
[10] L. Camunas-Mesa, A. Acosta-Jimenez, C. Zamarreno-Ramos, T. Serrano-Gotarredona, and B. Linares-Barranco, "A $32\times32$ Pixel Convolution Processor Chip for Address Event Vision Sensors with 155ns Event Latency and 20Meps Throughput," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 58, no. 4, pp. 777-790, Apr. 2011.
[11] T. Teixeira, A. Andreou, and E. Culurciello, "An Address-Event Image Sensor Network," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 4467-4470, May 2006.
[12] E. Culurciello and A. Savvides, "Address-Event Image Sensor Network," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 955-958, May 2006.
[13] E. Culurciello and A. Andreou, "CMOS Image Sensors for Sensor Networks," Analog Integrated Circuits and Signal Processing, vol. 49, no. 1, pp. 39-51, 2006.
[14] T. Teixeira, E. Culurciello, E. Park, D. Lymberopoulos, A. Barton-Sweeney, and A. Savvides, "Address-Event Imagers for Sensor Networks: Evaluation and Modeling," Proc. Information Processing in Sensor Networks, pp. 458-466, Apr. 2006.
[15] Z.M. Fu and E. Culurciello, "A 1.2mw CMOS Temporal-Difference Image Sensor for Sensor Networks," Proc. IEEE Int'l Symp. Circuits and Systems, May 2008.
[16] S. Pellegrini and L. Iocchi, "Human Posture Tracking and Classification through Stereo Vision and 3D Model Matching," J. Image and Video Processing, vol. 2008, pp. 7:1-7:12, 2008.
[17] C. Wu, H. Aghajan, and R. Kleihorst, "Real-Time Human Posture Reconstruction in Wireless Smart Camera Networks," Proc. Int'l Conf. Information Processing in Sensor Networks, pp. 321-331, 2008.
[18] R. Jafari, W. Li, R. Bajcsy, S. Glaser, and S. Sastry, "Physical Activity Monitoring for Assisted Living at Home," Proc. Int'l Workshop Wearable and Implantable Body Sensor Networks, Mar. 2007.
[19] Z. Fu and E. Culurciello, "Fall Detection Using an Address-Event Temporal Contrast Vision Sensor," Proc. IEEE Int'l Symp. Circuits and Systems, May 2008.
[20] G. Virone, M. Alwan, S. Dalal, S.W. Kell, B. Turner, J.A. Stankovic, and R. Felde, "Behavioral Patterns of Older Adults in Assisted Living," IEEE Trans. Information Technology in Biomedicine, vol. 12, no. 3, pp. 387-398, May 2008.
[21] H. Su and F.-G. Huang, "Human Gait Recognition Based on Motion Analysis," Proc. Int'l Conf. Machine Learning and Cybernetics, vol. 7, pp. 464-468, Aug. 2005.
[22] B. Boulay, F. Bremond, and M. Thonnat, "Posture Recognition with a 3D Human Model," Proc. IEEE Int'l Symp. Imaging for Crime Detection and Prevention, pp. 135-138, June 2005.
[23] J. Isaacs and S. Foo, "Hand Pose Estimation for American Sign Language Recognition," Proc. 36th Southeastern Symp. System Theory, pp. 132-136, 2004.
[24] K. Takahashi, T. Sakaguchi, and J. Ohya, "Remarks on Real-Time Human Posture Estimation from Silhouette Image Using Neural Network," Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, pp. 370-375, Oct. 2004.
[25] E. H-Jaraha, C. Urunuela, and J. Senar, "Detected Motion Classification with a Doublebackground and a Neighborhood-Based Difference," Pattern Recognition Letters, vol. 24, pp. 2079-2092, 2003.
[26] L.H.W. Aloysius, G. Dong, H. Zhiyong, and T. Tan, "Human Posture Recognition in Video Sequence Using Pseudo 2-D Hidden Markov Models," Proc. Eighth Control, Automation, Robotics, and Vision Conf., pp. 712-716, Dec. 2004.
[27] P. Spagnolo, M. Leo, A. Leone, G. Attolico, and A. Distante, "Posture Estimation in Visual Surveillance of Archaeological Sites," Proc. IEEE Conf. Advanced Video and Signal Based Surveillance, pp. 277-283, July 2003.
[28] J. Triesch and C. von der Malsburg, "A System for Person-Independent Hand Posture Recognition against Complex Backgrounds," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 12, pp. 1449-1453, Dec. 2001.
[29] M. Gottardi, N. Massari, and S. Jawed, "A 100µw $128 \times 64$ Pixels Contrast-Based Asynchronous Binary Vision Sensor for Sensor Networks Applications," IEEE J. Solid-State Circuits, vol. 44, no. 5, pp. 1582-1592, May 2009.
[30] E. Artyomov and O. Yadid-Pecht, "Adaptive Multiple-Resolution CMOS Active Pixel Sensor," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 53, no. 10, pp. 2178-2186, 2006.
[31] A. Teman, S. Fisher, L. Sudakov, A. Fish, and O. Yadid-Pecht, "Autonomous CMOS Image Sensor for Real Time Target Detection and Tracking," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 2138-2141, May 2008.
[32] A. Fish, L. Sudakov-Boresha, and O. Yadid-Pecht, "Low-Power Tracking Image Sensor Based on Biological Models of Attention," Int'l J. Information Theory and Applications, vol. 14, no. 2, pp. 103-114, 2006.
[33] S. Mizuno, K. Fujita, H. Yamamoto, N. Mukozaka, and H. Toyoda, "A $256 \times 256$ Compact Cmos Image Sensor with On-Chip Motion Detection Function," IEEE J. Solid-State Circuits, vol. 38, no. 6, pp. 1072-1075, June 2003.
[34] J. Choi, S.-W. Han, S.-J. Kim, S.-I. Chang, and E. Yoon, "A Spatial-Temporal Multiresolution CMOS Image Sensor with Adaptive Frame Rates for Tracking the Moving Objects in Region-of-Interest and Suppressing Motion Blur," IEEE J. Solid-State Circuits, vol. 42, no. 12, pp. 2978-2989, Dec. 2007.
[35] P. Lichtsteiner, C. Posch, and T. Delbruck, "A $128 \times 1280$ 120dB 15µs Latency Asynchronous Temporal Contrast Vision Sensor," IEEE J. Solid-State Circuits, vol. 43, no. 2, pp. 566-576, Feb. 2008.
[36] D. Kim, Z. Fu, and E. Culurciello, "A 1-mw CMOS Temporal-Difference AER Sensor for Wireless Sensor Networks," IEEE Trans. Electron Devices, vol. 56, no. 11, pp. 2586-2593, Nov. 2009.
[37] E-Lab, E. Culurciello's Laboratory at Yale Univ., http://www.eng. yale.eduelab/, 2011.
[38] C. Farabet, B. Martini, P. Akselrod, S. Talay, Y. LeCun, and E. Culurciello, "Hardware Accelerated Convolutional Neural Networks for Synthetic Vision Systems," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 257-260, 2010.
[39] C. Farabet, B. Martini, P. Akselrod, B. Corda, S. Talay, Y. LeCun, and E. Culurciello, "Bio-Inspired Vision Processor for Ultra-Fast Object Categorization," Proc. High Performance Embedded Computing Workshop, 2010.
[40] E. Culurciello, R. Etienne-Cummings, and K. Boahen, "Arbitrated Address Event Representation Digital Image Sensor," Proc. IEEE Int'l Solid-State Circuits Conf. Digest of Technical Papers, pp. 92-93, May 2001.
[41] J.A. Lenero-Bardallo, T. Serrano-Gotarredona, and B. Linares-Barranco, "A 3.6µs Asynchronous Frame-Free Event-Driven Dynamic-Vision-Sensor," IEEE J. Solid-State Circuits, vol. 46, no. 6, pp. 1443-1455, June 2011.
[42] L. Chen, G. Lu, and D. Zhang, "Effects of Different Gabor Filter Parameters on Image Retrieval by Texture," Proc. 10th Int'l Multimedia Modelling Conf., vol. 1, pp. 273-278, July 2004.
[43] D. Huttenlocher, G. Klanderman, and W. Rucklidge, "Comparing Images Using the Hausdorff Distance," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 9, pp. 850-863, Sept. 1993.
[44] D.-G. Sim, O.-K. Kwon, and R.-H. Park, "Object Matching Algorithm Using Robust Hausdorff Distance Measures," IEEE Trans. Image Processing, vol. 8, no. 3, pp. 425-429, Mar. 1999.
[45] J. Chen, M.K. Leung, and Y. Gao, "Noisy Logo Recognition Using Line Segment Hausdorff Distance," Pattern Recognition, vol. 36, no. 4, pp. 943-955, 2003.
[46] C.-H. T. Yang and S.-H. Lai, "Hybrid Image Matching Combining Hausdorff Distance with Normalized Gradient Matching," Pattern Recognition, vol. 40, no. 4, pp. 1173-1181, 2007.
[47] Y. Gao and M. Leung, "Face Recognition Using Line Edge Map," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 6, pp. 764-779, June 2002.
[48] M. Barnsley, Fractals Everywhere. Academic Press, 1988.
[49] M. Litzenberger, C. Posch, D. Bauer, A. Belbachir, B.K.P. Schon, and H. Garn, "Embedded Vision System for Real-Time Object Tracking Using an Asynchronous Transient Vision Sensor," Proc. 12th Signal Processing Education Workshop, pp. 173-178, Sept. 2006.
[50] T. Delbruck and P. Lichtsteiner, "Fast Sensory Motor Control Based on Event-Based Hybrid Neuromorphic-Procedural System," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 845-848, May 2007.
[51] R. Serrano-Gotarredona, M. Oster, P. Lichtsteiner, A. Linares-Barranco, R. Paz-Vicente, F. Gomez-Rodriguez, L. Camunas-Mesa, R. Berner, M. Rivas-Perez, T. Delbruck, S.-C. Liu, R. Douglas, P. Hafliger, G. Jimenez-Moreno, A. Ballcels, T. Serrano-Gotarredona, A. Acosta-Jimenez, and B. Linares-Barranco, "CAVIAR: A 45k Neuron, 5M Synapse, 12G Connects/s AER Hardware Sensory Processing Learning Actuating System for High-Speed Visual Object Recognition and Tracking," IEEE Trans. Neural Networks, vol. 20, no. 9, pp. 1417-1438, Sept. 2009.
[52] http://www.eng.yale.edu/elab/research/svision svision.html, 2011.
[53] http://www.eng.yale.edu/elab/research/postures postures. html, 2011.
[54] Java AER Open Source Project, http:/jaer.wiki.sourceforge. net/, 2011.
[55] Y. LeCun, F. Huang, and L. Bottou, "Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2004.
[56] F.J. Huang and Y. LeCun, "NORB Object Database," http://www.cs.nyu.edu/~ylclab/datanorb-v1.0-small /, 2009.
[57] M. Riesenhuber and T. Poggio, "Hierarchical Models of Object Recognition in Cortex," Nature Neuroscience, vol. 2, pp. 1019-1025, 1999.
[58] K. Takahashi and T. Uemura, "Real-Time Human Body Posture Estimation Using Neural Networks," JSME Int'l J. Series C Mechanical Systems, Machine Elements, and Manufacturing, vol. 44, no. 3, pp. 618-625, 2001.
[59] T. Masquelier and S.J. Thorpe, "Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity," PLoS Computational Biology, vol. 3, no. 2, p. e31, Feb. 2007.
[60] R. Vogelstein, U. Mallik, E. Culurciello, R. Etienne-Cummings, and G. Cauwenberghs, "Saliency-Driven Image Acuity Modulation on a Reconfigurable Silicon Array of Spiking Neurons," Proc. Advances in Neural Information Processing Systems, Dec. 2004.
[61] M. Oster, Y. Wang, R. Douglas, and S.-C. Liu, "Quantification of a Spike-Based Winner-Take-All VLSI Network," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 55, no. 10, pp. 3160-3169, Nov. 2008.
[62] A. Linares-Barranco, M. Oster, D. Cascado, G. Jimenez, A. Civit, and B. Linares-Barranco, "Inter-Spike-Intervals Analysis of AER Poisson Like Generator Hardware," Neurocomputing, vol. 70, no. 70, pp. 2692-2700, May 2007.
[63] B. Zhao and S. Chen, "Realtime Feature Extraction Using MAX-Like Convolutional Network for Human Posture Recognition," Proc. IEEE Int'l Symp. Circuits and Systems, May 2011.
[64] Z. Fu, T. Delbruck, P. Lichsteiner, and E. Culurciello, "An Address-Event Fall Detector for Assisted Living Applications," IEEE Trans. Biomedical Circuits and Systems, vol. 2, no. 2, pp. 88-96, June 2008.
[65] T. Teixeira, D. Jung, G. Dublon, and A. Savvides, "PEM-ID: Identifying People by Gait-Matching Using Cameras and Wearable Accelerometers," Proc. ACM/IEEE Int'l Conf. Distributed Smart Cameras, pp. 1-8, 2009.
[66] T. Teixeira and A. Savvides, "Lightweight People Counting and Localizing in Indoor Spaces Using Camera Sensor Nodes," Proc. ACM/IEEE Int'l Conf. Distributed Smart Cameras, pp. 36-43, 2007.
18 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool