In this paper, we address the problem of image sequence analysis by classification, which is becoming increasingly important in the context of real-time vision systems for object recognition and motion analysis. As an example, we regard the recognition of walking pedestrians in complex traffic scenes. Polynomial support vector machines are applied to complete image sequences, representing extremely high-dimensional input patterns, and to reduced feature, sets obtained by standard “global” principal component analysis. These approaches are compared to the adaptable time delay neural network (ATDNN) algorithm based on receptive fields that perform a “local” spatio-temporal processing of the image sequence, generating feature sets that are classified by polynomial support vector machines in an extended version of the ATDNN algorithm. The computational complexity of the local approaches is up to two and the memory demand up to four orders of magnitude lower than the corresponding values for the global approaches while the recognition performance of the local approaches is even higher than that of the global ones.