A great challenge in tracking multiple objects is how to locate each object when they interact and form a group. We view it as a binary classification problem. It is important to base the classification on the currently most discriminative features. We derive a unified framework for learning feature extraction and classification in appearance-spatial space for multiple object tracking. In this framework, both classifier design and feature evaluation are accomplished by minimizing an criterion which corresponds to an upperbound of classification error. There, the most discriminative features, as variables, minimize the criterion function, whereas the classifier, as a function, minimizes the criterion functional. The resulting system offers high accuracy for real-time tracking of nearby multiple objects in complex and dynamic scenes.