Flat appearance-based systems, which combine clever image representations with standard classifiers, might be the most effective way to recognize objects using current technologies. In the future, however, it seems probable that hierarchical representations might have better performance. In such systems, the image representation consists of a sequence of sets of features, where each subsequent set is computed based on the previous sets.

The main contributions of this paper are to: (1) pose the question "what is the best way to employ discriminative methods for hierarchical image representations?"; (2) enumerate some of the alternative hierarchies while drawing connections to recent work by brain researchers; (3) study experimentally the different alternatives. As we will show, the strategy used can make a substantial difference.