Problems such as object recognition or image retrieval require feature selection (FS) algorithms that scale well enough to be applicable to databases containing large numbers of image classes and large amounts of data per class. We exploit recent connections between information theoretic feature selection and minimum Bayes error solutions to derive FS algorithms that are optimal in a discriminant sense without compromising scalability. We start by formalizing the intuition that optimal FS must favor discriminant features while penalizing discriminant features that are redundant. We then rely on this result to derive a new family of FS algorithms that enables an explicit trade-off between complexity and classification optimality. This trade-off is controlled by a parameter that encodes the order of feature redundancies that must be explicitly modeled to achieve the optimal solution. Experimental results on databases of natural images show that this order is usually low, enabling optimal FS with very low complexity.