Humans have the ability to recognize objects in a cluttered scene in 100s of milliseconds. Computer algorithms operate at a much lower performance level compared to humans. Furthermore, it has proven to be particularly difficult to develop algorithms to recognize all objects in a category, such as, all cat faces vs dog faces, because of the large in-class variability. The distinguishing features can vary significantly among different objects in the same class. A similar case can be made for other categories, such as, cars, human faces, etc. In this paper we approach this problem using a model of the human visual system. The human visual system can be divided into two major pathways, commonly called the ‘what’ and where’ pathways. The ‘what’ pathway recognizes an object in a scene, but not its specific location. In this paper we present a biologically inspired hierarchical ‘what’ neural network that can successfully classify objects into categories.