In this paper, we address the problem of recovering an optimal salient image descriptor transformation for image classification. Our method involves two steps. Firstly, a binary salient map is generated to specify the regions of interest for subsequent image feature extraction. To this end, an optimal cut-off value is recovered by maximising Fisher’s linear discriminant separability measure so as to separate the salient regions from the background of the scene. Next, image descriptors are extracted in the foreground region in order to be optimally transformed. The descriptor optimisation problem is cast in a regularised risk minimisation setting, in which the aim of computation is to recover the optimal transformation up to a cost function. The cost function is convex and can be solved using quadratic programming. The results on unsegmented Oxford Flowers database show that the proposed method can achieve classification performance that are comparable to those provided by alternatives elsewhere in the literature which employ pre-segmented images.