This paper presents a novel theory for learning generic prior models from a set of observed natural images based on a minimax entropy theory that the authors studied in texture modeling. We start by studying the statistics of natural images including the scale invariant properties, then generic prior models were learnt to duplicate the observed statistics. The learned Gibbs distributions confirm and improve the forms of existing prior models, and more interestingly inverted potentials are found to be necessary, and such potentials produce patterns and enhance preferred image features. The learned model is compared with existing prior models in experiments of image restoration.