The demand for automatically annotating and retrieving medical images is growing faster than ever. In this paper, we present a novel medical image annotation method based on the proposed Semantic Error-Correcting output Codes (SECC). With this annotation method, we present a new semantic image retrieval method, which exploits the high level semantic similarity. For example, a user may query the system using an image of arm while he/she expects images of hand. This cannot be realized by traditional retrieval methods. The experimental results on the IMAGECLEF 2005 annotation data set clearly show the strength and the promise of the presented methods.