Medical image simulation produces virtual images from software representations of imaging devices and virtual object models representing the human body. Object models consist of the geometry of the objects (e.g. organs, tissues, pathological structures, etc.) and of their physical parameters used for the simulation. The diversity of this information makes it difficult to share and reuse across simulation modalities and users. We address this issue by explicitly describing object models using a semantic approach. In particular, we developed an ontology that contains the relevant concepts and relations of the domain of object models for image simulation. This ontology is used to annotate object models and to describe their content and structure. In this paper, we present the construction steps of this ontology, the representation choices and we illustrate how it is used to annotate object models.