Image annotation is becoming increasingly important for clinical applications and medical research. Especially, collaborative image annotation can harness the collec- tive intelligence from distributed users. There are several challenges to model medical image annotations: i) Medi- cal image annotation includes not only textual annotation but also graphical annotation, which can consist of mul- tiple areas or layers; ii) To support collaborative anno- tation from multiple users, collaboration has to be sup- ported and represented in a proper way; iii) Both tex- tual and graphical annotations should be searchable, e.g., support of spatial queries; iv) annotation history needs to be tracked and searchable in applications such as clinical trials; and v) To enable convenient annotation, and save cost, lightweight system should be provided. Most cur- rent medical image annotation applications are workstation based applications, and use proprietary methods to repre- sent annotations, which make it difficult to search, share, and exchange annotations. Meanwhile, the Web is evolving quickly on supporting strong collaboration through several new technologies, including Ajax (Asynchronous JavaScript and XML) and SVG (Scalable Vector Graphics). Especially, SVG now becomes a standard language for vector graphics on the Web, which, indeed, is recently natively supported by latest Web browsers. This brings a new opportunity to support collaborative image annotations on the Web. In our work, we develop a comprehensive data model to support both textual and graphical annotations and their history on 2-D medical images, and implement the data model with Work done while visiting Siemens Corporate Research the standard language SVG. This data model can support powerful queries on annotations including complex textual, spatial, collaboration, group, and history queries based on standard XML Query language XQuery. Besides, the ap- proach can be used to support collaborative image annota- tions in a lightweight Web browser environment.