In augmented reality (AR) application, registering a virtual object with its real counterpart accurately and comfortably is one of the basic and challenging issues in the sense that the size, depth, geometry, as well as physical attributes of the virtual objects have to be rendered precisely relative to a physical reference, which is well-known as the calibration or registration problem. This paper presents a systematic calibration process to address static registration issue in a custom-designed augmented reality system, which is based upon the recent advancement of head-mounted projective display (HMPD) technology. Following a concise review of the HMPD concept and system configuration, we present in detail a computational model for the system calibration, describe the calibration procedures to obtain the estimations of the unknown transformations, and include the calibration results, evaluation experiments and results.