A method for matching three-dimensional objects against a library of models from an observed sequence of silhouettes is presented in this correspondence. Based upon the observed silhouettes, the three-dimensional structure of the object is constructed and refined. The principal moments and three primary silhouettes are computed for the constructed three-dimensional objects to represent the aggregate and detailed structure parameters. The adaptive matching technique requires that sufficient silhouettes be added to modify the structure of the unknown object until consistent and steady matching results are obtained. The library for matching is based on three primary silhouettes of the model objects. Experiments conducted show a fast convergence to a consistent result may be achieved provided that a reasonable choice of silhouettes is made.