In this article, we propose an object tracking method using the shape representation network (SRN), which represents an object's shape with the arbitrary degree of blurring. The SRN is a feed-forward neural network composed of several edge description units each of which describes an edge shape with a blurring parameter, and one combining unit, which represents various kinds of object's shape with a blurring parameter. In the first frame, the SRN is trained by minimizing squared errors between the frame and the template represented by the SRN. From the second frame, object is tracked by affine transformation of the template created in the initial frame. Originally, the blurring parameter has been held fixed to a constant value during the object tracking processes, however when an object has large movements between frames, this method can not keep tracking the object precisely through the whole image sequences. To solve this problem, we introduced a coarse-to-fine approach into the template matching process by changing the blurring degree of a template. The coarse-to-fine approach improves the tracking performance of SRN, when a target moves rapidly and there is a large difference in object postures between adjacent frames.