Template matching is one of the key problems in computer vision and has been widely used in tracking, recognition and many other applications. Traditional methods are usually slow because the template needs to be matched to every location in the image and the matching involves element-byelement floating point multiplications. The process is even slower when multi-scale matching is needed. This makes it not suitable for time-critical applications. In this paper, we present a novel approach to accelerate multi-scale template matching. The main computation saving is achieved by representing the template as a linear combination of a small number of Haar-like binary features. This representation replaces the element-by-element floating point multiplications with several additions thus significantly improves the speed. In addition, such simple features can easily adapt to template scale changes with negligible extra computation cost. Experiments show that the proposed method can achieve speed improvement up to two orders of magnitude.