This paper describes ?layered search spaces? (LSS) to accelerate recognition of a large category set. The basic concept is to employ pivots into a search space of character pattern prototypes. Given an input pattern, it is compared only with the pivots and those close to it are selected. Then, it matched with prototypes close to the selected pivots. This paper introduces multiple layers. An input pattern is compared with the top-layer pivots and those close to it are selected. Then, it is compared with the 2nd-top-layer pivots close to the selected top-layer pivots. This comparison is repeated until in the base-layer and a small set of candidate prototypes are selected. We applied this method to a handwritten Japanese character recognizer with the result that the coarse classification time was reduced to 47.1% and the whole recognition time was reduced to 46.2% while keeping classification and recognition rates as the original.