In this paper, we report a speaker adaptation method when a small subset of word classes is available for the adaptation. A spectral transformation approach is used to adapt to a new speaker without c hanging the parameters of speaker-independent recognizer. By using “judge” network the degradation of the recognition rates for non-adapted word classes is minimized, which leads to the improvement of overall word recognition rates. The pruned judge network uses less parameter, but shows better generalization capability than full-connected linear judge networks. Remarkable reduction of error rates is achieved for adapted 10 words, while maintaining almost same recognition rates for the non-adapted words. The results demonstrated much better recognition rates compared to the base-line system.