A measure of stability for a wide class of pattern recognition algorithms is introduced to cope with over-fitting in classification problems. Based on this concept, constructive methods for designing effective stable algorithms are developed. New algorithm is represented as convex combination of the initial algorithms with weights that depend both from the location of the point being classified and from the degree of local stability of each algorithm. Either a set of parametric algorithms from the same model or algorithms that belong to different models may be used for such fusion.