Without an independence assumption, combining multiple classifiers deals with a high order probability distribution composed of classifiers and a class label. Storing and estimating the high order probability distribution is exponentially complex and unmanageable in theoretical analysis, so we rely on an approximation scheme using the dependency. In this paper, as an extension of the second-order dependency approach, the probability distribution is optimally approximated by the third-order dependency and multiple classifiers are combined. The proposed method is evaluated on the recognition of unconstrained handwritten numerals from Concordia University and the University of California, Irvine. Experimental results support the proposed method as a promising approach.