Error Correcting Output Coding (ECOC), an information theoretic concept, seems an attractive idea for improving the performance of automatic classifiers, particularly for problems that involve large number of classes. In this paper, we look at the conditions necessary for reduction of error in this framework and introduce a new version of ECOC. To show the error reduction procedure and compare the new algorithm with traditional one, we use an artificial benchmark on which we are able to control the rate of noise to investigate the behavior of system in different parts of input space, as well as a few real problems.