Conventional additive normalization methods suffer from its linear convergence rate. We proposed a new additive normalization method for the computation of exponential function. This new method can allow the values of the normalization factors be obtained directly from the remaining terms without any computation. Furthermore, the number of the normalization factors that are obtained in each normalization stage is twice that obtained from its previous stage. As a result, the convergence rate of our method is exponential. These two advantages of our new method can significantly enhance the speed and reduce the hardware cost of the additive normalization method. Based on our proposed additive normalization method, we have designed the architectures of a 24-bit exponential unit and a 53-bit exponential unit. From the results of our thorough simulations on the 24-bit unit, we have verified that our proposed algorithm and our error analysis method are correct.