The steady growth in size and complexity of communication networks has necessitated corresponding advances in the underlying networking technologies including communication protocols. This multi-faceted growth has rendered analysis of today's ultra-large networks, a complex task. Simulations have been used to model and analyze communication networks. Complete models of the ultra-large net-works need to be simulated in order to study crucial scalability and performance issues. Discrete event simulations of such ultra-large networks, with limited hardware resources is complex due to their sheer size. This paper presents the issues involved in the design of a framework to enable ultra-large simulations, consisting of millions of nodes. The parallel simulation techniques used, the application program interface needed for model development, and the results from experiments conducted using the framework are also presented.