Grid based distributed simulations are becoming more and more important with the increasing number of largescale modeling and simulation applications. Distributed simulations are evolving with modern distributed computing techniques and are facing new challenges such as interoperability, reusability, scalability, etc. Distributed Interactive Simulation and High Level Architecture have been the dominant distributed simulation standards for the past few years, and HLA is still the backbone for supporting federate based distributed simulations. However, HLA relies heavily on centralized Runtime Infrastructure (RTI) and is not easy to scale for large-scale applications. Also, its interoperability is limited since it does not use a fully opened standard such as Service Oriented Architecture (SOA). Therefore, lot research has been done to promote the next generation of simulation architecture. Such efforts result in the XMSF, which tries to integrate SOA with distributed simulation. In the meantime, Peer-to-Peer network based distributed simulations are also attracting more researchers to investigate the feasibility of decentralized architecture for large-scale distributed simulations. In this paper, we propose a hierarchical service oriented JXTA-core multi-layered architecture for large scale distributed simulations. Our particular design consideration is dynamic reconfigurable and realtime capable distributed simulation infrastructure, and we also aim to address most of the concerns regarding gridbased large-scale distributed simulation. We further verify our design through a formal DEVS simulation based modeling. We believe that a decentralized framework will be dominant in the area of distributed simulations in the near future due to its flexibility, scalability, and the ease of reconfiguring simulation applications.