In this paper, we present a methodology, which enables designing optimal macro data flow graphs that represent computation and communication patterns for the FDTD problem in irregular computational areas. The macro data flow graphs are executed in a MIMD system. Communication is implemented with a Remote Direct Memory Access facility. To obtain minimal communication overheads, the rotating buffers mechanism has been introduced. It is used to control (and synchronize) data flow during execution of fine grain applications with irregular communication. Several methods for data flow graph partitioning and merging are presented. In these methods we take into consideration both computational load balance and minimal communication overhead. Experimental results obtained by simulations are discussed.