Communication on-the-fly is a new, very efficient method of transferring data between shared memory processor clusters. It is based on synergy of processor switching between clusters and multiple reads of data from a transmission network shared by many processors. Effective use of this communication requires from the program to be specially structured. This paper presents a proposal of scheduling technique for programs given as Extended Macro-Dataflow Graphs. Scheduling is based on introducing proper transformations to the program graphs that reduce execution time of the program. These transformations work only on small subgraphs of the whole graph, but give global execution speedup. They are designed to use special architectural features such as communication on-the-fly and processor switching between clusters. Property of locality allows to preserve in every step the advantages of transformations applied to the graph earlier. The paper presents a set of such transformations. To show usefulness of them, an exemplary scheduling algorithm, which uses these transformations and an example of its application are presented.