This paper addresses the problem of performance modeling of large-scale distributed systems with emphases on communication networks in heterogeneous multi-cluster systems. The study of interconnection networks is important because the overall performance of a distributed system is often critically hinged on the effectiveness of this part. We present an analytical model to predict message latency in multi-cluster systems in the presence of processor heterogeneity. The model is validated through comprehensive simulation, which demonstrates that the proposed model exhibits a good degree of accuracy for various system sizes and under different operating conditions.