When conducting empirical studies, replications are important contributors to investigate the generality of the studies. By replicating a study in another context, it is investigated which impact the specific environment has, related to the effect of the studied object. In this paper, we define different levels of replication to characterise the similarities and differences between an original study and a replication with particular focus on prediction models for identification of fault-prone components. Further, we derive a set of issues and concerns which are important in order to enable replication of an empirical study and to enable practitioners to use the results. To illustrate the importance of the raised issues, a replication case study is presented in the domain of prediction models for fault-prone software components. It is concluded that the results are very divergent depending on how different parameters are chosen, which demonstrates the need for well documented empirical studies to enable replication and use.