Assumptions underlying the convergence proofs of reinforcement learning (RL) algorithms like Q-learning are violated when multiple interacting agents adapt their strategies on-line because of learning. Empirical investigations in several domains, however, have produced encouraging results. We evaluate the convergence behavior of concurrent reinforcement learning agents using game matrices as studied by Claus and Boutilier [1]. Variants of simple RL algorithms are evaluated for convergence under increasing number of agents per group, scale up of game matrix size, delayed feedback and game matrix characteristics. Our results show surprising departures from that observed by Claus and Boutilier, particular for larger problem sizes.