In this paper we present a distributed reinforcement learning system that leverages on expert coordination knowledge to improve learning in multi-agent problems. We focus on the scenario where agents can communicate with their neighbors but this communication structure and the number of agents may change over time. We express coordination knowledge as constraints to reduce the joint action space for exploration. We introduce an extra learning level to learn when to make use of these constraints. This extra level is decentralized among the agents, making it suitable for our communication restrictions. Experiment results on tactical real-time strategy and soccer games show that our system is effective in online learning as opposed to existing methods that use individual constraints on agents and coordinated action selection.