For autonomous neural network systems one usually needs fast learning and good generalization performance, and there will inevitably be a trade-off between these two requirements. Using evolutionary techniques can generate high performance networks, but this often leads to unwanted side effects, such as occasional instances of very poor performance. This paper explores the problems that arise for traditional evolved neural networks using a range of evolutionary approaches, and shows how they can, to a large extent, be overcome by allowing the networks to evolve age dependent plasticities.