This experimental study addresses outlier rejection performance of Multilayer Perceptron (MLP) and variations of Radial Basis Function (RBF) networks. Variations include performing Principal Component Decomposition at the RBF centers (PCA-RBF) and adding a regularization term to encourage small variances. Training is performed with and without outliers. The MLPs perform worse than the RBFs. In both cases, the results indicate that, if no regularization term is used, then training with outliers can significantly improve the ability of the networks to reject outliers. A significant result is that training the PCA-RBF network with the regularization term and no outlier, we achieve as good as performance as training with outliers.