In extracting a polynomial surface patch near an intensity or range discontinuity, a robust estimator must tolerate not only the truly random bad data ("random outliers"), but also the coherently structured points ("pseudo outliers") that belong to a different surface. To characterize the performance of least median of squares, M estimators, Hough transforms, RANSAC, and MINPRAN on data containing both random and pseudo outliers, we develop two analytical measures, "pseudo outlier bias" and "pseudo outlier breakdown". Using these measures, we find that each robust estimator has surprisingly poor performance, even under the best possible circumstances, implying that present estimators should be used with care and new estimators should be developed.