Optical flow estimation can be formulated as two regression stages, derivative estimation and optical flow constraints (OFC) solving. Traditional approaches use Least Squares at both stages and are sensitive to assumption violations. To improve estimation accuracy especially near motion boundaries, we use a Least Trimmed Squares (LTS) estimator to solve the OFC, obtaining a confidence measure for each estimate; and at place with low confidence, we use another LTS estimator to robustify derivative estimation. This adaptive two-stage robust scheme has significantly higher accuracy than non-robust algorithms and those only using robust methods at the OFC stage. Advantages are illustrated on both synthetic and real data.