Most existing methods for edge detection rely on contrast dependent thresholds. We show that a local measurement defined by the ratio of the smallest to the largest eigenvalue of the second moment matrix of filter kernels, can be used to separate smooth, low curvature curves and straight lines from noise, independent of contrast, in both color and gray level images. This is done without applying a threshold to the gradient magnitude. The edge images are defined as zero crossings in the gradient direction. The covariance matrix can easily be computed for both gray level images and color images. Further we show the potentiality of such a measure by integrating it with the Hough transform to extract long straight lines in noisy color images. The method is shown to successfully extract consistent line features from color images of a scene, captured under drastically different lightening conditions.