This paper presents an efficient method to detect faces from image sequences such as news or visual surveillance. To speed up the search, the prior probabilities of the face locations in the image are adaptively estimated and are used in the Ising search algorithm. The search points in the Ising search are selected depending on the estimated prior probabilities. The information obtained by the previous search points in the given image is effectively utilized through spin flip dynamics of the Ising search. If a face is found, the prior probabilities are updated with forgetting. This makes adaptation to the changes of the environment possible. The proposed search method was applied to the news images captured during about a week. The proposed method is about 10 times faster than Ising search method without prior probability estimation.