This paper deals with the analysis of broadcast soccer video. To recognize interesting events such as a goal, estimation of ball movements is necessary. It is, however, sometimes difficult to detect a ball by a simple color and shapebased method when it overlaps with players and lines. We therefore develop a method of estimating a ball route during such overlaps by considering spatio-temporal relationships between players, lines, and the ball. The method can deal with difficult cases such as the one where a ball disappears at a player and re-appears from another player. Experimental results show the effectiveness of the method.