In soccer games, understanding the movement of players and a ball is essential for the analysis of matches or tactics. In this paper, we present a system to track players and a ball and to estimate their positions from video images. Our system track players by extracting shirt and pants regions and can cope with the posture change and occlusion by considering their colors, positions, and velocities in the image. The system extracts ball candidates by using the color and motion information, and determines the ball among them based on motion continuity. To determine the player who is holding the ball, the position of players on the field and the 3D position of the ball are estimated. The ball position is estimated by fitting a physical model of movement in the 3D space to the observed ball trajectory. Experimental results on real image sequences show the effectiveness of the system.