A billiard ball tracking system is designed to combine with a visual guiding interface run on a PC to instruct users of game playing strategy. The system makes use of a vision system for tracking of the cue ball, and target ball locations. A visual display system instructs users of an ideal striking direction of a cue ball using a cue stick to sink a selected object ball into a target pocket in the real world. Both sub-systems are fully integrated through a least square error calibration process, which correlates the real world, and the virtual world pool ball coordinates. The vision system has been tested to extract and recognize the different balls.. identity and their center positions. The system has been tested to work together successfully in a mini-prototype pool table. Users are able to adjust the cue stick on the table according to the visual instruction displayed on a PC. Arbitrarily selected object ball can be driven into the target pocket accordingly.