We create a framework for physical interaction with virtual human. The virtual human reacts to the input of the user in realtime. The virtual human has an ocular system model, which mimics human ocular system, to generate natural and human-like gaze motions and to create inputs for the cognitive model of the virtual human. The visual input for virtual human is generated from the dynamics simulator for the virtual world. The gaze target is chosen by vying of bottom-up and top-down attentions. The motion controller for the eyes and head is based on observations and models of human gaze motion. We create a virtual boxing game as an application of proposed framework. Proposed framework succeeds to create natural gaze reaction. In addition, by changing the balance of the bottom-up and top-down attention, looks of concentration of the virtual human can be tuned.