This paper proposes a method for creating human movements by imposing positional constraints of end-effectors at multiple key-frame. We introduce hierarchical reinforcement learning for efficiently searching postures at each key-frame among the huge number of possible candidates. The mechanical structures of virtual characters are also hierarchically decomposed so as to suit the learning mechanism, and each hierarchy prepares templates of discretely sampled postures for narrowing down the searching space.Our method automatically generates complicated sequential movements so that the resulting motions optimize dynamic manipulability for enhancing naturalness of realistic human behaviors.