In this paper, extended learning scale is proposed to improve the efficiency of reinforcement learning. The learning scale is defined and its impact on the performance of learning is investigated. Based on the correlation of the spatial or temporal neighboring states, fuzzy state and ant colony optimization are incorporated into reinforcement learning for the extension of learning scale. In the simulation experiments, the proposed learning methods with extended learning scale are applied in a robot path planning problem. The experimental results indicate that the extension of spatial and temporal learning scale improves the learning efficiency.