Sampling issue is an important problem in image based rendering. In this paper, we propose an adaptive sampling method to improve the Layered Depth Image framework. Different from the existing methods of interpolating or splatting neighboring pixels, our method selects a set of sampling views based on the scene analysis that can guarantee the final rendering quality. Furthermore, the rendering speed is accelerated by the pre-computed patch lookup table, which simplifies the reference view selection process to a simple lookup of a hash table. We have implemented our method. The experiment study shows the advantage of the method.