In the area of image-based modeling and rendering (IBMR), a new scene under a novel illumination condition is synthesized by interpolating reference images. However, for good quality rendering tremendous reference images under various illumination conditions are required. This paper presents a two-level compression method for reference images. In the first level, the spherical harmonic transform is used to approximate the plenoptic function of each pixel. In the second level, we use the embedded zerotrees wavelet (EZW) method for removing the spatial redundancy in spherical coefficients produced from the first level process. Simulation results show our approach is much superior than compressing reference images with JPEG.