We present the light field oracle, a novel mathematical concept for the acquisition, processing and representation of light fields. We first compute a hierarchical representation from a set of sparse image samples using a combination of wavelet transform and scattered data interpolation. The light field oracle then progressively acquires image data and selectively refines this initial representation. By comparing the actual input image to the corresponding reconstruction from the wavelet pyramid, the oracle dynamically decides on whether the new sample is needed and, if necessary, inserts it into the representation. Our incremental update scheme exploits the spatial localization of wavelets and allows for highly efficient image decomposition. Likewise, image reconstruction for rendering is computed locally in the wavelet domain and does not require a global inverse transform. The wavelet hierarchy along with fast decomposition and rendering operators constitutes a powerful mathematical framework also amenable to compression.