The recent emergence of clouds is making the vision of utility computing realizable, i.e., computing resources and services can be delivered, utilized, and paid for as utilities such as water or electricity. This, however, creates new resource provisioning problems. Because of the pay-as-you-go model, resource provisioning should be performed in a way to keep resource costs to a minimum, while meeting an application's needs. In this work, we focus on the use of cloud resources for a class of adaptive applications, where there could be application-specific flexibility in the computation that may be desired. Furthermore, there may be a fixed time-limit as well as a resource budget. Within these constraints, such adaptive applications need to maximize their Quality of Service (QoS), more precisely, the value of an application-specific benefit function, by dynamically changing adaptive parameters. We present the design, implementation, and evaluation of a framework that can support such dynamic adaptation for applications in a cloud computing environment. The key component of our framework is a multi-input-multi-output feedback control model-based dynamic resource provisioning algorithm which adopts reinforcement learning to adjust adaptive parameters to guarantee the optimal application benefit within the time constraint. Then a trained resource model changes resource allocation accordingly to satisfy the budget. We have evaluated our framework with two real-world adaptive applications, and have demonstrated that our approach is effective and causes a very low overhead.