Adaptive background modeling/subtraction techniques are popular, in particular, because they are able to cope with background variations that are due to lighting variations. Unfortunately these models also tend to adapt to foreground objects that become stationary for a period of time; as a result such objects are no longer considered for further processing. In this paper, we propose the first (to our knowledge) statistically consistent method for incorporating feedback from high-level motion model to modify adaptation behavior. Our approach is based on formulating the background maintenance problem as inference in a continuous state Hidden Markov Model, and combining it with a similarly formulated object tracker in a multichain graphical model framework. We demonstrate that the approximate filtering algorithm in such a framework outperforms the common feed-forward system while not imposing a significant extra computational burden.