To perform as desired in a dynamic environment a vision system must adapt to a variety of operating conditions by selecting vision modules, tuning their parameters, and controlling image acquisition. Knowledge-based (KB) controller-agents that reason over explicitly represented knowledge and interact with their environment can be used for this task; however, the lack of a unifyingmethodology and development tools makes KB controllers difficult to create, maintain, and reuse. This paper presents the INVICON toolkit, based on the IndiGolog agent programming language with elements from control theory. It provides a basic methodology, a vision module declaration template, a suite of control components, and support tools for KB controller development. We have evaluated INVICON in two case studies that involved controlling vision-based pose estimation systems. The case studies show that INVICON reduces the effort needed to build KB controllers for challenging domains and improves their flexibility and robustness.