In this paper, we discuss a real-time vision architecture which provides a mechanism of controlling trade-off between the accuracy and the latency of vision systems. In vision systems, to acquire accurate information from input-images, the huge amount of computation power is usually required. On the other hand, to realize real-time processing, we must reduce the latency. Therefore, under given hardware resources, we must make difficult trade-off between the accuracy and the latency so that the quality of the system's output keeps appropriate. To solve the problem, we propose confidence-driven scheme, which enables us to control the trade-off dynamically and easily without rebuilding vision systems. In the confidence-driven architecture, the trade-off can be controlled by specifying a generalized parameter called confidence, which relatively indicates how accurate the analysis should be. Here, we present the concept of confidence-driven architecture, and then, we show a shared memory which uses confidence-driven scheme. Using confidence-driven memory, we can use imprecise computation model to reduce the latency without a large decrease of accuracy.