This paper presents an Open Source Vision Library (OpenVL) for hardware acceleration of video-based surveillance systems and other computer vision applications to facilitate low latency, real-time response. Our approach is inspired by the success of OpenGL in promoting the development of hardware acceleration for computer graphics. Our goal is to create OpenVL as a standard interface for computer vision applications that can work seamlessly on different software and hardware platforms supporting distributed camera arrays. It allows users to easily recover useful information about real dynamic scenes quickly, and in a portable manner across various software and hardware platforms. Finally, we implement an example surveillance system, called a Local Positioning System (LPS), to validate the critical underlying concepts of OpenVL.