The design of new diagram recognition systems remains a challenging problem. Ideally, recognition systems should accept real-world input, perform robustly, fail gracefully, and be implemented in a timely manner. In reality, the intricacy involved in implementing recognition systems for diagram notations makes this a challenging open problem. One solution to these challenges is the design of middleware to speed the development of robust applications. Middleware takes the form of a framework or toolkit for the creation of applications. This paper describes a retargetable framework which can be used to speed the development of robust interactive sketch recognition systems. The system includes a drawing surface to capture interactively created drawings, a set of generic segmentation routines, a character recognizer, and a common interface for integrating domain-specific components. The framework has been used to construct systems for the recognition of UML, math, and molecular diagrams. Work is on-going on the design of additional generic recognizers of logical structure and spatial layout of diagrams.