The proliferation and acceptance of reticle enhancement technologies (RET) like optical proximity correction (OPC) and phase shift masking (PSM) have significantly increased the cost and complexity of sub-100 nm photomasks. The photomask layout is no longer an exact replica of the design layout. As a result, reliably verifying RET synthesis accuracy, structural integrity, and conformance to mask fabrication rules are crucial for the manufacture of nanometer regime VLSI designs. In this paper, we demonstrate a physical model based mask layout verification system. The new system consists of an efficient wafer-patterning simulator that is able to solve the process physical equations for optical imaging and resist development and hence can achieve high degree accuracy required by mask verification tasks. It is able to efficiently evaluate mask performance by simulating edge displacement errors between wafer image and the intended layout. We show the capabilities for hot spot detection, line width variation analysis, and process window prediction capabilities with a sample practical layout. We also discuss the potential of the new physical model simulator for improving circuit performance in physical layout synthesis.