Voice over IP (VoIP) systems are gaining in popularity as the technology for transmitting voice traffic over IP networks. As the popularity of VoIP systems increases, they are being subjected to different kinds of intrusions some of which are specific to such systems and some of which follow a general pattern. VoIP systems pose several new challenges to Intrusion Detection System (IDS) designers. First, these systems employ multiple protocols for call management (e.g., SIP) and data delivery (e.g., RTP). Second, the systems are distributed in nature and employ distributed clients, servers and proxies. Third, the attacks to such systems span a large class, from denial of service to billing fraud attacks. Finally, the systems are heterogeneous and typically under several different administrative domains. In this paper, we propose the design of an intrusion detection system targeted to VoIP systems, called SCIDIVE (pronounced "Skydive"). SCIDIVE is structured to detect different classes of intrusions, including, masquerading, denial of service, and media stream-based attacks. It can operate with both classes of protocols that compose VoIP systems - call management protocols (CMP), e.g., SIP, and media delivery protocols (MDP), e.g., RTP. SCIDIVE proposes two abstractions for VoIP IDS .Stateful detection and Cross-protocol detection. Stateful detection denotes assembling state from multiple packets and using the aggregated state in the rule matching engine. Cross protocol detection denotes matching rules that span multiple protocols. SCIDIVE is demonstrated on a sample VoIP system that comprises SIP clients and SIP proxy servers with RTP as the data delivery protocol. Four attack scenarios are created and the accuracy and the efficiency of the system evaluated with rules meant to catch these attacks.