Network traffic policy verification is the analysis of network traffic to determine if the observed traffic is in compliance or violation of the applied policy. An intuitive approach is the use of machine learning techniques based on specific network traffic characteristics. These traffic characteristics are also known as features, which have to be extracted and selected carefully to build robust and accurate learning models. Thus, finding the best possible learning model in combination with extracting the best possible feature-set is a necessary requirement to design accurate traffic classification models. While feature selection can be automated to find the best subset of a given set of features, there are no known mechanisms to solve the problem of feature extraction. Thus, extracting the best possible features has to be done empirically. In this work we present a framework to simplify the empirical model selection and feature extraction process.