Although Network Intrusion Detection Systems (nIDS) are widely used, there is limited understanding of how these systems perform in different settings and how they should be evaluated. This paper examines how nIDS performance is affected by traffic characteristics, rulesets, string matching algorithms and processor architecture. The analysis presented in this paper shows that nIDS performance is very sensitive to these factors. Evaluating a nIDS therefore requires careful consideration of a fairly extensive set of scenarios. Our results also highlight potential dangers with the use of workloads based on combining widely-available packet header traces with synthetic packet content as well as with the use of synthetic rulesets.