Web Services have a variety of qualities, such as flexibility,simplicity,support communication through firewalls and provide interconnection between distributed applications over the Internet. However, the Web Services performance depends on the toolkit implementation and the choice of the SOAP payload transport protocol. It is necessary to investigate the performance limitations of Web Services implementations. The performance evaluation process can demand good amount of time, money and a developer with a good experience in the Web Services technology and toolkit. This paper presents an open source utility to automate the performance comparison of different Java Web Services toolkits. At this moment, JWSPerf utility supports three Java toolkits - Apache Axis, Sun Java Web Services Developer Pack and Systinet Server for Java. Beyond, a general-purpose benchmark was defined to compare the results and explore the influence of the complexity and the size of the data types.