The Adaptive Parallelism Environment is introduced as a means of effectively utilizing MPP processing resources in a multi-programmed MIMD or distributed system. It achieves this by dynamically calculating the optimal number of threads for a given program at runtime. The optimality calculation considers both the number of processing elements available to handle additional workloads and the estimated computational speedup gained by using additional threads versus the communications overhead of using additional threads. The Adaptive Parallelism Environment is composed of three sections: a load balancer, which migrates threads and provides information regarding the availability of processing elements, a code analyzer, which estimates the number and composition of instructions in a potential thread as well as the number of communications needed to synchronize the potential thread, and a runtime environment, which gathers information from the load balancer and code analyzer and performs the runtime calculations to estimate the optimal number of threads.