In this paper, we present a detailed study of how to design efficient and scalable flow control mechanisms in MPI over the InfiniBand Architecture. Two of the central issues in flow control are performance and scalability in terms of buffer usage. We propose three different flow control schemes (hardware-based, user-level static and user-level dynamic) and describe their respective design issues. We have implemented all three schemes in our MPI implementation over InfiniBand and conducted performance evaluation using both micro-benchmarks and the NAS Parallel Benchmarks. Our performance analysis shows that in our testbed, most NAS applications only require a very small number of pre-posted buffers for every connection to achieve good performance. We also show that the user-level dynamic scheme can achieve both performance and buffer efficiency by adapting itself according to the application communication pattern. These results have significant impact in designing large-scale clusters (in the order of 1,000 to 10,000 nodes) with InfiniBand.