The explosion of WWW traffic has triggered great interest in distributed web server systems. Among various distributed web server architectures, the DNS-based distributed system is a promising solution in terms of performance, scalability and availability. DNS(Domain Name Server) name caching has significant effects on the load balance of distributed web server systems, due to the traffic skewness it causes. Also, there is page caching along whole paths between clients and web servers, and page caching schemes could affect the system load balancing a lot. In this paper, we examine various caching issues, including the load balancing algorithms of the DNS, the locations of caches, and the page caching policies, for the DNS-based web server system. We use stochastic processes to model the web traffic, and compare the load balance performance of these algorithms and policies based on simulation results. We found that, the DNS load balancing algorithm, which takes into account both client domain information and server load information, could yield best load balance performance among various algorithms, and a server-side cache with the policy which only caches popular web pages could significantly improve both the cache hit ratio and the load balance.