Job scheduling has attracted much research attention in recent years. Various job scheduling methods have been proposed and evaluated under different workload and grid conditions. However, few efforts have been made to analyze the underlying causes that lead to the performance results of the proposed scheduling methods. This paper presents our studies on analysis of job scheduling performance in computing grids from the perspective of resource fragmentation. Two parts of job scheduling will impact on resource fragmentation: job selection and site selection. A series of simulations have been conducted to investigate the effects of resource fragmentation in terms of average waiting time of all jobs. The performance results indicate that resource fragmentation plays an important role on job scheduling performance. Based on the results, good job selection and site selection mechanisms are proposed to form an effective job scheduling method which could reduce resource fragmentation and thus improve system performance, achieving more than five times performance improvement compared to primitive job scheduling methods.