Real-time media servers are becoming increasingly important as the Internet supports more and more multimedia applications. In order to meet these ever increasing demands, real-time media servers will be responsible for supporting a large number of clients with a wide range of QoS requirements. While techniques such as aggregation of state information for scalability have been proposed in the literature such as with Differentiated Services, the per-stream effects of such aggregation are poorly understood. In this paper, we explore the effects of aggregated state information and propose a granularity aware (m, k) queue management (GAQM) which improves control over the tradeoff between scalability/granularity and QoS performance. Specifically, we identify the necessity of balancing aggregation groups according to critical characteristics such as relative deadlines. We present detailed examples of GAQM and evaluate our work through simulation studies.