Today, VLSI systems for computationally demanding applications are being built as Systems-on-Chip (SoCs) with a distributed memory sub-system which is shared by a large number of processing elements. The memory sub-system is a potential performance bottle-neck in the system. In this paper, we consider such a distributed memory sub-system and study the impact of address space distribution on system performance. For a given application on such a system, we introduce the notion of address assignment quality. We show that this assignment quality metric is strongly correlated with memory sub-system throughput over large regions of the design space. We show this using open loop performance modeling of the memory sub-system, and justify this using a queueing and a Markov chain analysis. Further, we develop a detailed memory sub-system model for a multi-processor simulation system built on the Augmint [14] framework. Using two (highly parallel) applications (matrix multiplication and bubble sort) we show that application throughput and assignment quality are strongly correlated over large regions of the design space. We infer that maximization of the assignment quality metric can be a fundamental goal in designing memory sub-systems and in developing applications in such Systems-on-Chip.