Pervasive computing systems is becoming widespread as the next generation of computing systems. Many middleware-based researches are proposed to provide services while vanishing the pervasive computing systems into the background environment. Security is of critical importance to protect system resources as well as user privacies. In this paper, we propose a security middleware architecture for heterogeneous pervasive devices to collaborate with high security confidence while ignoring the security details. We elaborate a trust management system in the infrastructure as one of the security middleware components. Our middleware architecture provides dynamic reconfiguration to fulfill security requirements of heterogeneous service providers and consumers