In grid environment, many resources can be shared. But the owner of the resource can decide, access, modify the access policy. Only legal user can access resources. Further, the communication activities in grid sometimes cross multi-domain. So a flexible secure access control mechanism is needed. After we have realized secure data exchange between communication parties in grid with the help of computer technology up to date, especially we focus on access control policy research. A access control structure for cross-domains grid which integrates X.509v4 PMI (Privilege Management Infrastructure) with the XML security solution is presented. Further, a fine-grained access control mechanism, which has integrated the temporal RBAC in PMI is proposed. The corresponding temporal role assignment algorithm is given as a key technique. The access control mechanism improved the security of the system no matter the access comes from inside networks or outside networks.