As semantic web grows, security concerns increase. One concern is controlling accesses to resources in this environment. In order to infer whether the access is allowed or not, different information of different entities including contextual information should be involved. From access control point of view, we divide the entities in semantic web into three categories: resources (objects), requesters (subjects), and environment (infrastructure, time, and location). In this paper, we present a semantic-based context-aware access control framework to be applied in semantic web, considered as a multi-domain environment. To handle context information in the framework, we propose a context ontology to represent contextual information and employ it in the inference engine. The proposed ontology classifies the context of a semantic web environment and represents the elements of contextual information and their relationship in an abstract level. We illustrate how the access control framework handles the contextual information with the proposed context ontology.