The indexing of spatio-temporal data is important for retrieval by spatio-temporal queries. The previous techniques on spatio-temporal indexing miss the semantics of the application since they are usually based on traditional indexing structures that has little to no semantic information incorporated. In those systems, the semantic queries were executed by using the low-level index structures. In this paper, we introduce a novel indexing method for spatiotemporal data: semantic sequence state graph (S3G). S3G maintains the properties of events-objects locations for efficient spatio-temporal queries. In S3G, the spatial information is maintained in states whereas semantic events that result in temporal ordering link the states. S3G supports our SMART(semantic modeling and retrieval) system.