A software system can be modeled by a transition system. In the existing approaches for software modeling, such as FSM, EFSM, and TM etc., states often have specific physics semantics which often represent variables, processes, or modules, etc. In this paper, a data mining approach is introduced into software modeling to do state definition in a different way. The approach is used to extract interesting relationships among program methods and a weighted hypergraph is constructed based on the mining results. Then the hypergraph is partitioned into k clusters which are used to define states in the transition system, using a hypergraph partitioning algorithm. States derived in this way have many particular properties. Some experiments about this approach are also presented in this paper.