In this paper, a framework of query classification is proposed for text-based image retrieval. The classification process in this framework consists of three phases. In the first phase, two basic classifiers are employed to classify the image query into certain categories. They are easily implemented but have weak adaptability to various web image queries. Therefore, in the second phase, an expanded classifier is implemented to compensate for the possible incapability of the basic classifiers. In this step, the inferential relationships extracted by Information Flow (IF) are exploited to disclose the meaning underneath the image query, which makes the classification more accurate. In the third phase, the strengths of all three individual classifiers are leveraged together to achieve the most possible efficient classification. The experimental results prove the effectiveness of this framework in dealing with image query classification.