Finite sample size has always been a problem in determining the retrieval accuracy of a Content-Based Image Retrieval (CBIR) system. Though a good amount of research has been done in the statistical pattern recognition field, no such effort is shown in relation to CBIR. In this paper, we considered image retrieval as a dichotomous classification problem and studied the effect of sample size on the retrieval accuracy. We reported experimental results and analysis with two different image databases of size 2000 and 500, both having 10 semantic categories. For both data sets, we showed the variation of precision with sample size. We also studied the effect of sample size on retrieval accuracy as Relevance Feedback (RF) is applied. For both data sets, the nett improvement in precision with RF increases with sample size.