Ideation is an essential component of creativity and problem-solving. Researchers have measured the quality of ideation treatments by assigning quality scores to each unique idea generated in each session and then by calculating one or more of the sum-of-scores, average-quality-score, or count-of-good-ideas measures. We discuss the validity of these three measures and the potential biases associated with the sum-of-scores and average-quality measures. An experimental study comparing multiple levels of social comparison was used to illustrate the differences in the quality measures and the results revealed that research conclusions were dependent on the quality measure used. Implications for future research are discussed including a recommendation that future ideation research adopt the count-of-good-ideas measure for assessing ideation quality.