In this paper, we present a new voting-based object labeling method that is robust to background clutter. The conventional simple voting method shows very poor performance under clutter. To reduce the effect of clutter, first we aggregate the weights between the features and the support features using similarity and proximity. Through the recursive weight aggregation process, features belonging to the same objects get stronger weights, and features belonging to clutter get weaker weights. Then, we vote the weightaggregated features to get the object labels. We validate the enhancement of the proposed method by using an open database and a real test set.