Both High Dynamic Range images and their tone mapped correspondents contain relative luminance values which have to be mapped on a scale of available gray levels of a display. Such mapping includes brightness adjustment, which has a direct impact on the final image appearance and the observers' assessment of image quality. We conduct a psychophysical experiment in which subjects adjust image brightness to match their preference. We observe that the brightness choice is consistent across subjects and is primarily affected by image content. We investigate popular methods for automatic brightness adjustment and show a significant inaccuracy for a group of images. The incorrect brightness adjustment degrades in these cases perceived image quality. We identify characteristics of images that are highly correlated with the subjects' choice of brightness and develop an improved model for the brightness adjustment.