Various techniques can be used to improve visualization experiences for individuals with color vision deficiency, including recoloring, pattern superposition, and the use of a color-perception simulation model for assisting visualization designers.
According to recent estimates, approximately 200 million individuals have some form of color vision deficiency (CVD). 1 , 2
Such a condition affects their private and professional lives, impacting their ability to effectively interpret color-coded information. As such, these individuals can’t fully benefit from standard scientific, medical, and information visualization techniques. Figure 1
illustrates the problem with a situation faced on a daily basis by individuals with CVD. Figure 1b
shows a simulation of how one such individual sees the pie chart shown in Figure 1a
. Note the difficulty to identify sectors A and B. Because decisions based on ambiguous information might have undesirable and even catastrophic implications, addressing this ambiguity problem is relevant both socially and economically. Fortunately, digital media has become the prevalent form of image and video consumption, making it practical for us to use enhancement techniques to compensate for the loss of color contrast experienced by these individuals.
Figure 1.. Ambiguity caused by color vision deficiency (CVD). (a) Pie chart. (b) Simulated view of an individual with CVD (deuteranope) for the chart in (a). The identification of sectors A and B is ambiguous.
This article describes a series of techniques that we developed over the past few years to enhance visualization experiences for individuals with CVD. These include recoloring techniques, the use of patterns to encode color, and a model for simulating the color perception of these individuals. While recoloring techniques and patterns can be used to disambiguate color-coded information, color-perception simulation provides valuable feedback for visualization designers to create more effective color scales and transfer functions capable of reaching broader audiences. Before discussing these techniques, let’s briefly review the classes and causes of CVD.
Color Vision Deficiency: A Brief Review
Human normal color vision requires three types of cone cells (the retina’s photoreceptors involved in color sensation). If one type of cone is missing, the individual is called a dichromat
. Dichromats can be further classified as protanopes
, or tritanopes
, if the missing photo-pigment is more sensitive to the long, medium, or short wavelengths of the visible spectrum, respectively. Alternatively, changes in the composition of the photopigments can shift their spectral sensitivity. 1
In this case, the individuals are called anomalous trichromats
, and likewise can be classified as protanomalous
, or tritanomalous
. Individuals having a single or no type of photopigment are called cone monochromats
and rod monochromats
CVD is an inherited condition associated with the X chromosome, and whose incidence varies with ethnic groups. For the male population, current estimates suggest that approximately 7.40 percent of the Caucasian population, 4.17 percent of Asians, and 2.61 percent of Africans have some level of red-green CVD. 1
The prevalence is considerably smaller in the female population: 0.50 percent, 0.58 percent, and 0.54 percent, respectively, for the same ethnic groups. Unfortunately, there’s no clinical or surgical treatment available for CVD.
Ideally, recoloring techniques intended to enhance color contrast for individuals with CVD should satisfy a long list of desirable features:
preserve the “perceptual distances” among the original colors;
preserve, as much as possible, the original colors;
satisfy a global consistency property;
preserve the original image luminance;
exhibit temporal coherence; and
achieve real-time performance.
Some of these requirements might conflict with others.
Global consistency means that all pixels with a given color in the original image should be mapped to a single shade in the recolored one. Preserving the luminance values of the original pixels is key to avoid luminance polarity reversal: a distracting artifact that happens if brighter pixels become darker or vice-versa after recoloring. Temporal coherence and real-time performance are necessary for interactive visualization sessions.
Recently, several recoloring techniques have been proposed trying to recover the color contrast missed by dichromats. 3 – 10
Some of these techniques require user assistance, and the quality of their results is highly dependent on the user-provided parameters. 3 , 7
Most of the automatic recoloring techniques are non-deterministic and insufficiently fast for use in interactive applications. 4 – 6 , 8
Two automatic recoloring techniques 9 , 10
try to address the aforementioned list of requirements, and have been evaluated in visualization contexts by individuals with CVD. As with other automatic techniques, they’ve been designed for dichromats, but are also applicable to anomalous trichromats, although using a reduced color gamut (we discuss these next).
A technique introduced by Giovane Kuhn, Leandro Fernandes, and myself 9
models the recoloring process as a mass-spring optimization, which can be efficiently implemented on GPUs. The technique tries to satisfy requirements 1 through 6. Similar to another approach, 5
the optimization is first applied to a set of quantized colors, and the results of this first step are then used to optimize the entire set of colors. Due to the use of quantization, it’s unclear how it can enforce temporal coherence. Although this technique achieves interactive rates, it’s still not sufficiently fast for real-time recoloring on the fly.
The algorithm consists of three steps:
• color quantization;
• optimization of the set of quantized colors; and
• reconstruction of the recolored image from the set of optimized (quantized) colors.
Quantization can be performed using any technique, such as uniform quantization or k
-means. In the L*a*b* color space, the quantized colors are orthographically projected onto a plane that approximates the dichromat’s color gamut (see Figure 2
). Such a projection mimics the dichromat’s perception for the given color. Each projected color is then treated as a particle, and a spring is set between each pair of particles. The rest length of any given spring is given by the perceptual difference between the pair of quantized colors associated with the particles it connects. The mass of each particle is defined as the reciprocal of the perceptual difference between its corresponding quantized color and the dichromat’s perception. The L* coordinate is kept constant to preserve luminance, and optimization is performed in 1D. After optimization, the particles’ positions on the dichromat’s plane define the optimized colors.
Figure 2.. Planes that approximate the color gamut of each class of dichromacy in the L*a*b* color space (after my work with Giovane Kuhn 9 ). (a) Protanopes, (b) deuteranopes, and (c) tritanopes.
The use of perceptual differences between pairs of quantized colors as the spring rest lengths makes the technique try to enforce that the recolored image will preserve such differences. Also, the more similarly a color is perceived by a normal trichromat and by a dichromat, the bigger its mass—and therefore, the bigger its resistance to move. These are desirable features. On the other hand, the restrictions of luminance and original-color preservation impose constraints to the 1-D optimization. This makes it harder for the optimization to preserve the perceptual color differences found in the original image and, as a result, it might be trapped in some local minimum.
show examples of images recolored with Kuhn’s technique, and illustrate the points discussed. Figure 3b
shows a simulated view of a deuteranope for the graph shown on its left. Note how the perceived colors of the two lower bars are essentially indistinguishable. Figure 3c
shows the recolored image, where the color of the middle bar has been preserved, as it can be perceived by deuteranopes. Given a deuteranope’s reduced color gamut ( Figure 2b
), the original green bar has been recolored with a brighter shade of yellow (blue’s opponent color).
Figure 3.. Optimization-based recoloring. (a) Reference image. (b) Simulated view of a deuteranope for the reference image. (c) Recolored image produced by Kuhn’s mass-spring optimization technique.
Figure 4.. Optimization-based recoloring can be trapped in a local minimum. (a) Reference image. (b) Simulated view of a protanope for the reference image. (c) Recolored image produced by Kuhn’s mass-spring optimization technique trapped in a local minimum. (d) Recolored result obtained with the exaggerated-contrast version of Kuhn’s technique.
illustrates the case in which the set of constraints causes the optimization to be trapped in a local minimum. Figure 4b
shows the simulated view of a protanope for the image shown in Figure 4a
. Figure 4c
shows the recolored version of Figure 4a
using Kuhn’s technique. Note that although it contains some dark blue shades, it exhibits significantly less contrast than the original image. To handle these situations, we can use a variant of the technique that exaggerates color contrast instead. It essentially modifies the aforementioned algorithm by scaling the springs’ rest length by a given factor α (typically
). Figure 4d
shows the result obtained with this variant of the algorithm applied to Figure 4a
Evaluation by Participants with CVD
Both variants of Kuhn’s technique were evaluated by a group of 14 male volunteers with CVD. For the experiment, we used three sets of images. The first set ( Natural
) consisted of photographs of natural scenes. The second set ( InfoVis
) contained information visualization examples, while the third one ( SciVis
) consisted of scientific and medical visualization images. The original and recolored versions of these images were evaluated by the volunteers using the method of paired comparisons. 11
The results of the evaluation indicated that for the InfoVis set, the volunteers preferred the recolored images obtained with Kuhn’s technique. For the SciVis set, the volunteers preferred the recolored images with exaggerated contrast (as they can avoid the local minima situations depicted in Figure 4c
). An interesting outcome of this experiment was the fact that for the Natural set, the volunteers preferred the original images over the recolored ones, despite the acknowledged enhancement in contrast in the latter (see Figure 5c
). According to the participants, the colors in these images didn’t match their previous experiences and, as such, didn’t look “natural.” The results of this experiment suggest that recoloring techniques for CVD can be effective for visualization applications, as the color scales used to encode information tend to be somehow arbitrary. A description of the experiment and its results are detailed elsewhere. 9
Figure 5.. Example of a recolored natural image. (a) Reference image. (b) Simulated view of a deuteranope for the reference image. (c) Recolored image using Kuhn’s technique.
As previously mentioned, a dichromat’s color perception can be estimated by orthographically projecting the image colors onto the plane in Figure 2
that approximates the dichromat’s color gamut. Thus, in previous work, my colleague and I 10
introduced an efficient recoloring technique based on the observation that whenever dichromats experience significant loss of color contrast, most of it can be recovered by orthographically projecting the original colors onto a plane π (in the L*a*b* color space and containing the L* axis) that maximizes (in a least-squares sense) the projected colors’ separation. Using this approach, recoloring simply consists of using the projected coordinates on plane π as the new color coordinates in the dichromat’s gamut plane. Figure 6
illustrates this process.
Figure 6.. Projection-based recoloring technique. (a) The perception of a protanope for colors C1 to C3 have little contrast (represented by the yellow dots C1’ to C3’). (b) Orthographic projection of the original colors onto a plane that maximizes color separation. (c) Recoloring consists of using the coordinates of these projections on the dichromat’s original plane. This is equivalent to rotating the plane shown in (b) to align it with the dichromat’s plane.
This technique is quite efficient, achieving real-time performance and avoiding the local minima issues associated with optimization-based approaches. The technique also supports temporal coherence (the details of which are explained elsewhere 10
). As such, it satisfies all the desirable features, except for preservation of original colors (although it preserves gray shades). Perceptual differences are partially preserved. Figure 7
shows an example of images recolored using this technique.
Figure 7.. Image recoloring using a projection-based approach. 10 (a) Reference image. (b) Simulated view of a protanope for the reference image. (c) Recolored image.
Because of its real-time performance, such a technique can be easily integrated with any existing visualization application in a minimally invasive way. This is achieved as follows: once the application has written an image to the frame buffer, it calls a function that implements the recoloring algorithm. Such a function then reads the frame buffer’s content, recolors it, and writes it back, returning the control to the application. The application then swaps the front and back buffers. Using this approach, it becomes practical to perform real-time recoloring for dichromats with temporal coherence during visualization sections. Figure 8
illustrates the concept.
Figure 8.. Real-time recoloring for dichromats during a visualization section. (a) Reference frame. (b) Recolored frame. (c) A simulated view of a deuteranope for the reference image, for comparison.
Using Patterns to Encode Color Information
An alternative to recoloring is to superimpose patterns on colored image regions. Properly designed patterns allow individuals with CVD not only to disambiguate but also to identify colors, by memorizing the rules of pattern formation. 12
Its use, however, trades spatial resolution for the patterns. Thus, it becomes hard to use them in small areas. Moreover, applying patterns over regions containing textures or detailed content whose colors change dynamically, such as in the case of exploratory visualization, might be distracting and mask some important data features. Nevertheless, the use of patterns can be useful for disambiguating static content, and for communicating color information among normal trichromats and individuals with CVD. 12 , 13 Figure 9
illustrates this notion. Figures 9a
show a colored pie chart and how it would be perceived by a protanope, respectively. Note that in Figure 9b
it’s impossible to distinguish between sales in North America and Europe, as well as between sales in Australia and South America. By superimposing (on the original chart) patterns whose orientations and contrast levels uniquely represent colors in a meaningful way ( Figure 9c
), we can solve the ambiguity problem ( Figure 9d
) and allow proper communication between normal color vision and CVD individuals. Note that the original colors remained unchanged for normal trichromats ( Figure 9c
). A detailed description of how to create such patterns is provided elsewhere. 12
Figure 9.. Using patterns to solve ambiguity. (a) Reference chart. (b) Simulated view of a protanope for the reference chart. (c) Reference chart with superimposed patterns. (d) Simulated view of a protanope for the chart in (c).
Designing More Effective Visualizations for CVD
In previous sections, we discussed strategies aiming to minimize the loss of color contrast experienced by individuals with color vision deficiency. Here, we address the issue of assisting visualization designers in understanding how individuals with CVD perceive colors. This should help designers define more effective color scales and transfer functions to reach broader audiences.
Gustavo Machado, Leandro Fernandes, and I introduced a physiologically based model for simulating color perception that can handle normal color vision, anomalous trichromacy, and dichromacy in a unified and consistent way. 14
For anomalous trichromacy, it can simulate spectral sensitivity shifts of the affected photopigments in the 0- to 20-nm range. A shift of 20 nm results in a perception similar to the one of the corresponding class of dichromacy.
Controlled experiments involving a group of 13 individuals with CVD (4 protanopes, 4 protanomalous, 2 deuteranopes, and 3 deuteranomalous) and 17 normal trichromats have shown that the results produced by this model closely match the perception of people with CVD. Figure 10
shows some simulated results for protanomaly produced using this model. Note the gradual loss of color contrast as the amount of shift in the spectral sensitivity increases. Figure 11
shows similar results for deuteranomaly.
Figure 10.. Simulation of protanomaly. The numbers in parenthesis represent the amount of shift in the spectral response of the photopigment more sensitive to the long wavelengths. A shift of 20 nm results in a perception similar to the one of a protanope.
Figure 11.. Simulation of deuteranomaly. The numbers in parenthesis represent the amount of shift in the spectral response of the photopigment more sensitive to the medium wavelengths. A shift of 20 nm results in a perception similar to that of a deuteranope.
The simulation process can be implemented as a single matrix-vector multiplication per pixel. A detailed description of how to compute the matrix required for any CVD-severity degree is presented elsewhere. 14
Such matrices can also be obtained by linear interpolation from a set of precomputed matrices available at www.inf.ufrgs.br/~oliveira/pubs_files/CVD_Simulation/CVD_Simulation.html.
Given its simple implementation and real-time performance, this color-perception simulation model can be easily integrated with any visualization application. The visualization designer can then interactively select the class of CVD (s)he would like to simulate (such as protanopia, deuteranopia, protanomaly, or deuteranomaly), getting instant visual feedback. Figure 12
shows the visible male’s head, with a given transfer function, seen by a normal trichromat, a protanomalous (10 nm), and a protanope. The designer can switch between normal color vision and the simulation of any degree of CVD instantly. This feedback allows her/him to interactively refine the transfer function to reach a broader audience.
Figure 12.. Integration of the color-perception simulation model with a visualization system. The visualization designer can instantly evaluate the perception of CVD individuals and refine the transfer function to effectively reach a broader audience.
Color vision deficiency affects approximately 200 million individuals worldwide, compromising their ability to effectively perform color-related tasks and, therefore, fully exploit the benefits of visualization. Here, we’ve considered some of the techniques used for improving visualization experiences for individuals with CVD, including recoloring, pattern superposition, and the use of a color-perception simulation model for assisting visualization designers. Although all of these techniques have limitations, their judicious use can significantly improve visualization experiences for individuals with CVD.
I would like to thank the many volunteers with CVD who tested the techniques described here, providing invaluable insights and feedback. Several people contributed to the development and evaluation of these techniques, in particular Gustavo Machado, Giovane Kuhn, and Leandro Fernandes. This work was supported by Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) and Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) grants (305613/2007-3, 476954/2008-8, 305613/2007-3, 142627/2007-0, 131327/ 2008-9, 200284/2009-6, and 308936/2010-8). Figures 4a and 7a were rendered using software developed by Francisco Pinto. Figure 5a is from other work, 15 and Figures 3a , 10a , and 11a are from http://commons.wikimedia.org. Figure 9 is by Aditi Majumder
Manuel M. Oliveira is an associate professor of computer science at the Federal University of Rio Grande do Sul (UFRGS), in Brazil. His research interests cover most aspects of computer graphics, but especially the frontiers among graphics, image processing, and vision (both human and machine). Oliveira has a PhD from the University of North Carolina at Chapel Hill. Contact him at email@example.com.