This Article 
 Bibliographic References 
 Add to: 
Digital Paint Systems: An Anecdotal and Historical Overview
April-June 2001 (vol. 23 no. 2)
pp. 4-30

The history of digital paint systems derives from many things—chance meetings, coincidences and boredom, artistic license, brilliant researchers, a wealthy benefactor, and, of course, lawsuits. Alvy Ray Smith tells the fascinating story—facts first, then anecdotes—in his own words.

1. A.R. Smith, Digital Paint Systems—Historical Overview, tech. memo 14, Microsoft Corp., Redmond, Wash., May 1997. Essentially an earlier submission to AMPAS, which used it in granting a technical Academy Award—the Scientific and Engineering Award—to R. Shoup, A.R. Smith, and T. Porter in Feb. 1998.
2. A.R. Smith, Painting Tutorial Notes, tech. memo 38, Lucasfilm, Marin County, Calif., Apr. 1982. Includes Digital Paint Systems Survey fromComputer Graphics World, Apr. 1982, pp. 62-65. These were the tutorial notes for Siggraph 82.
3. B. Robertson, "Paint Systems," Computer Graphics World, vol. 11, no. 4, Apr. 1988, pp. 62-68. A good late 1980s survey article.
4. M. Hiltzik, Dealers of Lightning: Xerox PARC and the Dawn of the Computer Age, Harper Collins, New York, 1999.
5. G. Lavendel, A Decade of Research, Xerox Palo Alto Research Center, 1970-1980, R.R. Bowker Co., New York, 1980. An impressive record of accomplishment, presented as a collection of published papers.
6. G.E. Pake, "Research at Xerox PARC: A Founder's Assessment," IEEE Spectrum, vol. 22, no. 10, Oct. 1985, pp. 54-61. PARC from the management's point of view—compare with Perry and Wallich, "Inside the PARC," below.
7. T.S. Perry and P. Wallich, "Inside the PARC: The 'Information Architects,'" IEEE Spectrum, vol. 22, no. 10, Oct. 1985, pp. 62-75. PARC from the researchers' point of view—cf, Ref. 6. Contains an interesting graphic—the Silicon Valley family tree rooted in PARC.
8. D.K. Smith and R.C. Alexander, Fumbling the Future, How Xerox Invented, Then Ignored, the First Personal Computer.New York: William Morrow&Co., 1988.
9. I believe that one of the reasons Pixar (and related endeavors) has been so successful is that we fully pursued both halves of the computer "picturing" world with equal tenacity.
10. It should be noted that the two worlds intersect in some places, the most notable being in so-called texture mapping, where a digital image, often painted, is "wrapped" onto a 3D geometric surface to give it the appearance of reality. A very hot topic of current research is so-called image-based rendering, which intimately combines the two realms.
11. E. Catmull, T. Porter, T. Duff, and I received a technical Academy Award, the Scientific and Engineering Award of AMPAS in Feb. 1996 for inventing the alpha channel.
12. J.E. Miller personal communication, Bell Labs, Murray Hill, N.J., July 1978.
13. W.J. Kubitz and W.J. Poppelbaum, "The Tricolor Cartograph: A Display System with Automatic Coloring Capabilities," Information Display, vol. 6, Nov.-Dec. 1969, pp. 76-79.
14. R.G. Shoup, Old Software on the Color Video System, Xerox PARC memo, Feb.4 1975. Contains a complete manual for the system.
15. R.G. Shoup, SUPERPAINT Program, Xerox PARC memo, Feb.5 1975. First use of the term SUPERPAINT (that I write as SuperPaint here).
16. R.G. Shoup, "Some Experiments in Television Graphics and Animation Using a Digital Image Memory," SMPTE 13th Television Conf., San Francisco, Feb. 1979. Published inDigital Video, vol. 2, Soc. of Motion Picture and Television Engineers, Scarsdale, N.Y., Mar. 1979, pp. 88-98. Republished inDatamation, vol. 25, no. 5, May 1979, pp. 150-156.
17. It had a fixed palette of 6×6×6 colors displayed at the bottom of the screen, variable-sized rectangular brushes, and could save or restore images to or from files; Jim Blinn, personal communication, Microsoft Corp., Redmond, Wash., May 1997; also see R. Leavitt,Artist and Computer. Cover photo shows Duane Palyka painting digitally using Jim Blinn's Crayon program.
18. The night the E&S hardware first ran, in fact (Jim Blinn, personal communication, Microsoft Corp., Redmond, Wash., May 1997).
19. A.R. Smith, Paint, tech. memo 7, NYIT, July20 1978. Also issued as tutorial notes at Siggraphs 78-82. Reprinted inTutorial: Computer Graphics, 2nd ed., J.C. Beatty and K.S. Booth, eds., IEEE Computer Soc. Press, Los Alamitos, Calif., 1982, pp. 501-515. Also reprinted inSeminal Graphics: Pioneering Efforts that Shaped the Field, R. Wolfe, ed., a publication of ACM Siggraph, 1998, pp. 427-441. See Appendix C in particular. Contains a brief history of paint programs, including pre-Shoup experiments. First public mention of RGB paint, airbrushing, and so forth.
20. See R. Leavitt,Artist and Computer.
21. J. Blinn personal communication, Microsoft Corp., Redmond, Wash., May 1997.
22. Jim Blinn recalls that he had the program with him when he returned from a summer at NYIT (J. Blinn, personal communication, Microsoft Corp., Redmond, Wash., May 1997).
23. G. Stern personal communication, Spitz Inc., Chadds Ford, Pa., May 1997.
24. I have a dated notebook, from December 1976, covering the Ampex installation.
25. J. Blinn, Jim Blinn's Corner: Dirty Pixels, Morgan Kaufmann, San Francisco, Calif., 1998, pp. 131-135. In particular, see chapter 12,NYIT: How I Spent My Summer Vacation, 1976. This is his Sept. 1993 column fromIEEE Computer Graphics&Applications.
26. M. Levoy, "Frame Buffer Configurations for Paint Programs," tutorial for Siggraph 79, 1979. Revised in May 1980 and reissued as tutorial notes for Siggraphs 80-82.
27. N. Negroponte, "Return of the Sunday Painter," The Computer Age: A Twenty-Year View, M.L. Dertouzos and J. Moses, eds., MIT Press, Boston, Mass., 1979 (first paperback edition, 1980), pp. 21-37. Features one of my earliest color reproductions,Egg on Toast, at a higher resolution than most computer graphics images at that time. It would not fit into a frame buffer so I had to execute it by frame-buffer-size pieces.
28. N. Negroponte, "Raster Scan Approaches to Computer Graphics," Computers and Graphics, vol. 2, no. 3, 1977, pp. 179-193.
29. The earliest dated documentation I have for this code is dated 13 January 1978. Ed was preparing for Siggraph 78. Siggraph typically has a paper-due date of early January of the corresponding year, so this is probably about when the invention actually occurred, although it might have happened in December 1977, to avoid the last-minute crunch against the paper deadline. I was also preparing a paper for Siggraph 78. The date on the submission is 6 January 1978, and the code I used to generate figures for the paper is dated 28 December 1977.
30. A.R. Smith, Alpha and the History of Digital Compositing, tech. memo 7, Microsoft Corp., Redmond, Wash., Aug. 1995. Essentially an earlier submission to AMPAS, which used it in granting a technical Academy Award—the Scientific and Engineering Award—to A.R. Smith, E. Catmull, T. Porter, and T. Duff in Feb. 1996.
31. A.R. Smith, "Table Paint," Tutorial Notes, Jet Propulsion Laboratory, California Inst. Tech nology, Pasadena, Calif., Oct. 1979. Also issued as tutorial notes at Siggraphs in 1980 and 1981.
32. It had a specialized color map structure that divided the available bit depth intoibits for border colors,jbits for infill colors, andkbits for the antialiased transitions between these colors,i+j+k= 8; Marc Levoy, personal communication, Stanford Univ., Palo Alto, Calif., May 1997.
33. M. Levoy personal communication, Stanford Univ., Palo Alto, Calif., May 1997.
34. The first commercial paint product. Shown on show floor of National Association of Broadcasters (NAB) convention in March or April 1980. Quantel Paintbox was shown in a suite (see 1980s entry inTable 1high res imagein the main text).
35. T. Porter, NAB Trip Report, Lucasfilm memo, San Rafael, Calif.,17 Apr. 1981.
36. A.R. Smith, "Special Effects for Star Trek II: The Genesis Demo, Instant Evolution with Computer Graphics," American Cinematographer, vol. 63, no. 10, Oct. 1982, pp. 1038-1039 and 1048-1050. Mentions Tom Porter and his paint program.
37. T. Porter, Picture Coding and the Paint System, Lucasfilm memo, San Rafael, Calif.,13 Feb. 1981.
38. T. Porter, Picture Handling Using Staging Areas, Lucasfilm memo, San Rafael, Calif.,17 Feb. 1981.
39. T. Porter, The Paint System Design, 1st Pass Technical Memo, Lucasfilm memo, San Rafael, Calif.,18 Mar. 1981.
40. John still works for Lucasfilm. I understand that Thomas brought a background in image processing to bear on the product as well.
41. I do not know the original name for Photoshop.
42. R.L. Phillips, "Computer Graphics in Court: The Adobe/Quantel Case," Computer Graphics, vol. 32, no. 3, Aug. 1998; also available online at.
43. Jim Blinn recalls that Garland's program supplanted his own at Utah, after Garland's return from NYIT, because of a larger feature set; J.Blinn, personal communication, Microsoft, May 1997. Both Garland and Jim recall that the two programs were separate developments; J. Blinn, personal communication, Microsoft, May 1997; also, G. Stern, personal communication, Spitz Inc., Chadds Ford, Pa., May 1997.
44. J. Rosebush, Computer Animation: An Historical Survey, thesis, Syracuse Univ., School of Public Comm., Syracuse, N.Y., Jan. 1979. "Prepared to fulfill the requirements of third comprehensive examination" (This came from the title page of the thesis). This earlier history was never published. Contains a frame buffer history. Judson did, however, contribute significantly as a writer to the recent documentary video,The Story of Computer Graphics, premiered at the 26th Siggraph, Los Angeles, Calif., 1999.
45. I installed Paint at Ampex in December 1976, working with Junaid Sheikh (who would suffer from the Quantel patents in a later entrepreneurial effort).
46. In 1977, Ephraim combined frame-buffer-to-frame-buffer and file-to-frame-buffer copy and composite routines into a general-purpose Copy program. In 1978, he added tablet control to this program to get painting, which inherited 8-bit or 24-bit mode, antialiasing, airbrushing, and so on from the earlier Copy program. In 1978-1979, he fleshed this out into his ept painting system; E. Cohen, personal communication, R/Greenberg Associates, New York, May 1997.
47. I had also noted this. See Smith, "Table Paint," tutorial notes. Dick Shoup based his Aurora products on this concept, too; R.G. Shoup, personal communication, Interval Research, Palo Alto, Calif., May 1997.
48. Dick actually called his environment the Color Video System, including a paint program called Paint; R.G. Shoup,Simple-Minded Animation on the Color Video System, Xerox PARC memo, 9 Aug. 1974. Dick formally proposes to hire me in this memo, reporting on several animations that I had already created on the system. Also, R.G. Shoup,Old Software on the Color Video System, Xerox PARC memo, 4 Feb. 1975. Contains a complete manual for the system. He added the SuperPaint program to the system in Feb. 1975; R.G. Shoup,SUPERPAINT Program, Xerox PARC memo, 5 Feb. 1975. First use of the term SuperPaint (I refer to the entire system as SuperPaint for convenience.) The complete system is in the permanent collection of the Computer Museum and is still (occasionally) functional.
49. T. Porter personal communication, Pixar, Richmond, Calif., May 1997.
50. M. Gardner, Wheels, Life and Other Mathematical Amusements, W.H. Freeman, New York, 1983. Contains reprints of all his "Game of Life" writings from Scientific American.
51. R.C. Minnick, J.C. Huang, R.G. Shoup, and A.R. Smith, "Cellular Logic," Hardware, Software, Firmware Trade-offs: Proc. 1971 IEEE Int'l Computer Soc. Conf., IEEE Press, New York, 1971, pp. 25-30.
52. J. von Neumann, Theory of Self-Reproducing Automata, A.W. Burks, ed., Univ.of Illinois Press, Urbana, 1968. Completed after von Neumann's death.
53. This edition of the book never reached final publication although I was paid for my contribution, which eventually became my "Introduction and Survey of Polyautomata Theory," article published inAutomata, Languages, Development, A. Lindenmayer and G. Rozenberg, eds., North-Holland Publishing Co., New York, 1976, pp. 405-422. This served as the proceedings of an international conference on "Formal Languages, Automata, and Development," Noordwijkerhout, the Netherlands, Apr. 1975. Artificial Life 0. A complete survey of CA (cellular automata) theory. For nontrivial, self-reproducing CA, see A.R. Smith, "Simple Nontrivial Self-Reproducing Machines,"Artificial Life II, C.G. Langston et al., eds., Addison-Wesley, Reading, Mass., 1992, pp. 709-725. These were the proceedings of the "Workshop on Artificial Life" held Feb. 1990, Santa Fe, N.M. Based on my 1969 PhD dissertation work.
54. R.G. Shoup, Simple-Minded Animation on the Color Video System, Xerox PARC memo, Aug.9 1974.
55. Short for picture element. Others, notably IBM and AT&T, triedpel, a term never adopted by the computer graphics community. A pixel is commonly three or four bytes. Frame buffers and digitized images are often measured in mega- or gigapixels. Three bytes, three times eight bits, can hold 224= 16,777,216 values, or colors in the case of a pixel—that's 16 megacolors.
56. The Shoup frame buffer (picture memory) is only conceptually 2D. For economic reasons, he implemented it with shift-register memory chips, which means that a given pixel has to be shifted, one pixel at a time, out of the chips to get access to it. It is organized as a single, long shift register with counters that keep track of what line and pixel you are on. The now-popular random access memory (RAM) chip, allowing direct access to a given pixel, became cheap enough only later. In either case, software techniques disguise the underlying hardware and make it appear as a 2D array. The first dense-enough RAM chips (Intel 1103) available when Shoup made his design decision were twice as expensive and half the density of the shift-register chips. The second frame buffer he planned to build, but never did, would have been RAM based.
57. If you look closely at a color TV screen, you can see the tiny red, green, and blue phosphor dots that glow at different intensities to make colored light. The television monitor mentioned is one used in a broadcast studio rather than at home, although a home set works as well. The clean TV signal at a studio deteriorates by the time it reaches a home receiver. The Shoup frame buffer generated "broadcast quality" video adhering to the NTSC (National Television Standards Committee) standards for TV in the US.
58. SuperPaint and most paint programs intended for use by professionals actually employ a stylus on an electronic tablet instead of a mouse, use of which is likened to painting with a bar of soap.
59. This does not mean that each phosphor dot or RGB (red, green, blue) triad of dots corresponds exactly to a single pixel. To anyone who has changed the screen resolution of a computer monitor from, say, 640×480 to 1024×768, this should be obvious. The number of screen phosphors remains constant while the number of pixels displayed increases a great deal.
60. Arguably the first in the sense that the Shoup frame buffer was the first serious one used for interactive graphics, there being earlier experiments—for example, a 3-bit (8-color) frame buffer at Bell Labs (see Appendix C ofSeminal Graphics: Pioneering Efforts that Shaped the Field, R. Wolfe, ed., ACM Press, New York, 1998). There were undoubtedly image memories in use by the military at this time, but we have little information about them, such as whether they were interactive. Some historians and patent attorneys draw a distinction between Shoup's picture memory, based on shift-register chips, and the later E&S frame buffer, built from RAM chips. This is a minor distinction. From a programmer's viewpoint, either can be addressed randomly, although the implementation is certainly slower and clumsier for shift-register memory than RAM.
61. HSV is later institutionalized in the PostScript language—renamed HSB (B for brightness)—at the company Adobe Systems by its founder, John Warnock, another graduate of PARC (and the University of Utah before that); and see A.R. Smith, "Color Gamut Transform Pairs,"Computer Graphics, vol. 12, no. 3, Aug. 1978, pp. 12-19 (Siggraph 78 Conf. Proc.); reprinted inTutorial: Computer Graphics, second edition, J.C. Beatty and K.S. Booth, eds., IEEE Computer Soc. Press, Los Alamitos, Calif., 1982, pp. 376-383. Discusses the HSV algorithm.
62. A.R. Smith, Vidbits (video), Xerox PARC, 1974. Exhibited at the Museum of Modern Art, New York, 1975, and on WNET television show VTR, New York, 1975.
63. I'm particularly tickled by theirCadillac Ranch"sculpture" outside Amarillo, Texas, about 100 miles from my hometown. It consists of about ten of the biggest tail-fin Cadillacs ever manufactured in Detroit, of successive years, buried nose first in a row in the middle of a wheat field.
64. W.M. Newman and R.F. Sproull, Principles of Interactive Computer Graphics, 2nd ed., McGraw-Hill, New York, 1979. Jack Bresenham is currently a professor of computer science at Winthrop University. He retired from IBM as a senior technical staff member in 1987 after 27 years. He is best known for fast line and circle algorithms.
65. S. Brand, "Fanatic Life and Symbolic Death among the Computer Bums," Rolling Stone, Dec.7 1972. Describes the climactic culture clash between the California and New York branches of Xerox over PARC.
66. From Shoup quote in T.S. Perry and P. Wallich, "Inside the PARC: The 'Information Architects,'" IEEE Spectrum.
67. J.T. Kajiya, I.E. Sutherland, and E.C. Cheadle, "A Random-Access Frame Buffer," Proc. IEEE Conf. Computer Graphics, Pattern Recognition, and Data Structure, IEEE Press, New York, 1975, pp. 1-6. Describes the E&S frame buffer.
68. A. Schure quoted in "N.Y.I.T. Puts Computers to Work for TV," by L. Gartel and J.L. Streich, Millimeter, June 1981. The source of the famous quote.
69. I became familiar with BCPL at Xerox PARC and used it briefly at NYIT. The C language is a direct descendant of BCPL, the "B" to its "C". The "CPL" part evidently stands for Cambridge [University] Programming Language.
70. A lovely estate nearby NYIT with several houses for various members of the McGrath family, in-laws of David Rockefeller. David DiFrancesco and I, and later Garland Stern, shared the chauffeur's quarters over another four-car garage.
71. R. Adams, Watership Down, Avon Books, New York, 1975. From a review in The London Times: "I announce, with trembling pleasure, the appearance of a great story."
72. I did implement a crude approximation to this kind of airbrushing, too, with a brush of random dots clustered more densely in the center. See D. Em, TheArt of David Em: 100 Computer Paintings, text by D.A. Ross and D. Em, Harry N. Abrams Inc., New York, 1988.
73. E. E shwiller et al., Sunstone, video, created at NYIT, 1979.
74. Quantel Limited, Plaintiff, v. Adobe Systems Incorporated, Defendant, in the US District Court for the District of Delaware, before the Honorable Roderick R. McKelvie, Courtroom 4A, J. Caleb Boggs Federal Bldg., 844 King St., Wilmington, Del., Sept 1997. I testified on 12 Sept. 1997.
75. I had not yet received my second technical Academy Award, with Dick Shoup and Tom Porter, for "pioneering inventions in digital paint systems," awarded in 1998 (see Ref. 1).
76. R.G. Shoup and A.R. Smith, "1990 ACM Siggraph Awards: Computer Graphics Achievement Award," Computer Graphics, vol. 24, no. 4, Aug. 1990, pp. 17-18 (Siggraph 90 proceedings).

Alvy Ray Smith, "Digital Paint Systems: An Anecdotal and Historical Overview," IEEE Annals of the History of Computing, vol. 23, no. 2, pp. 4-30, April-June 2001, doi:10.1109/85.929908
Usage of this product signifies your acceptance of the Terms of Use.