The Community for Technology Leaders

We Have a Winner: Programming Contests Show Impressive Growth


Pages: pp. 99-103

Global software programming contests are expanding in number, size, and scope. Student programmers and, increasingly, their professional colleagues are taking the opportunity to "show their stuff" under the trying conditions of controlled competition. For the winners, the visibility they receive can be instrumental in jump-starting a career (besides the cash and merchandise prizes they might receive). For sponsors, the contests mean beneficial exposure to prospective employees, partners, and customers.

As the contests have outgrown single facilities, and as the private sector has joined the academic sector to fund and conduct these larger contests, skeptics have raised questions about their true utility and ultimate fairness—in both measuring and rewarding competitors' efforts. Also, empirical comparison of contests with different rules, regulations, and rewards has been difficult, as has any attempt to determine which model might be most beneficial to the most participants. In general, however, organizers say that these competitions provide intangible benefits that justify their proliferation. These benefits—meeting and competing against like-minded scholars and programmers, and learning the professional culture of the developer community—promote the community's sense of solidarity.

A must-have credential

According to one industry executive, programming contests have evolved into a must-have credential for developers.

"Good talent is expensive pretty much anywhere, so there's an awfully big investment made once somebody makes a hiring decision," says Jack Hughes, founder and chairman of TopCoder, a Glastonbury, Connecticut-based company that bases its business model on a "competitive community" approach to developing software. TopCoder holds ongoing competitions in which winners receive compensation for designing system components or top-performing algorithms. Any member of the TopCoder Community (which is free to join) may participate. The company runs development contests under its own banner as well as global sponsored contests such as Google's Code Jam and America Online's TopCoder Collegiate Challenge.

Hughes says the contests take some of the guesswork out of committing an investment in a new hire.

"Those folks doing the hiring are going to look for as many data points as possible to take the risk out of that decision, and that's why these contests are becoming more and more visible, and more people are entering them. There's probably not an order-of-magnitude difference in what a winning contestant can command in compensation, but I don't think it's too far off. Winning an Association of Computing Machinery competition, winning a TopCoder competition, is a write-your-own ticket kind of event, and that's a big deal for a lot of people."

The ACM granddaddy

The original student programming competition, the ACM's International Collegiate Programming Contest, traces its roots to a 1970 competition held by the Upsilon Pi Epsilon honor fraternity at Texas A&M University. Since 1997, the contest has grown by a factor of seven, from 840 teams representing 560 universities to 5,606 three-person teams representing 1,737 institutions.

In early 2007, 85 three-person teams that qualified in global regional competitions will converge for the ICPC world finals at Baylor University in Waco, Texas. The contest runs five hours, in which the teams face juried questions of varying difficulty. The contest's executive director, Baylor computer science professor William Poucher, says the ACM contest's cachet can indeed be at least partially attributed to its long history.

"Unlike the other competitions, we actually grew up with the disciplines of computer science and computer engineering," Poucher says. "The vast majority of people in the industry can't remember there not being the ACM International Collegiate Programming Contest. You could probably go to the typical research and development labs of major companies, and the developers, many of them, have a T-shirt from the ICPC world finals somewhere in their wardrobe."

As the discipline of computer science has grown, so too has the ICPC contestant pool and the difficulty of the programming tasks involved.

"The problems are harder than ever," Poucher says. "In computer science, the information base has doubled every five years since 1950, so there's more to learn, but the contestants are sharper than ever. It used to be the contest was six hours long on mainframes, and we wouldn't see any problems submitted until the fourth or fifth hour. Now, we have solutions at the world finals presented 10 or 15 minutes into the contest, and three-quarters of those contestants are not native English speakers."

In recent years, in fact, contestants from both the former Soviet bloc and China have performed extremely well. Two of the four gold-medal teams in the 2006 finals hailed from Russia and one from Poland. In the 2005 finals, two gold-medal teams once again came from Russia, and one came from China.

Poucher and TopCoder's Hughes ascribe the strong showing of Eastern European and Chinese programmers to several factors that dovetail well. Some of these factors include the collaborative global culture of academia, strong secondary educational system emphasis on science and technology in these countries, and the rise of Internet-enabled programming paradigms that eliminate physical distance as a barrier.

The IEEE upstart

One of the world's other giant engineering societies, the IEEE, is just beginning its own programming contest this year.

The IEEE's programming contest is a bit different from the ACM contest. Ricardo Varela, the IEEE's student representative from Region 8, which encompasses Europe, Africa, the Middle East, and Russia, says the contest was envisioned as a global activity for its student branches. Unlike the ACM contest, in which the finalists converge on one locale for a short, intense period of programming, the IEEE Xtreme 24-Hour Programming Contest will be conducted at IEEE student branches worldwide at the same time. Varela says the contest is tentatively scheduled for early December. The goals behind the contest, the IEEE says, are to provide student branches with a new and interesting activity; increase the number of activities that focus on the computer, programming, and information technology fields; and differ from other competitions and from other IEEE activities.

Varela says the contest also aims to increase the IEEE's visibility among technologically minded students of many disciplines. Doing so will not only strengthen the IEEE, but Varela says it might also broaden the base of programming skills in industry as recruiters familiar with the IEEE's solid reputation become aware of the new activity.

"One of our goals is to diversify a little bit more," he says. "The IEEE has gone past electrical engineering to cover so many fields, such as biomedicine, telecommunications, and computer science. We say the IEEE today really stands for 'the institute of engineers and everybody else!' And, when you look at those who are studying some of those other disciplines, there are some who are great coders even if they are not computer science people. What you want as a company is to hire the best coders, not necessarily computer science people, and diversifying the total spectrum of people who are participating in programming contests is going to be good. The IEEE can use its size as the largest computer science and engineering organization in the world to attract participants."

Varela says he will be happy to see 100 to 150 teams—somewhere between 300 and 500 people—take part in the inaugural Xtreme contest.

Keeping it private

While the ACM and IEEE contests strive to strengthen the organizations' ties to academia, the private sector has also become deeply involved in running global programming contests and in attempting to diversify beyond the campus and beyond programming. Google's Code Jam, for instance, is open to both student and nonstudent programmers; Anne Driscoll, Google's staffing-programs manager, says this year's registrants included 60 percent students and 40 percent nonstudents (with 85 percent of the finalists being students). She adds that this year's Code Jam attracted 20,000 contestants from 143 nations, up from 14,000 in 2005.

Microsoft has run its Imagine Cup contests annually since 2003. In 2004, the Imagine Cup attracted 10,000 participants; in 2006, it attracted 68,000 contestants. (The competition is open to high school and college students.) The competition has expanded from its original category of software design to nine categories, including short film, digital photography, and interface design. Imagine Cup director Emanuele Ognissanti says that, after the inaugural contest, students and educators asked Microsoft to include related fields that needed technology savvy but not necessarily programming skill. According to Ognissanti, although some might take the cynical view that Microsoft adds invitational categories "depending on what products we are introducing," the added categories actually lower the entry barriers by giving students a wider range of options in which to express themselves.

And, as high-level scripting languages and frameworks begin to make coding easier and more attractive to a wider audience, other sponsors are expanding their categories and core skill sets as well. TopCoder's Hughes says they decided to include Python in this year's Code Jam, although few contestants chose to use it.

"We want to cover the whole spectrum, from artificial intelligence and compiler theory all the way up to the scripting languages, Ajax, and so on," Hughes says.

One critical facet of the industry-sponsored contests is the emphasis on recruitment and networking. Cross-pollination already exists between the academic and company-sponsored contests, and between competitors who follow each other's contests. For example, the IEEE's Varela, who works for Google, has interviewed contestants at Code Jam, and Microsoft's Ognissanti says Google made an offer to one of the Imagine Cup's top teams after this year's finals.

TopCoder's Hughes says the objective criteria by which contests compel students to perform could well provide some compulsory, objective evaluations for those seeking employment with leading firms.

"I think it started out with the hardcore computer science guys who got in there for the fun of it," he says. "But, as employers have started asking more and more things like, 'Do you have a TopCoder rating? Do you have contest participation?' they [students] are seeing it now as it has to be part of their CV if they want to get a job at Microsoft, Google, or what have you."

Voices of caution

While contest promoters speak glowingly of the spirit of camaraderie and community inherent in the competitions, others have voiced some concerns.

One example appeared in a recent edition of the journal Informatics in Education. Researchers from the University of Waterloo—a perennial top-performing university at the ICPC—question prevailing assumptions about algorithmic contests' broad-based utility in strengthening the computer science community:

Algorithmic programming contests in the style of ICPC, IOI and TopCoder emphasize a specific subset of the skills and techniques that comprise informatics. Are these the most important skills, or simply those that are easiest to measure? Are these timed contests attractive [to] a broad population of potential participants, or do they select a narrow demographic? Are the contests appealing to an audience beyond the immediate participants and their acquaintances? Do the contests accurately convey—to students, educators, and the public at large—the nature of informatics? ( informatics_in_education/htm/ INFE071.htm)

In a less scholarly vein, Kevin Shockey, a blogger for the O'Reilly Network, takes Microsoft to task for a legal notice it issued for the Imagine Cup( onlamp/blog/2005/11/stark_ criticism_of_microsofts.html). Specifically, he criticizes the requirement that contestants must sign over to Microsoft the right to use, for any purpose, intellectual property, including ideas, concepts, know-how, or techniques, contained in contest applications. Shockey writes that this would let Microsoft lift student ideas for nothing. Microsoft's Ognissanti, however, says the vast number of ideas that cross the company's transom in the course of the contest could lead to specious litigation. A student might claim that a Microsoft product was based on an Imagine Cup idea, even though the concept was created coincidentally and separately.

The intellectual property issues in Code Jam could be even more problematic for contestants who wish to retain rights to their work. That contest's rules state in part that those who win and redeem a prize must license to TopCoder and Google rights to all information submitted during the tournament, including rights to source code and other executables ( pl/?module=Static&d1=google 06&d2=rules).

In essence, Ognissanti says, clauses such as these serve to introduce students to the often labyrinthine world of IP rights and rewards. He cited a Croatian Imagine Cup team that created a first-aid application and then donated the rights to local hospitals.

"They got so much good PR for that," he says.


Ultimately, says the ICPC's Poucher, you should consider contests in the same vein as world-class athletic competitions, in that top competitors must dedicate themselves to the pursuit of total excellence in ways that go beyond bare coding ability.

"It's not a matter of finding companies looking to hire a number of folks," Poucher says. "It's an athletic activity in this sense: An athlete who trains to run the 100-meter dash doesn't just try to run fast. That athlete pays attention to nutrition, and proper rest, and the most effective techniques as discovered by his or her colleagues around the world. At the end of the day, their entire way of life changes for the better, and that's what we want to see in ICPC contestants."

Given the fast-growing ranks of programming contestants around the world, it's also evident that many young technologists want to make those changes for the better themselves.

Funny Business, Serious Business: O'Reilly OSCON 2006

Bart Massey

The 2006 O'Reilly Open Source Conference, held in Portland in July, gave me a lot to digest—no, no, not just the excellent complimentary food. I left the conference chewing over ideas about the relationships between open source software developers, their relationships with the business community, and the relationship of the entire technology complex to society.

The conference—larger than ever—was, as always, a great deal of fun. There were about 2,700 participants this year, according to OSCON PR person Suzanne Axtell, up from about 2,100 in 2005. OSCON appears to be meeting its goal of rapid growth quite handily.

Developers et al.

The 2006 mix of attendees seemed much more skewed than in previous years, away from open source developers and technical types. There was a wide mixture of open source corporate representatives, industry managers coming up to speed on open source, and folks from the wider community satisfying their curiosity about the whole phenomenon.

Therein lies an interesting dilemma. For OSCON to grow and to have broad impact, it must cater to more than just the technically focused developer community. However, to be a serious conference instead of a glorified trade show, enough of the developer community must participate to add substance and depth. With the decline of other open-source-developer events in the US—for example, the end of the Freenix track of the Usenix Annual Technical Conference —OSCON fills an important role as something of a US counterpart to events such as the Australian

Furthermore, the open source marketplace's continued maturation has brought in a second wave of open source entrepreneurs. These entrepreneurs are drawn mainly from the proprietary arena and seem more interested in pursuing their existing strategies than in the open source ideals and community. Unfortunately, preexisting strategies often fare poorly in the open source arena. It is, after all, a peculiar place, with its own culture, mores, and vision.

Impressions and highlights

The conference keynotes were of a generally high quality. The talks I enjoyed most featured humorous insights. Damian Conway's two talks, "The DaVinci Codebase" and "openTalk 2.0," were tour-de-force commentaries. Robert Lefkowitz's "5 A Day" was a fine piece of comic analysis. More serious keynote fare included Carl Malamud on influencing change at the federal level, and original IBM PC team member Dave Bradley on "25 Years of the IBM PC." The tutorial and technical sessions ranged from low-level Linux to high-level legal guidance. (For more on the presentations, see os2006.)

The exhibit floor this year captured most of the major industry players with a stake in open source, including Novell, Intel, HP, and Google. IBM donated its exhibit space to showcase open source research at Oregon's two main university centers: Oregon State University's Open Source Laboratory and several of my open software/hardware student groups at Portland State University (the Portland State Aerospace Society, the Software-Defined Radio team, and our X Window System developers). Sun was also there, continuing last year's efforts to promote OpenSolaris.

Much of the rest of the exhibit space was taken up by start-ups making big OSCON publicity pushes. Many start-ups, such as Greenplum, Laszlo Systems, and Shopzilla, reminded me a lot of dot-com-era companies, but with important twists: their products and business plans were clear and understandable, their marketing was fairly mainstream and professional looking, and the level of risk assumed was low.

The OSCON "hallway track" was active as always. I ran into many old friends and famous faces in the corridors. The usual high level of hospitality offered by OSCON, including such amenities as inflatable lounge chairs in the session areas and continuously available free snacks in the exhibit area, greatly facilitated these interactions.

OSCON has long had the tradition of folks holding concurrent conferences and activities, typically at no cost to participants. This year, O'Reilly provided on-site space for more of them than ever (hosting, in addition, colocated conferences on Perl, PHP, and Python). One of the more interesting side events was OSCamp, which Brandon Saunders organized to explore new ideas in open source. OSCamp was run in the Open Space self-organizing format ( Although the event was sparsely attended, the Open Space format was a success.

The "Boys' Club"

The open source community has received a lot of criticism for its high gender imbalance, even compared to the rest of the software industry. OSCON definitely reflected this imbalance; while no precise registration numbers were available when I last chatted with Suzanne Axtell, my impression was that only five to ten percent of the attendees were female.

Sadly, a new development at OSCON 2006 might exacerbate this problem. Several presenters' slides sported sexualized images of semiclothed women. In addition, there was a "booth babe": an attractive young woman in a harem outfit wandering around the exhibit hall handing out grapes. Of the several women attendees I asked about all this, about half found it uncomfortable—one so much so that she felt inclined not to attend again.

I'm hopeful that this development was an anomaly. Axtell suggested that, while some steps will be taken to prevent a repeat at OSCON next year, there's only a limited amount O'Reilly can do to control the speakers' content without damaging the conference.

It seems to me that this situation arises from the confluence of several streams of change in the open source world. The mainstream marketers are entering the game, and they tend to use sex as a selling tool. As the open source development community expands, it absorbs an ever-larger percentage of the young, overwhelmingly male enthusiast community. Internet culture itself is highly sexualized, and this aspect can't help but seep into the Internet-focused technical community. Given some careful thought and hard work, hopefully these factors can be controlled and open source can be made a friendlier place for women.

Open source future

My report on OSCON 2005 described it as "entering the mainstream." At OSCON 2006 I began to get a clearer picture of that mainstream and where it might lead. It seems unavoidable that the pressure of the larger business and social community will change open source, sometimes for the worse. At the same time, peaceful coexistence seems possible in the near term, and relatively benign integration farther out.

I've watched with fascination over the past 20 years as open source has grown from hobbyist sharing of programs to major industrial significance. As we approach the cusp of open source adoption, I view with even more fascination the future stretching out before us.

Contest Home Pages

2006 Computer Society International Design Competition,

ACM International Collegiate Programming Contest,

Google Code Jam,

Imagine Cup,


About the Authors

Bart Massey is an associate professor of computer science at Portland State University, where he teaches and researches open source, software engineering, and AI. Contact him at
58 ms
(Ver 3.x)