Issue No.01 - January/February (2009 vol.26)
Published by the IEEE Computer Society
Hakan Erdogmus , National Research Council Canada
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/MS.2009.3
For the 2008 ICSE and Agile conferences, The magazine's editor in chief prepared a flash poll to gauge key perceptions regarding the separation between software engineering research and software development practice. The results for each conference were interesting, but the differences between the two groups were especially revealing.
The International Conference on Software Engineering and the Agile (formerly Agile/XP Universe) series have been two constants in my annual conference circuit since 2001. Both are excellent events in their own right, but I also enjoy them for the contrast they provide. ICSE is mainly an academic conference, the playground of applied software engineering researchers. Agile, on the other hand, is mainly a practitioner's event, the playground of developers, testers, software consultants, and, more recently, software project managers. I've always been intrigued by the differences in the two audiences' perspectives. Both perspectives are valuable, but I see the differences to be especially revealing.
For the 2008 ICSE and Agile meetings, I prepared a flash poll to gauge key perceptions regarding the separation between software engineering research and software development practice. I tweaked and transposed the questions as appropriate to suit each conference's profile. Armed with tablet PCs, two volunteers hit the hallways of the two conferences' venues on a pulse-checking mission.
The Conference Profiles
ICSE is the largest international, and one of the two top-rated, academic conferences in software engineering. It also features an industry track. With 2008 conference attendance at more than 1,100, we were able to survey 150 delegates, for a sampling rate of 13 percent.
Figure 1 shows our poll results. Of the respondents, 89 percent identified their role as researcher. Only 33 percent identified themselves as practitioners. These groups included the 23 percent of the respondents who identified themselves as both practitioners and researchers. Sixty-six percent had an academic affiliation, 23 percent had an industry affiliation, and 11 percent had a government affiliation.
Agile 2008 is predominantly a practitioner's conference, but it does feature research tracks, including technical papers and an emerging-results session. In 2008, attendance exceeded 1,700 delegates. Although the Agile series and its predecessors attract international participants, they have always been held in North America, so most of the delegates represent that continent.
Figure 2 shows the Agile 2008 poll results. We sampled 72 delegates, just over 4 percent of the total attendance. Although this number isn't quite above the magic 5 percent mark, it's still good enough for the results to be meaningful. The profile in terms of participants' roles is almost the inverse of ICSE, with 81 percent of the respondents identifying themselves as practitioners and 38 percent identifying as researchers. Again these groups overlapped, including the 22 percent of the respondents who identified themselves as both practitioners and researchers. Respondents' affiliations presented a somewhat more homogeneous picture than those of ICSE, with a distribution of 82 percent industry, 3 percent government, and only 11 percent academia.
In general, Agile attendees were more evenly divided in their opinions than ICSE attendees. For the first six questions (the ones on perceptions), clear majorities are altogether absent in the Agile version of the poll, whereas they're dominant in the ICSE version. So despite the fact that ICSE is more diverse in terms of the attendees' geographical origins, it appears to be more uniform than Agile in terms of cultural beliefs. Could this be because the research culture tends to be more uniform? If that's so, then perhaps culture trumps geographical diversity, and we shouldn't expect geographical diversity to skew the observations too much.
ICSE's 2008 motto was "Driving World Business." The poll used this motto both as an ice-breaker and to probe the delegates' confidence in their field's impact on industry. So, my volunteer asked the respondents whether they really thought software engineering was driving world business. The majority, 61 percent, happily agreed, and only 5 percent disagreed. However, a significant portion—one third of the respondents—weren't sure.
For the Agile version of the poll, I changed the first question to evaluate the delegates' interest in t-hat conference's research tracks. Remarkably, over one-third indicated that they were planning to attend or had already attended at least one research track. The rest either didn't know or didn't plan to, in equal proportions. What's even more encouraging is that the percentages don't change much when projected onto respondents who identified themselves as practitioners. There was anecdotal evidence of this cross-interest as well. Indeed, two research sessions that I observed were well attended, and a large preconference workshop dedicated to research in progress was jam packed and very lively. Research tracks seemed to bring variety and add value to the conference.
Who's Ahead, Who's Trailing?
The second question embodied my main motivation in conducting the polls: we asked respondents which community they thought was leading the pack. Fifty-three percent of ICSE respondents thought software engineering research is leading software practice, compared to 34 percent who thought otherwise. This result isn't too surprising to me given that ICSE is mainly a research conference. The percentages were similar for the group that contained researchers and the group that contained practitioners.
On the Agile side, 39 percent of the respondents thought software practice is leading software engineering research, and only 13 percent thought otherwise. Again, this is a rather unsurprising result. However, the responses aren't homogeneous between groups. The positive impression of software practice relative to software research increased from 39 percent to 52 percent when we restricted responses to those who identified themselves as researchers. Researchers who've been exposed to the other side or who straddle both worlds overall have a higher opinion of software practice. This observation is particularly telling, especially because the converse doesn't appear to hold for ICSE attendees: practitioners at ICSE are in a similar situation to researchers at Agile, but the practitioner group doesn't have a higher opinion of software research as leading the pack than the average ICSE attendee.
The Need for More Collaboration
I'd like to think that neither community is ahead per se relative to the other. Researchers and practitioners sometimes tackle different problems. And when they do tackle similar problems, they often address them at different levels, with different senses of urgency, with different methods, and most important, from different perspectives. Such diversity is good for solving complex problems. Thus, I prefer to believe that they have much to learn from each other, and this brings us to the inevitable subject of cross-collaboration. Not enough is happening on that front. And my own perceptions are echoed in the polls' results.
Respondents from both conferences who think that collaboration between software engineering research and industry is adequate are indeed a rather small minority, at 14 percent for ICSE and 13 percent for Agile. It's encouraging that both audiences acknowledge the need for more collaboration, but are they putting their feet where their mouths are? The acknowledgment is nice, but more widespread, real action would be even nicer.
Insufficient Mutual Awareness
In the same vein, admitting failure was also forthcoming as far as mutual awareness is concerned. The direction of the sentiment was homogeneous between the two conferences.
ICSE respondents who thought that the awareness level of software practice among software engineering researchers is inadequate outnumbered those who think it's adequate by nearly two to one. Interestingly, this proportion was lower—four to three—for Agile respondents, who apparently assumed a more even stance.
Conversely, ICSE respondents almost unanimously thought that practitioners are insufficiently aware of software engineering research. This group outnumbered those who thought otherwise by almost 10 to 1. Although the sentiment's direction was the same for Agile respondents, the proportions differed considerably. Agile respondents who thought that practitioners are insufficiently aware of software engineering research still outnumbered those who thought the opposite, but only by a ratio of less than two to one—again, a much less dramatic stance compared to ICSE.
The Importance of Cross-Participation
When asked about the level of industry participation in the conference, ICSE respondents took an overall neutral position, with only 25 percent believing the level was adequate, slightly more believing it was inadequate, and the rest remaining indifferent.
The response was similar when we asked the Agile attendees the converse question: 35 percent thought researchers' participation at Agile was adequate, 22 percent thought it was inadequate, and the rest were indifferent. This doesn't sound like a crowd cheering for more academic involvement in the conference.
So let's recap what people were saying: We need more collaboration. We need more mutual awareness. But we don't care about having more of the other side at our conference. Oh well.
In the sampled audiences overall, practitioners' impressions of software engineering research were less negatively biased than researchers' impressions of software development practice. More cross-pollination was deemed desirable in general, but whether anything is to be done about it wasn't clear to either audience.
Nonetheless, inaction doesn't suit the organizers who will shape the two conferences in the near future. Surely, both conferences will continue to deliver their usual high-quality content for their respective main audiences. On top of that, ICSE '09 is gearing up for a stronger-than-ever industry track, led by IEEE Software's very own Frances Paulisch and Philippe Kruchten. Reciprocally, expect similar efforts from Agile '09 for a beefed-up research program led by organizing committee members Tore Dyba, Yael Dubinsky, and Philippe Kruchten.
If you know how other software engineering and developer conferences encourage interaction between researchers and practitioners, please write to me at email@example.com.
I am grateful to volunteers Jennifer Schiller and Chandra Sreeraman for conducting the polls and to Frances Paulisch and Philippe Kruchten for their comments on an earlier version of this essay. Also, thanks to Frances Paulisch, Grigori Melnik, Rachel Davies, and Ahmed Sidky, who were members of the two conferences' organizing committees, for giving me permission to do the polls, providing logistic support, and securing Jennifer and Chandra's help.