January/February 2009 (Vol. 26, No. 1) pp. 9–11
0740-7459/09/$31.00 © 2009 IEEE
Published by the IEEE Computer Society
Published by the IEEE Computer Society
IEEE Software's 25th-Anniversary Top Picks
|About the Selection Process|
PDFs Require Adobe Acrobat
Over the past 25 years, from 1984 through 2008, IEEE Software published more than 1,200 peer-reviewed full-length articles. As part of our 25th-anniversary celebration, Software's editorial and advisory boards embarked upon the ambitious task of distilling those articles into a compact list of recommended reading.
About the Selection Process
To select the articles, we relied on three sources: nominations, Web analytics, and citation statistics. We invited the members of Software's boards as well as former editors in chief to nominate their favorite full-length, peer-reviewed articles that appeared either in a special focus section or as a standalone piece. We excluded from the competition non-peer-reviewed content such as columns, short essays, news articles, letters, point-counterpoint pieces, interviews, and guest editor introductions.
The call for nominations generated 37 titles. We augmented this initial set using the remaining two sources. Web analytics consisted of download data from IEEE's digital libraries: IEEE Xplore and the Computer Society Digital Library. The heterogeneous download data covered the period from 2004 to 2008. The data were filtered, sanitized, consolidated, and reduced through set thresholds. The end result of this step was a set of 56 articles with relatively high PDF downloads. Citation statistics were based on Thomson ISI and Scopus data. Daniel O'Leary analyzed the Thomson ISI data from 1984 to mid-2008. His analysis is also published as a separate piece in Software's current issue (see p. 12). This set had a strong bias for older articles, so we also requested assistance from editorial board member Diomidis Spinellis, who provided us with year-by-year citation statistics from an alternative source, Scopus, covering the period 2000–2008. We then merged the two sets of citation statistics to come up with 119 articles.
The next step involved condensing the three overlapping sets of 212 articles into a shortlist for a selection committee to review. We started with the list of 37 nominated articles. Then we identified criteria and thresholds to select articles for addition to the shortlist from the remaining two sets. We moved seven of the 56 articles in the high-downloads set and six of the remaining 119 articles in the highly cited set to the shortlist. The end result of this step was a shortlist of 50 articles.
To optimize the available review resources, we then placed each of the 50 articles in the shortlist into one of three reader-interest levels on the basis of the number of nominations the article received and the number of downloads and citations it generated. Five of the nominated articles with the strongest evidence of reader interest were assessed at level 1. Ten nominated articles were assessed at level 2. The remaining 35 articles with sufficient evidence of reader interest in terms of citations or downloads, but not necessarily both, were assessed at level 3. Three selection committee members evaluated each level 3 article. Two selection committee members evaluated each level 2 article. Level 1 articles received a single evaluation each. The committee's evaluations rated the articles according to historical importance, appeal to Software readers, information value, applicability, popularity, and relevance. We didn't ask the committee to reassess an article's publication worthiness, just its relative worthiness of being singled out as recommended reading. We then reviewed the committee's evaluations and deliberated in several iterations to reach consensus. In one instance, we consulted with an outside expert for confirmation. Finally, we settled on 35 articles that represented, in the committee's opinion, the best of the bunch. This reduced set is our final selection (see Table 1 ).
The selection committee was diverse. Including the three chairs, it comprised 22 members of the editorial and advisory boards, representing all our coverage areas as well as varied interests, geographical regions, professional roles, and generations. The final selection reflects this diversity. Our goal wasn't to eliminate subjectivity—doing so in such an endeavor is impossible. A different set of experts would undoubtedly come up with a different list. Thus it's entirely possible that we've missed articles that you think deserve to get on the final list. If we missed your favorite full-length articles, write to us and let us know about the hidden gems. Don't forget to explain why the articles should have been included.
So, even if we can't guarantee that all these articles will appeal to everyone, our 25th-Anniversary Top Picks list hopefully contains something for every taste and orientation. Whether or not you agree with these articles' ultimate messages, you'll find them packed with wisdom and timeless ideas. We especially invite you to check out the older articles that have stood the test of time: you might be surprised by the degree of relevance they still possess many years after publication.
We've tagged the articles so that you can quickly identify your particular interests by topic or by category. Although some articles could easily fall into more than one category, we have only indicated the most fitting category for each article. If you're interested in a topic overview, scan the topic column and look for titles with the tag "O." If you're looking for original analyses based on experiences, observations, or surveys, look for the tag "E." Articles expounding pioneering ideas that had a significant impact are tagged "P," recent cutting-edge work is tagged "C," and reflections on specific themes, areas, or the profession in general are tagged "R."
Hakan Erdogmus, editor in chiefFrances Paulisch, chair of the Advisory BoardJohn Grundy, associate editor in chief
We wish to extend our sincere thanks to the Selection Committee: Elisa Baniassad, David J. Blaine, Annie Combelles, John Favaro, Robert Glass, Ann Hickey, Gargi Keeni, Neil Maiden, Grigori Melnik, Maurizio Morisio, Linda Rising, Martin Robillard, Helen Sharp, Forrest Shull, Diomidis Spinellis, Wolf-gang Strigel, Laurence Tratt, and Uwe Zdun. Special thanks go to Daniel O'Leary and Diomidis Spinellis, who provided the citation data. We're also grateful to several members of Software advisory and editorial boards as well as past editors in chief for nominating their favorite articles.