As engineers we sometimes think of ourselves as objective applied scientists. Yet IEEE defines engineering as “promoting the development and application of electrotechnology and allied sciences for the benefit of humanity, the advancement of the profession, and the well-being of our members.” Technology is the means; the end is society, culture, and people. Engineering is a culture, and IEEE is a society that represents electrical engineering’s professional face to the world: a society embedded in that culture.
Every culture has its sacred stories: literature that underlies social progress. Our culture has entrusted the keeping of its written record to the IEEE and ACM. These are that society’s sacred, eternal foundations, capturing hard-won insights and enabling the spread of knowledge and innovation. So priceless are these that the cost of admission to this library is high: approval by reviewers, distinguished series editors, and by program committees and their chairs.
Nature News announced on 24 February that IEEE has removed over 100 published papers after discovering “that every single one is nothing more than fancy-sounding gibberish.” (http://www.nature.com/news/publishers-withdraw-more-than-120-gibberish-papers-1.14763). IEEE was aware of the general problem back in 2012 and removed some papers then; two weeks ago, they removed more based on input from Cyril Labbé at Fourier University in Grenoble.
It’s bad enough that these supposed engineering truths were removed: it’s tantamount to revoking the Ten Commandments. It’s easy to blame those who scammed the system, but it’s unlikely they were seeking publication accolades. They have in fact provided a service in showing a weakness in the system.
The most worrisome issue isn’t that bogus papers made it into the IEEE canon, but rather that the same process that admits meaningless papers can just as well accept ungrounded or misleading papers by everyday authors. And the IEEE isn’t alone. The 2011 ACM SPLASH conference published a workshop paper about work with which I was familiar, with a claim of formally proven results. I knew that no such proof was possible and published a workshop paper at the same conference in 2012 documenting a concrete counter-example to their “proof”.
Academic publication processes are unfortunately plagued by sloppiness, over-eagerness, opportunism, politics, and returns of mutual favors. My colleague Doug McIlroy (the inventor of UNIX® pipes) advised me in 1991:
It was pointed out in my NRC committee on whither computer science that conference proceedings, even the prestige ones like STOC, FOCS, and SIGGRAPH, deal in incomplete papers quickly refereed. Many illustrious computer scientists have given summaries of results in proceedings papers and then never fleshed them out; more than once it has happened that the result is wrong, at least in detail. Thus a surprising theorem announced in proceedings should be regarded with skepticism. If you regard your work as of more than transient value, and hope that others will too, you will go the extra mile for archival publication. Conference proceedings keep the pot boiling on the stove; the archival literature provides the floor for the stove to stand on.
The Lean mantra is: Have the right process and you’ll build the right product. IEEE reviewed this problem once in 2012 and “refined our processes to prevent papers not meeting our standards from being published in the future.” Process improvement remains a never-ending quest.
Another Lean mantra is: Go down to the Gemba. Even if only for reasons of changed context, you owe it to yourself to validate crucial technical foundations before moving forward on them. That it appears in writing doesn’t make it true.