• Exactly who are these "peer organizations"? Are they peer organizations simply because they develop software? Or do they have to be in the same general industry? What about size? Do Motorola's peers include a 10-person start-up? Do the peers of an organization that produces Web content include NASA as well as Podunk Savings and Loan?
• How many peer organizations have to recognize the practice as best? Must it be a certain number or percentage? If it's a percentage, how do we know how many peer organizations exist?
• Who's responsible for collecting data about which peer organizations recognize the practice and which don't? Consultants? Researchers? Academics? Magazine editors? I was at a recent conference in which the keynote speaker (an academic) proposed that a special board of notable researchers be formed to determine and publish a list of best practices. In principle, this sounds like a pretty good idea. But few researchers are purely observers—most are (or at least try to be) change agents. If change agents sit on the board, which is likely, they'll no doubt innocently record observations about organizations they've influenced or tried to influence. This seems to guarantee a skewed observation.
• How does a peer organization recognize a best practice? Does simply performing an activity make it a best practice? If other peer organizations don't perform the activity, does that imply that not performing it should be a best practice? For example, some companies use inspections and some don't. If performing an activity is the only requirement, we could end up with two best practices: using inspections and not using inspections.
• What about organizations in which some units use a particular practice and others don't? Must an organization always use a particular practice for it to be recognized? My experience leads me to believe that large gaps often occur between different business units' practices (and often even in projects within a given business unit). Certainly an organization's use of almost any software development practice will differ depending on whom you ask.
• And what about conflicting best practices? CMM versus agile? Middleware versus native OS support? J2ME versus WAP? Objects versus datastores? Surely not everyone agrees on every good idea ever introduced in the software development community.