Agile Careers

 

Jim ("Cope") Coplien is an old C++ shark who now integrates the technological and human sides of the software business as an author, coach, trainer, and executive consultant. He is one of the founders of the software pattern discipline, and his organizational patterns work is one of the foundations of both Scrum and XP. He currently works for Gertrud & Cope, is based in Denmark, and is a partner in the Scrum Foundation. He has authored or co-authored many books, including the recently released Wiley title, Lean Architecture for Agile Software Development. When he grows up, he wants to be an anthropologist. 

Log in to register a comment.

Subscribe here.

Blogs Blogs
« Back

Swarming

Monte Python’s Brian reminds us that we (most of us, anyhow) “are all individuals.” Rugged individualism is the hallmark of many cultures — most notably of Americans who celebrate a pioneering spirit borne of its expansionist 18th and 19th centuries. Perhaps some day I’ll  dedicate some thoughts to that topic. But today I’d like to draw your thoughts to work-in-the-many.

Anthropologists sometimes debate whether homo sapiens are social animals like apes or loners like bears. I tend to lean to the social animal side of the argument. At this point, as you project where today’s installment is leading, you’re likely to roll your eyes in anticipation of yet another one of those homilies about teams. Teams are all the rage these days, and appropriately so.

However, today I want to talk about swarms. Yes: humans swarm, though the collected entity is rarely as physically identifiable as the apiarian analogue. Such structures have long existed in human culture. Religious denominations often fit this model, as do political parties. Swarms long pre-date the Internet, though the Internet has brought life to swarms with instant communication and connectedness that were unthinkable five years ago. I dare say that will remain a timeless claim no matter how far in the future you are reading this article. That’s why swarms are so important: our swarm-connectedness is bound to grow over the decades.

Back in 2000, Dick Gabriel’s OOPSLA keynote talked of the importance of swarm development in software. Open source had already been in full swing for many years. Forget pair programming: bug density plummets under the gaze of thousands of pairs of eyes. I can launch a question into the swarm about an Apple API and get a useful answer in minutes or hours. That was unthinkable even from dedicated help desks just a few years back.

Join a swarm — not a club — of thousands or millions of people. Clubs are about stature; swarm members are anonymous. Clubs are social and fun; while there’s a bit of that in a swarm, it’s not really part of the swarm dynamic. Clubs just gather; good swarms do work. Yet some swarms generate identities that emerge at a larger scale. The open source movement is a shining example — millions of anonymous workers building some of the greatest software on earth. The pattern community is another, and while it is like a bed of tulips where much the same flowers recur on an annual basis, it carries a vision of a shared worldwide identity. It is perhaps a club in transition to becoming a swarm if history so favors it.

Contemporary engineers are team creatures. Today’s engineering career path follows much of history as individuals hop the stepping stones from one team identity to another.  For all their good, however, teams are remarkably insular. They interact tens times as much among themselves as with members of other teams. More social progress comes from swarms than teams.

It’s important to note that “swarm” is not a synonym for “good.” While mutual trust characterizes some swarms, it is hardly germane to all of them. Invoking Gabriel again, we note “dangerous waterholes” in the savannah where both the lion and gazelle “swarm” to drink. If the lion kills the gazelle there, then the gazelles will stop coming there at all and will eventually be wiped out for want of water. Tune swarm parameters just right and you get mass movement psychology — a cancerous, collective insanity.

Are you a part of a swarm? Know what kind of swarm you belong to, or swarm into a good one.

Comments
Trackback URL:

http://www.geraldmweinberg.com/Site/Programming_Psychology.html

Weinberg wrote about "egoless programming" back before the beginning. This seems to be similar to your point here about "swarms".

But while I very much like Weinberg's later writings, I'm afraid I've come to believe he was 100% wrong about "egoless" being a good thing. Neither am I a fan of open source. Nor do I want to be a termite. Nor do I want to find bugs being introduced in my software and have no idea how they got there. The formal quality processes I know all demand responsibility, traceability, configuration control.

Oh, of course I understand the hope here, that we should accept help and not defend our work because it is ours, even if it is wrong, even if it turns out we are the worst person in the group to do X. The problem is, that's all negatives. What if we are the *best* person in the group to do X, and we spend all day fixing others' work because they keep swarming all over things?

For open source, it is just barely possible that 1,000 swarmers can do better work than 3 semi-dedicated professionals. But I doubt it, because the 1,000 will never be headed in exactly the same direction, they are not even communicating to try, they do not even share an idea that this might be a good thing.

It's fun to see your code, your efforts, be a small part of a big project. Open source may give you your only chance to do that. But professional jobs are not about swarms. And we haven't even gotten into the ideas of parasites, hackers, criminals, and nutjobs infiltrating your swarm.

Posted on 2/20/12 4:51 AM.

I think you have not have enough experience in open source to judge it this way. Yes, open source is not a silver bullet. You cannot develop software for free by just making it open source (a mistake that many companies do). But it's also wrong to say that open source doesn't produce software as well as do comercial products.

To avoid using the overused Linux kernel example, i will use the field of video coding. Of the most used video players out there VLC, Mplayer clasic, Xine, Quicktime and Windows Media Player, the top three are open source projects, beating the hell out of the two comercial ones in all regards (features, supported formats, performance, bugs). Actually, one could say that the comercial ones are only that widespread because they come installed by default on Windows and MacOS respectively.

Or if you dig deeper and look at video coding libraries, there is one that is the most used by far: libav/ffmpeg. It is the driving force behind most software video players, all open source players use it, and quite a few of the comercial ones too. It is even the enabling key technology behind youtube. It wouldn't be economically possible for youtube to buy or write decoders for all those video formats that are and have been used, weren't it for libav/ffmpeg. Or take the x264 encoder library, which competes with comercial h.264 encoders, both in features and speed (and some say it's actually better).

And all these projects work. Their direction is not always clear cut, as it is in a comercial enviorment, but they have a direction (sometimes even multiple) and the "product" is not only quite good, it beats even comercial developments. The people involved in those projects are in the a few people, to a few dozens range. Over the years, some projects have had contributions by hundreds if not thousands of people.

To sum it up: Open source works. Even if thousands of people are involved, there is a direction. But it is not a silver bullet that will magically solve your development problems.

Posted on 2/20/12 5:41 AM in reply to Joshua Stern.

I have worked with high-dependability systems in different sectors for more than 30 years, so the notion of a swarm of (stinging!) programmers would give me sleepless nights.

Over many years I have come to rely on my cluster theory, that is, competence clusters. Therefore, I would have to agree with Joshua that "professional jobs are not about swarms".

Posted on 2/20/12 6:42 AM in reply to Joshua Stern.

07078

I agree with you insights but I think the methaphor of a "farmers market" more aptly fits the model of a successful large (more than 50 people) software project. There are many well strucutred aspects and some critical unstrucutred aspects of the project. The structure must becomes more disciplined as the project (market approaches closing) approaches delivery.

I wrote about this abstraction in my book published by Wiley 2005 "Trustworthy Systems through Quantitative Software Engineering." Best regards Jim.

Larry Bernstein

Posted on 2/20/12 7:47 AM.

First let me admit that I have not participated in an open source project, so my comments are directed specifically to the concept of swarms, not the practice.

Others have remarked that human beings are poor at describing what they want, but remarkably capabable at telling you what they don't like. I am speculating that a swarm is a great concept for finding faults or bugs, but I wonder if it is efficient at developing the optimal solution.

From a Dawinian perspective, evolution somehow manages to arrive at amazingly complex and inventive solutions, but it takes a long time. I suspect this is the same with swarms - they find faults quickly, but they design a new solution slowly.

Finally, I read in a blog by Joel Spolsky (Joel on Software) the following thought:

"Five Antonio Salieris won't produce Mozart's Requiem. Ever. Not if they work for 100 years."

I am trying to imagine a tune written by a swarm....

Posted on 2/20/12 8:12 AM.

Your blog was written by one person, not many. The comments you've received from the teeming millions didn't cause you to comment or to change your blog despite the overwhelming bearish response. In a way, yes, because the comments are here, but your work product is unchanged, because it is yours, and yours alone.

Individuals invent. There were swarms of cell phones before the iPhone, but Apple, the classic cathedral company, nailed it. Facebook was created by one, very impressive hacker. Zynga is an insular, tightly-controlled company, and created the most popular game in the world (perhaps), or was it Angry Birds created by yet another tiny company without a swarm of contributors. Warren Buffet works in a small group. The arts are overwhelmingly individualistic endeavors.

Kevin Kelly has been touting swarms for two decades. His predictions have failed to materialize. You can read more here:

http://www.viarob.com/my/page/BookReview_Out_of_Control

And, yes, he was the sole author of this book.

Programs are the most complex machines in the world. Anthills and beehives are some of the world's simplest. There's a reason why ants and bees don't run the world, and humans do.

Posted on 2/20/12 3:46 PM.

I'm fascinated to hear that open source work is anonymous. Richard Stallman, Linus Torvalds, Doug Cutting had something to do with the work. In Open Source people have reputationon as more than a swarm.
Corporations on the other hand like to have swarms so that all the rights are theirs.

Are you thinking of Anonymous? That's the only swarm I know about where the leaders aren't known by name.

The excepti

Posted on 2/20/12 4:50 PM.

In response to Joshua Stern's post:

Like most people, I would guess that you fail to understand what Jerry meant by "egoless" team members and teams. Jerry and I were once chatting about how one commenter discounted egoless teams as a "quaint Victorian notion" based on an understanding that equated the term ego with power. You might check to see if you have the same misunderstanding. I think you might.

"Egoless" is more about sharing than it is about exercising one's power or not. If you are the best person to fix X and you keep fixing X, instead of training colleagues to do it, you aren't really a team player. It is a matter of getting everyone to the same level of power that you have — not to diminish one's own power.

One thing that makes swarms work is that its members are not headed all in the same direction. Groupthink is a powerful disabler. A diversity of perspectives is powerful. Someone non-expert in a particular class of algorithms may see a trap in the code to which the experts are blind. It isn't that swarms work because they add many different kinds of expertise: it is more because they bring so many commoners with respect to the topic of discussion. Fish can't see water: it's the outsider who is often the first to notice that the emperor is going mighty chilly this winter.

We'll leave you to your "professional job" while we continue to instead think about careers and about the bigger picture. As for your parasites, hackers, and criminals, we can add that to those who habitually lie to their managers as a way of getting their work done. I know a few of those. I think you do, too.

Posted on 2/26/12 9:02 AM in reply to Joshua Stern.

In response to Martin Allen's post:

Everyone is entitled to his or her opinion. Tell you what: tell me what compiler, what operating system, what window system, what browser, and what hardware you use. We'll then come back and see how much of their technology we can trace to mass community efforts.

I, too, have been working with high availability systems for a while — since about 1978, so we're of the same generation. Five nines of availability. Much of it relies on open source software. I don't know how our experiences can be so different.

It's easy to say "I believe" and blah blah blah. Let's get down to brass tacks.

Posted on 2/26/12 9:04 AM in reply to Martin Allen.

This is in response to Larry Bernstein's post –

Hi, Larry! I also like the farmer's market metaphor. Speaking of markets, you might read "The Cathedral and the Bazaar." You can find it at http://www.catb.org/~esr/writings/homesteading/cathedral-bazaar/. I spend much time in Middle Eastern bazaars, shopping mainly for spices and Oriental carpets, so I know the territory. And I sometimes live in the open source world which works in the same way. "The Cathedral and the Bazaar" — just a farmers' and merchants' market — is in fact what the open source community takes as its model of work. So I think we're on the same page...

Posted on 2/26/12 9:08 AM in reply to Lawrence Bernstein.

In response to Robert Nagler's post:

Let me quote from the anthropologist A. L. Kroeber, addressing himself to the matter of genius in society:

"Personalities of the very highest ability — whether in ruling, thinking, imagining, innovating, warring, or religious influencing — have long been known to occur in concentrations of country and period. The philosophy [of ancient Greece], for instance, is exemplified by Plato and Aristotle, by their predecessors like Democritus and Socrates, by their successors... Or we can equally well say that it is these men who produced the philosophy...

"Of these two related aspects, the philosophy and the philosophers, the activity and product of the personalities, there is no doubt which is the easier to grasp: the concrete persons... Hence ordinarily we tend to think first of Socrates the man and secondarily of what new ideas he produced...

"However, as long as the matter is viewed as one of persons, it remains rather meaningless... It came to be accepted that there are golden ages thickly studded with great men achieving unforgettable products; silver ages with somewhat lesser men... and ordinary times. But no reason was manifest, through many unstatistically minded centuries, for such fluctuations, which glaringly violate the principle of random distribution or accidental occurrence, and therefore suggest that there must be specific factors at work to produce the unevennesses.

"One possible misconception will be guarded against by its mention. The naive view of course is that great men do great things..."

Kroeber notes that most great discoveries were products of society and simultaneously came out through several inventors; we happen to remember the ones with the greatest publicists. Sunspots were discovered in 1611 by Galileo, Harriott, Scheiner, and Fabricius — in different countries, out of communication with each other. He lists dozens and dozens of inventions that followed this same pattern: the telescope (Jansen, Lippershey, Metius, in 1608), logarithms (Napier in 1614 and Bürgi in 1620), the telegraph (Henry, Morse, Steinheil, Wheatstone and Cooke, about 1837), the periodic table (Mendeleev and Meyer, in 1869), star parallax (Bessel, Henderson, and Struve in 1838). Why arbitrarily pick one individual on the pretense of his or her intellectual prowess? To do so is simply to be ignorant of history and human nature. But it is often the nature of professional engineering mentality to be ignorant of human nature, and to be blind to history. I'd say we have a case in point at hand.

And, yes: I regularly change my 'blogs in response to posted responses. I am about to make a major edit to an entry that Jeff Sutherland invited me to post on his site, pointedly countering equally ignorant feedback posted there.

So, you have grasped that which is easiest to grasp. Perhaps because you are an engineer, the mores both of culture and of science are beyond your reach. An engineer enamored of his track record in a job is mortified by the prospects of having been only the messenger of society's bringing an idea to fruition, and so he defends his position with rhetoric such as "sole author" or "inventor."

As for my work here: I stand on the shoulders of giants as the product of those who have generously shared with me over well more than half a century. My 'blog was written by many. I do not know how to credit them. We can identify by name only those who try to tear them down; those who built it up will remain anonymous to history. Their anonymity doesn't imply that they don't exist. I know I can say that as the author of this work, your comments to the contrary notwithstanding.

I leave you with your perspective that one of the greatest writers about human endeavor calls out as "meaningless" and "naive." And I leave you with your engineering perspective of individual invention, authorship, and patents. But you will need a better publicist than your current writing indicates that you currently can afford. Your job may depend on it. My career does not — but it does celebrate the hundreds of people to whom I am connected, and who have empowered me to be part of some great things.

Posted on 2/26/12 9:47 AM in reply to Robert Nagler.

This is in reply to Nick Nussbaum's post:

Yes, most open source workers are largely anonymous. Linux Torvalds these days does a very small fraction of the innovation and technical work: like a great project manager, he is more important for being than for doing. There are literally tens or hundreds of thousands, or perhaps tens or hundreds of millions (http://ask.metafilter.com/21769/How-many-people-have-worked-on-Linux). Tell you what: I'll buy you a bottle of Lynch-Bages if you can name one percent of them. I hold the rest to be anonymous.

Posted on 2/26/12 9:41 AM in reply to Nick Nussbaum.