Do It Yourself: Measuring the User Experience

Jitendra Mudhol

Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics, by Tom Tullis and Bill Albert, ISBN: 978-0-12-373558-4, Morgan Kaufman, 2008, 316 pp.

When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be. —William Thomson (Lord Kelvin)

Did you know that “usability/user experience specialist” made it to last year’s list of top careers in a US News and World Report survey? This news should give this important field some much-needed exposure and momentum.

According to the Oracle of User-Generated Expertise (aka Wikipedia), “user experience” is a term for describing all aspects of a user's interaction with a product, service, environment, or facility. In a global village, addressing the wildly diverse needs of customers worldwide isn’t just desirable—it’s necessary for survival.

I first approached this book with some skepticism. The words “Measuring the User Experience” conjured up images of fuzzy, analog entities forced to fit into some finite scale. As the User Experience Network (UXnet), a nonprofit interest group, defines it, “UX is the quality of experience a person has when interacting with a specific design.” So, how do you measure this quality?

From the outset, the authors make it clear that they aren’t writing a theoretical treatise, but a practical how-to guide. They want to advise readers about “what usability metrics to collect in what situations, how to collect them, how to make sense of the data using various analysis techniques, and how to present results in the clearest and most compelling way." As I read through the book, my skepticism didn’t just vanish. It changed to admiration for the authors’ clear writing and explanations of what UX metrics are and how to collect, analyze, and present them. My experience: a good feeling sweeping over me as I became increasingly interested in the book at the turn of each page. I highly recommend Measuring the User Experience.

This book is expertly edited—detailed where needed and cogent throughout. Its 11 chapters reflect a straightforward approach to the topic. It’s easy to read and product/technology agnostic, so the concepts apply across the different UX dimensions. Many illustrated steps, charts, and examples enhance the book’s usability. Each chapter ends with a nice summary. Chapter 10 presents six case studies written by other experts in this field. A companion website provides lots of useful information, including ready-to-use tables, links, and references.

Measuring the User Experience can stand by itself. I think it should be a must-read for anybody who’s trying to deliver satisfaction through a product or service: designers, product or service managers, engineers, quality-assurance folks and even entrepreneurs. I’m confident that it will become a prescribed text in courses that teach about measuring user experience. It provides sound practical suggestions for those working the nuts and bolts, and it gives managers a good sense of the topic’s essence. Because it’s a little general, it might have some limitations for more rigorous practitioners.

Here’s an interesting nugget: From my statistics classes, I’ve carried the rule of thumb that if the sample size is less than or equal to 30, the population must be normally distributed, and you must know the population standard deviation to use a well-known statistical formula that you’ll find in any book on statistics. Which brings us to a burning question: Are five participants enough to reliably identify usability issues? This is the “magic number 5” issue that has divided the researchers into two camps, one vociferously supporting that almost 80 percent of the usability issues will show up with the first 5 participants.

The authors present both sides of the argument, and then provide their take on it: “[F]ive participants per significantly different class of user is usually enough to uncover the most important usability issues.” This will work generally, they say, with two caveats: the scope of the evaluation must be fairly limited, and the user audience must be well defined and represented.

To evaluate this book I decided to apply this very example that the book sought to illustrate. I asked five participants to go through the book and tried to measure their experience of it. I used the USE (Usefulness, Satisfaction, and Ease)questionnaire from the chapter, “Self-Reported Metrics.” Now, I don’t know if books have been evaluated in this way, but the experiment proved interesting, and Figure 1 plots a summary of my findings in a radar chart. The book received pretty positive resonance with all five participants.

Figure 1. Feedback from five participants in a review of Measuring the User Experience. The radar chart shows the feedback in four dimensions: usefulness, ease of use, ease of learning, and satisfaction.

We are moving toward a world where the consumer is an active participant and contributor to the product-development process. This movement will make measuring user experience even more complex than it is now. This book will ease the initial part of this journey, but it might require another edition to scale up to more complex cross-cultural user experiences.

Jitendra Mudhol is vice president of engineering at Broman Wireless, a pioneer developer of multiplatform VoIP (Voice over Internet Protocols) solutions and products. Contact him at jsmudhol@bromanwireless.com; startupnewz.com/blog.