Mobile Usability Testing on a Budget
From MobileDesign
Any usability expert will tell you that the best and easiest method of making your application easy to use is to test it, early and often. Putting the application in front of potential users and having them attempt to complete normal tasks is the best way to understand user behavior at every stage of development, including storyboards, information design, interim development stages, and beta release. However, the mobile environment introduces several challenges associated with device proliferation, application modality, data collection, and user mobility.
Contents |
[edit] What is Usability Testing?
A usability test is a technique to assess the ease of use (or a few other factors) of an application by asking a user to actually use the application and observing user behavior with the application. You might record the application behavior and user behavior on videotape. Usability testing differs from a focus group in two major aspects: it measures user behavior instead of opinion and the user actually interacts with the application instead of merely looking at it. Usability testing of a web site for a computer is well understood in the usability community, as well as outside it. The same can be said for testing computer-based applications or consumer products. Unfortunately, mobile applications are frequently more complex.
Usability testing isn't necessarily expensive. A formal study of some design question will cost you $15,000 or more per test, but answering the question "what major design flaws does this application have?" can be done without a laboratory, without hiring a usability expert, and without extensive participant recruiting.
You can quickly put your application, in whatever current state, in front of a user and ask the user to say what they would do at each stage of the task. Repeat for a total of 3-4 users and you will have a list of some of the worst problems in your application. Go fix those problems and repeat the process with a new set of users. You'll find another batch of major problems. Don't spend more than one day per cycle.
[edit] Handling Device Proliferation
Every computer has a screen (at least 800x600 pixels these days), a keyboard, and a mouse. Every browser renders a select list the same way. Mobile devices differ a lot more, and they always will.
In an ideal situation, every major type of device targeted by the application would be tested with users. This includes not only phones versus PDAs, but any device with noticeable rendering differences of the same code.
Handling the complexity of device proliferation requires four strategies:
- Identify which devices 80% of your users will use to access your application. If this list is very short - 3 or fewer devices - plan on testing with all the devices on the list.
- Test certain things, such as under which heading a particular page belongs, on paper without a device at all. Be cautious when testing labeling this way, since a reasonable label on one device may collide with the operating system labeling on another device.
- Identify device families between which the user experience is the same or similar. You can assume that all Palm OS devices are largely the same, and all Pocket PC devices are largely the same. You can also identify families of phones that have the same characteristics, such as the Samsung 3500 and 8500. When you decide which of a device family to test with, choose the most limited - slowest, smallest display, fewest features, most difficult input - unless you can afford to ignore users of that device.
- When doing iterative testing, use different devices for each test. Remember that you are looking for major problems. You're not trying to make a statistical statement about user behavior. You are instead saying, "if some users in our test found a problem, then some users will find this problem."
Applications will be perceived differently based on device characteristics.
Devices might be:
- as small as possible, optimized for voice communication.
- quite large, optimized for data display
- optimized for gaming
- optimized for multimedia
I can read a news story on my small, 8-line mobile phone. I read every word, slowly because I read faster than I scroll. 20 minutes after reading the stories, I don't remember anything about them. They didn't get stored in memory.
Contrariwise, I can remember bits about the news I heard on the radio this morning and news I read on my computer this afternoon.
My point: the same application will be perceived and understood differently on different devices.
[edit] Handling Application Modality
Mobile applications are quite likely to be multi-modal, and this likelihood will only get higher as 3G technology both increases bandwidth and allows simultaneous voice and data communications. This makes early testing more of a challenge, because the user can not simply point at something on the screen and describe what to do with it; there is a simultaneous aural experience.
Voice applications have as many subjective elements as graphical applications, and most issues are difficult if not impossible to test before having an actor record the script. However, it is reasonably easy to test the information design portions of an application before even hiring an actor or prototyping anything.
"Wizard of Oz" testing involves three people: the user, the moderator, and the person who plays the computer. For a voice-only application, the "computer" will respond, from a written script, to the user's spoken commands. For a multi-modal application, the "computer" will also "display" a storyboard screen in front of the user, based on the user actions.
The beauty of Wizard of Oz testing is that it is easy to quickly design and prototype, as the prototype is hand-drawn screens and a script. Users can say what they would do, and see the results of their actions. With a voice-delivered application, you can also get some feeling for what sort of pauses and intonations are best for the user to understand the application. The result of a series of tests can be a precise user interface (voice plus visual) specification with a much lower risk of rework later in the development cycle.
[edit] Handling User Mobility
Already we have a much more challenging situation than computer application designers, and we have another major difficulty: mobile users are likely to be very distracted during application use, and they are rarely distracted during usability tests. A mobile user may be dealing with a customer, distracted by a waiter, receiving an incoming phone call, or paying attention to a meeting.
For formal, statistically precise usability tests, don't try to introduce distractions into the test. Unless you run dozens of users through the test (incurring a large cost), you'll lose the statistical validity of the test without a discernible benefit.
For informal, problem-identifying tests however, the mobility of the device actually helps make the test more realistic. Instead of asking a potential participant to come into a quiet room and use an application, you can take the application to a participant wherever that person may be. Many people will welcome an interruption over lunch if it means they get $50 for their troubles. Don't forget to have them sign an informed consent statement.
If you ask a participant to use your application over lunch and the test gets interrupted by a visitor or waiter, that's fine. Take the opportunity to watch what happens when the user resumes the task, and see what difficulties arise. If you can design the application so users don't have those problems when interrupted, your entire application will be easier to use overall.
[edit] Handling Data Collection
Another mobile usability testing challenge arises from data collection: most usability tests record the user's body language and interaction with the product using two cameras. These videotapes are useful for the product team to understand how users are reacting to their product. However, mobile devices are intended to be held. They are extremely personal, and users may pick them up, gesture with them, or lean back with them. For these and many other reasons, computer-based simulations of mobile devices are usually inappropriate for usability tests, but training a camera on a moving mobile device is a difficult problem.
Most solutions to this problem involve fixing the device in place and then pointing a camera at it, but [Norm Wilcox Associates]has a usability lab in a box designed for mobile devices. They created a lightweight sled that attaches to the mobile device. The sled has two small cameras: one pointed directly at the device screen, and the other pointed at where the user's face would be if the user were looking at the screen. This allows the device to act very much as if there were no cameras at all.
[edit] Testing Early and Often
Too often, product development teams don't bring in usability testing - or even usability expertise - until the late stages of product development. There are concerns about it lengthening the development time. Developers of mobile products are frequently under more strict deadlines and are faced with a more complex usability testing environment.
However, mobile applications are used under more challenging circumstances than computer applications, with more limited controls. It is therefore necessary that mobile applications be more usable than computer applications. Frequent usability testing will help ensure that your application has no major flaws, and mobile design expertise will help ensure that you have to make as few changes as possible.
Remember: you can perform a usability test on your storyboard document. You don't have to have a fully functional prototype.
