Mobile Usability Testing is Not Rocket Science!

MOBILE USABILITY TESTING IS NOT ROCKET SCIENCE!

Software testers do not make software; they only make them better.
—Anonymous

For some reason, people not involved with the field of UX (usability) have the idea that usability testing in general is as difficult as rocket science—especially for mobile and the constraints it comes with.

I hear from people that the main reasons for not conducting usability tests are that they don’t have the know-how, “enough” time, and, most commonly, the budget. Of course, some approaches can be very difficult, take a lot of time, and be very expensive. But they don’t have to be. Within this article, I will discuss some of the approaches we use for mobile usability testing to keep all these factors as low as possible. I’ll also address how mobile usability testing differs from desktop usability testing.

Let’s start easy

Being a usability or software tester doesn’t have to be a requirement to conduct usability tests on mobile devices. If you are a designer, project manager, marketer, or even an enthusiast, you can start doing a simple mobile usability test on your own. Just launch the app (or website) and start using it. However, the biggest mistake we make when doing this is that we do not take into consideration that mobile devices are being used in different settings. It surprises me that many people have the tendency to conduct mobile usability tests at locations that do not represent the actual environment where mobile devices are used. Instead of using (read, testing) your mobile app at the office or in a usability testing lab, start using your app on the go. Use it when you are on the train, when you are cooking, when you are waiting at the dentist, or when you are having lunch. This will be a much more realistic setting than testing in your Zen-inspired office with reliable Internet connectivity.

You can expand this approach by including your relatives, friends, and coworkers to this simple mobile usability test. Just like you did, let them use the app, and observe them while they interact with it. You will learn a lot just by looking at the screen and listening to what they have to say about the app afterward.

This simple, low-cost, and effortless approach for a mobile usability test might not provide you detailed metrics or a good basis for a substantiated conclusion that can affect big design decisions, but it does give you insight about how your app responds in different settings and how people use and think about your app.

Kick it up a notch

Testing on your own or with the closest people around you might not be the mobile usability testing approach you are looking for. You might want to include people who are actually representing your target audience, and at the same time keep the “quick and dirty” usability testing approach we discussed in the previous paragraphs. If this is the case, hallway testing is most likely the approach you would want to use.

For one of our clients, we designed a mobile app from the ground up. During the early stages of the design phase, we had to make a decision regarding the navigation of the app that would influence the entire design and structure. Before making such a big decision, we decided to test the navigation we had come up with first on usability. In order to do this, we created a low-fidelity prototype in just a few hours using an online prototyping tool, and we installed this on our mobile devices.

With a team of three, we went to one of the busiest shopping streets of the city—not to look for new clothes, but because our target audience was there: females between the ages of eighteen and twenty-five. Our team of three consisted of a greeter, a facilitator, and an observer. The role of the greeter was to spot people who would fit our target audience and persuade them to participate in our mobile usability test. Our greeter explained to the participants that the usability test would take only ten to fifteen minutes and that in return, the participants would be rewarded with a multistore gift card worth ten euros.

Once a passerby had decided to participate in our mobile usability test, the facilitator stepped in. His role was to explain our goal and expectations from the usability test, and to introduce the mobile app without giving away too much detail because of our nondisclosure agreement with the client. Next, the facilitator handed over the mobile device and gave the participant the tasks he or she had to fulfill using the mobile app. While the participant was trying to complete these tasks, our observer took notes about the participant’s progress. Since the goal of the usability test was to find out if our users understood the navigation of the app, we only kept track of whether the participant passed or failed a given task. To learn more about the participant’s motives and behaviors, we conducted a short postsession interview.

We repeated this process with ten other participants. As we noticed a pattern when comparing the results of all the usability tests, we decided that we no longer needed to include more participants. With a limited amount of time and budget, this approach of mobile usability testing, or hallway testing, gave us insight about how users responded to the navigation we came up with for our mobile app. Even though this approach does not provide detailed metrics, it is a “quick and dirty” way to help you make substantiated decisions in the early stages of the design phase.

Get more detailed

In many cases I find it more useful to have a more comprehensive approach when it comes to mobile usability testing. Usually, the mobile usability tests we conduct for our clients are as detailed as classic desktop usability tests. Even though this approach requires more know-how, has a larger time frame, and has a higher price tag, it does give us a more detailed and valuable view of how efficient, effective, and satisfying the product is from the user’s point of view.

For these kinds of usability tests, we spend more time when it comes to preparation compared to the other approaches. During the test sessions, we capture every single detail using recording equipment and software. We capture screen activity, the user’s facial expressions and voice, the input of the user (e.g., mouse clicks) while performing a task, etc. In order to run the test sessions without any discommodity, we test the hardware and software beforehand at the test location. We even perform connectivity tests if the usability test requires the use of an Internet connection. Running several pilot sessions is not an unnecessary luxury, so do not forget to include these into your preparation as well.

Another aspect that makes these kinds of usability tests more detailed are the metrics we are focusing on. We spend more time and put more thought into what kind of data we need to collect from the usability tests. Depending on the goal of the usability test, you can choose to collect quantitative data, qualitative data, or a combination of both. Without going into too much detail, quantitative data is the type of data that can be measured, like length, height, and volume. The other, qualitative data, cannot be measured but can be observed, like smells, tastes, and appearance. In the context of a usability test, quantitative data can be the measurement of how much time a user spends to complete a certain task, and qualitative data can be the observation of how a user reacts to a certain design.

Depending on the scope of the usability test, the scenarios the users need to walk though can also be quite long and more specific than the other usability testing approaches we have discussed in the early paragraphs of this chapter. For more complex scenarios, you may even need to provide supplementary information, like login credentials, dummy credit card details, etc. A good way to check whether the scenarios are being well interpreted by the participants is to run pilot sessions as mentioned previously.

Last but not least, finding users for these kinds of usability tests is much more difficult and time-consuming. It is not easy to persuade people to participate in a usability study that can take one to two hours of their time, especially when the target audience (or persona profile) encompasses those with a busy lifestyles. That is why we recommend that our clients attract participants with incentives in the form of cash, gift cards, coupons, etc. When the amount or type of incentive is right, which can differ per person, it usually is easier to attract participants. But keep in mind that this will increase the cost of your usability test.

Mobile usability testing constraints

Even though we use the same approach for mobile and desktop usability testing, we unfortunately have to face two constraints when conducting usability tests for mobile. The first and foremost constraint concerns recording equipment and software. As mentioned before, during classic usability tests we use recording equipment to, for example, capture screen activity. For desktop usability testing we can do this by mounting an external video camera facing the screen, or by installing specific screen-recording software on the desktop. These are two techniques we can’t use during mobile usability testing. Using a mounted video camera restricts the participant from moving the mobile device around freely while testing the mobile app. And installing and running recording software on a small device like a smartphone can slow down the performance of the mobile device, which may affect the progress of the usability test and lead to unrealistic test results.

To bypass this constraint, a mobile usability testing sled can be used. This piece of hardware enables us to capture the screen activity and the interaction between the user and the device by using a mounted webcam, all while providing the user with mobility. Unfortunately, since no additional recording software is being used, some features that might come in handy, like recording the input (e.g., screen taps), are not possible.

The second constraint we have to deal with when conducting usability tests for mobile is the testing environment. As explained earlier, unlike desktops, we use our mobile devices on the go. Therefore, it is advisable to conduct mobile usability tests in the environment where the app will be used. Depending on the context and target audience, you can conduct a mobile usability test in places such as a train, the university, a shopping mall, etc. However, since we are on the go, keep in mind that connectivity, lighting, and distraction play an important role during the usability test, and that a sudden change to any of these factors can influence the user’s behavior. So when determining a test location, conduct pilot sessions first to prepare yourself for what you can expect. In some cases, conducting an outdoor mobile usability test can be too risky. If that is the case, take a step back and try to find alternatives. Even though conducting mobile usability tests at the office or in a usability lab is not ideal, it is always better than not testing at all.