Ronin Blog

The best way to improve an application’s usability is to run extensive studies - they allow you to get an idea of what it’s like to walk a mile in your users’ shoes. Sometimes, just the practice of sitting down with another human being and verbally walking through your application’s features brings out certain perspectives you may not have considered before. This is especially true if you wear many hats during the development cycle.

At Ronin, we run a usability study every few weeks, focusing on new features. We’re usually very considerate of our users throughout our application design process, but you’d be surprised how much slips through when you’ve looked at the source code of your application longer than you’ve looked at the actual interface. Running a usability study is an exercise that usually brings in a large list of low hanging fruit and a moderate list of long term application improvements.

That said, from our experience running these studies, there are a few things we realized you can never fully understand from just the studies themselves.

User patience

When a study participant sits down next to you to walk through your application, they’re in no hurry. They’ve committed the next 30 minutes to 1 hour of their lives to the purpose of completing the study. Compared to the average 9 second attention span you’re likely to get from a real world user, this is too distant from reality to simulate.

Unfortunately, there isn’t much you can do to recreate a frantic in-the-wild scenario. This negatively affects the ability to learn about edge-cases and error handling the most. A usability study user is likely to run into some errors that they are patient enough to figure out during a study, but would otherwise cause them to throw their hands up in disgust. To counteract this, its best to read between the lines when it comes to negative reactions. Multiply observed frustration during studies by many magnitudes to really understand how a user may react.

Also, try to develop an understanding of how user patience can be a big factor. Spend time testing your home, promo, and marketing pages with an emphasis on asking for a stream of consciousness verbal description of what they see with a time limit. After all, if they stick around long enough to try your application, you’ve won half the battle when it comes to user patience.

User attentiveness

Another problem is that study participants are usually more alert and attentive than their real-world counterparts. They’re willing to take their time and figure out a problem. They’re willing to devote their undivided attention to the task at hand. Compare this to the more likely scenario of an actual user trying out your application at 2 a.m. with the TV running in the background and after surfing to your promo page by coincidence. It’s night and day.

To counteract this, make sure you get immediate verbal feedback after each action they take, to get a sense for what they’re thinking before the interface has had time to soak in their consciousness.

User skepticism

Most deceiving of all is the mindset of a usability user. Depending on what your relationship is with the user (stranger, guest, friend, acquaintance, co-worker from another department, spouse, child) you need to realize they’re walking into the study with some intentions. Sometimes they’re participating because you’ve promised to pay them; other times they’re participating because they want to help you out. Whatever the case, nothing simulates user skepticism.

When a random user hits your site or application (say from Google), they’re coming with a guarded mindset. They’re cautious more than curious and they’ll immediately look for warning signs. A poorly designed interface with no obvious indications that there is active support or lack of no-obligation demos or screenshots are an immediate turn off. If they eventually try your application or dig deeper into your site, they’ll probably be less willing to try anything but the most obvious of features.

When a user hits your site or application from a referral or other more qualified method, they’re coming with a curious mindset. They want to dive right in and see what all the fuss is about. They’re also likely to be the ones who have heard of alternatives or are currently using an alternative and now they’re using your application to do some feature comparisons.

When a user participates in your user study, you get none of this. They’re neither overly guarded nor overly cautious. It is more likely that they feel you are working together in unison towards a common experimental cause. You really don’t get the stinging skepticism that a real world user carries.

It helps to keep these things in mind when conducting usability studies. Sometimes it’s too easy to pretend that you’ve aggregated all the feedback you can by running a well-designed series of studies. Even armed with that data, keep in mind that the best feedback is silent feedback. Make sure to analyze your logs. Don’t dismiss your analytics tools. Look over real user data from real users in the wild. Couple that data with usability studies and only then do you have a good macro- and micro- level understanding of your own application.