Ronin Blog

We run plenty of usability studies here at Ronin. We run them so commonly, that we think of them as taste tests - it’s only any good if it fits the palettes of the everyday user. A certain magic happens when you combine the rich feedback mechanism that is usability studies with the instant gratification medium of the web applications.

Everytime we conduct a study, we walk away with plenty of improvement ideas. Then, we sort them in priority (including bonus points for biggest-bang-for-buck features) and we add them into the product, usually within one or two days. Simple as that. While that may seem mundane, it’s really only possible with the appliction distribution model of web-based applications. Instead of building and compiling the next version of software as an incremental release that our users have to download, we can ship it to them without them even knowing.

Case in point, after the last round of usability studies, we’ve made the following improvements, all of which were released Sunday evening, waiting for use on Monday morning:

• The ‘amount’ field during adding of payments is automatically filled out with the remaining balance of the invoice. This is so often the case, and it’s a big win for efficiency.
• Removed requirement that a ‘note’ be filled out during payment creation.
• Added two links to edit “from address” (and other additional informational fields, like tax ID) and “to address” right in the invoice interface. The most common case of client creation is during the invoice creation flow and additional address information is usually a necessary addition afterwards.
• Change the invoice summary section to read “Total Due” instead of “Remaining Balance” when there haven’t been any payments, partial or whole. Removed unnecessary tax information from the invoice summary when the invoice is taxless.
• Improved the invoice comments UI by moving up next to payments as a column. We noticed a lot of users mentally filtered out the comments section because it was so low and hidden on the page.

This is just one set of improvements we’ve made, but changes like this happen almost on a daily basis. Sometimes they’re big, sometimes they’re small, but they’re hopefully always a step in the right direction.

The best way to improve an application’s usability is to run extensive studies - they allow you to get an idea of what it’s like to walk a mile in your users’ shoes. Sometimes, just the practice of sitting down with another human being and verbally walking through your application’s features brings out certain perspectives you may not have considered before. This is especially true if you wear many hats during the development cycle.

At Ronin, we run a usability study every few weeks, focusing on new features. We’re usually very considerate of our users throughout our application design process, but you’d be surprised how much slips through when you’ve looked at the source code of your application longer than you’ve looked at the actual interface. Running a usability study is an exercise that usually brings in a large list of low hanging fruit and a moderate list of long term application improvements.

That said, from our experience running these studies, there are a few things we realized you can never fully understand from just the studies themselves.

User patience

When a study participant sits down next to you to walk through your application, they’re in no hurry. They’ve committed the next 30 minutes to 1 hour of their lives to the purpose of completing the study. Compared to the average 9 second attention span you’re likely to get from a real world user, this is too distant from reality to simulate.

Unfortunately, there isn’t much you can do to recreate a frantic in-the-wild scenario. This negatively affects the ability to learn about edge-cases and error handling the most. A usability study user is likely to run into some errors that they are patient enough to figure out during a study, but would otherwise cause them to throw their hands up in disgust. To counteract this, its best to read between the lines when it comes to negative reactions. Multiply observed frustration during studies by many magnitudes to really understand how a user may react.

Also, try to develop an understanding of how user patience can be a big factor. Spend time testing your home, promo, and marketing pages with an emphasis on asking for a stream of consciousness verbal description of what they see with a time limit. After all, if they stick around long enough to try your application, you’ve won half the battle when it comes to user patience.

User attentiveness

Another problem is that study participants are usually more alert and attentive than their real-world counterparts. They’re willing to take their time and figure out a problem. They’re willing to devote their undivided attention to the task at hand. Compare this to the more likely scenario of an actual user trying out your application at 2 a.m. with the TV running in the background and after surfing to your promo page by coincidence. It’s night and day.

To counteract this, make sure you get immediate verbal feedback after each action they take, to get a sense for what they’re thinking before the interface has had time to soak in their consciousness.

User skepticism

Most deceiving of all is the mindset of a usability user. Depending on what your relationship is with the user (stranger, guest, friend, acquaintance, co-worker from another department, spouse, child) you need to realize they’re walking into the study with some intentions. Sometimes they’re participating because you’ve promised to pay them; other times they’re participating because they want to help you out. Whatever the case, nothing simulates user skepticism.

When a random user hits your site or application (say from Google), they’re coming with a guarded mindset. They’re cautious more than curious and they’ll immediately look for warning signs. A poorly designed interface with no obvious indications that there is active support or lack of no-obligation demos or screenshots are an immediate turn off. If they eventually try your application or dig deeper into your site, they’ll probably be less willing to try anything but the most obvious of features.

When a user hits your site or application from a referral or other more qualified method, they’re coming with a curious mindset. They want to dive right in and see what all the fuss is about. They’re also likely to be the ones who have heard of alternatives or are currently using an alternative and now they’re using your application to do some feature comparisons.

When a user participates in your user study, you get none of this. They’re neither overly guarded nor overly cautious. It is more likely that they feel you are working together in unison towards a common experimental cause. You really don’t get the stinging skepticism that a real world user carries.

It helps to keep these things in mind when conducting usability studies. Sometimes it’s too easy to pretend that you’ve aggregated all the feedback you can by running a well-designed series of studies. Even armed with that data, keep in mind that the best feedback is silent feedback. Make sure to analyze your logs. Don’t dismiss your analytics tools. Look over real user data from real users in the wild. Couple that data with usability studies and only then do you have a good macro- and micro- level understanding of your own application.