2

As I mentioned in this question, I've been working on proof-of-concept applications for social network privacy and running usability studies to evaluate them for publication. I'm a beginner to user studies at best, and I'm ashamed to admit that I don't think I've been asking the right questions. I've generally asked users to compare the simplicity and usefulness of the applications to the privacy controls provided by other social networks, such as Facebook, but I'm curious if there are even basic questions I should be asking.

My questions are:

  • Are there any basic questions a user should be answering in any usability study?
  • Specifically in a comparative usability study, are there additional base questions a user should answer?
  • Can a user usefully "quantify" the usefulness or simplicity of something, even in a comparative case, or should questions be of a binary nature?
  • In a study trying to quantify the usability of an application versus simply make improvements, is there any advantage to asking open-ended questions?
Community
  • 1
ramblinjan
  • 529
  • 2
  • 14

2 Answers2

5

Before writing a survey, first do open-ended qualitative interviews, even if they're informal and brief. Here's a slideshow with tips about how to do that: http://www.slideshare.net/edanzico/user-interview-techniques

It'll give you a much better sense of how people think about the tools that you'll be surveying them about... things like the terminology you use, the way you phrase the questions, etc.

Quantifying questions aren't uncommon, but they're usually phrased more the way people think about them. For example, "I do x... Always /Sometimes / Never" vs. "What percentage of the time do you do x, y, and z?"

Kimberley D
  • 1,025
  • 5
  • 8
1

I strongly agree with Kimberly on interviewing user up-front for better context regarding what they will do, which will help you understand why they're doing it.

An important distinction - Do not blend market research (MR) and usability. Your interviews with the participant prior to the test should be to get a better understanding of their predisposition to the functionality you'll be testing. The test should be about their ability to complete a task.

You can supplement this with things like "time on task", "completion rates" or "user's ease of use rating" if you're trying to benchmark.

You should, however, avoid preferential questions or questions which assess a value to the product or feature. These questions should be left to market research studies.

In short - usability is less about whether the business should move forward, tweak or position a product in a certain way. It's about how to convert a known business problem (proven via MR) into a usable product.

p.s. To quantify use a Likert scale for responses, as Kimberly mentioned.

Dan Michael
  • 199
  • 5