5

By 'professional' study, I mean hiring an outside company to run the test (typically $10k+). By 'guerilla' I mean hallway testing, or 'Rocket Surgery made easy' style testing.

What is the benefit of the paid-for study?

Jon White
  • 1,798
  • 13
  • 21
  • There are a lot of things in-between the two -- are those off the table & the focus is just on the question you asked about what you get with an agency? – jcmeloni Mar 24 '12 at 00:16
  • I guess my question is really around what do I get over and above a guerilla test. So they are on the table. Should I edit the question? – Jon White Mar 24 '12 at 04:33
  • maybe? I mean, there are agencies that charge 10k+ & don't do much, and agencies that do a lot & don't charge 10k+, and a lot of in-house "guerrilla" work that gets a lot done, so for a useful answer it might be good to tease out the nuances of the situation. – jcmeloni Mar 24 '12 at 11:50
  • @jcmeloni I was going to edit but the answers below have addressed what I intended. – Jon White Mar 24 '12 at 15:25
  • It depends on how good your internal research capabilities are, and how good the external agency is... Hopefully you know exactly what you are getting for 10k :p – Michael Lai Jul 21 '14 at 00:41

3 Answers3

4

Preliminary remark: The list of benefits might be a bit different depending on your type of product and company size.

Benefits you can get from hiring an agency:

  • Questions to better understand your product and for a proper test setup, which can already be enlightening
  • Eventally a Heuristic Review upfront (aka Expert Review) to get to the hypothesis they want to test (can also be helpful itself)
  • Save time, i.e. on recruiting testers and organizing the test sessions themselfs
  • Often recruiting users matching your target group is done for you
  • If done externally, then you'll get a room with all the equipment like a usability lab with observation room and one-way-mirror, camera/microphone, recording sofware etc.
  • A list of all the 'usability bugs' they found, along with video recorded and hopefully a highlight video that you can show e.g. to the management
  • Credibility - especially in larger companies agency testing can pay off in terms of credibility of the results
  • Usually addresses concerns and/or **avoids discussions around the target group, small number of testers delivering sound results etc. - if not, the agency will advocate here
  • Some agencies offer consulting based on the results they found how to improve things
  • 'Access to more expertise' as separate point, brought up by PhillipW (thanks!)

The question was about benefits you can get from professional agency testing. Given a good agency I think professional testing can be really beneficial, especially if the company has't investing much time in testing so far. However, I think that guerilla can also be very beneficial, sometimes even way more than agency testing. But this would be off topic as the question is about the benefits of agency testing and not comparing the two.

greenforest
  • 8,786
  • 5
  • 37
  • 55
  • The subject line of the question to me suggests that a comparison is what's being asked for... – PhillipW Mar 24 '12 at 18:12
  • @PhillipW Hope I got some points addressed here. Feel free to add to this :-) – greenforest Mar 24 '12 at 19:34
  • Editing other people's responses feels sort of 'wrong' ! The only thing I'd add to your list is my 'access to more expertise' comment in my answer. – PhillipW Mar 25 '12 at 19:54
  • I am fine with others editing my posts, if in the interest of creating a great knowledge base. Anyway, thanks for the comment - added your point to the list above. Cheers! @PhillipW – greenforest Mar 25 '12 at 20:18
2

Access to more expertise. I'd say it was almost impossible to quantify this though.

An expert may just pick up one extra problem - but it could be the one 'killer problem' which is most important to deal with.

PhillipW
  • 11,290
  • 2
  • 33
  • 55
1

Trust and Credibility. You know when you pay a credible source, assuming that the consultants credentials are valid, that the results are trustworthy and reliable. You can use them inside your organization as a tool to convince management and sponsors. You will have higher chance of getting usability improvements implemented.

A guerilla test is not reliable, but it can still be accurate. The problem is that you don't know if the guerilla test is accurate or not. I would be reluctant to use a guerilla test as a tool to convince anyone. It would be shredded to pieces before I could say my first "But...".

As in many other things in life: What you pay is what you get (WYPIWYG)

Benny Skogberg
  • 54,996
  • 22
  • 140
  • 241