How do you track new feature/product adoption?

Question

Huzzah! You just launched your brand new feature/product X. You've been working on it for months, usability tested the crap out of it, and feel really good about how it all came together.

After a few days you become impatient that no one has mentioned your new feature. No one loves it. No one hates it. Maybe no one even cares at all.

How do you answer the question: is the thing that I just built actually helping anyone?

That can be hard to answer qualitatively. What if you knew, "2% of people have used feature X?" Is that any better? What about 10%? Are we getting warmer [and helping users achieve their goals] or just making it more likely for X to be used?

Knowing the answer also helps you build better features and products in the future.

How do you use data to help your teams build better products?

Just realized there are three pretty distinct questions in there. It all feels like one big blob of a thought in my skull. — noluckmurphy, Aug 17 '10 at 20:24
Is this a "how do you" or a "how should you" question? Cos they're probably 2 quite different things. :) — Mal Ross, Aug 18 '10 at 12:10
Both, I guess. If you were doing something you felt was ineffective, I'd want to know that too. — noluckmurphy, Aug 23 '10 at 14:34

score 4 · Answer 1 · answered Aug 18 '10 at 10:47

just a few random thoughts...

If possible, you could build a function into your website/product, so that actual use is measured. This would of course be interesting data for every product, not just new features. Could be used to prioritize usability efforts for example. You could even make the product simpler by removing unused features -- without the data, convincing a product team to remove things can be nearly impossible.

But any single number is useless without context and comparison, so you would need to collect the data before and after changes, or even continuously.

Come to think about it, how does my company get along without "anomynous usage data"?

I agree with the "need data to support removal" sentiment. – noluckmurphy Aug 23 '10 at 14:32 — noluckmurphy, Aug 23 '10 at 14:32

score 4 · Accepted Answer · answered Aug 18 '10 at 21:41

4

I find it easier to ask this question before we build the feature. If we have our success criteria before we start designing then it becomes much easier to figure out whether we've succeeded or not, and it also helps drive the design process in the right direction.

answered Aug 18 '10 at 21:41

adrianh

10,338
2
24
39

Follow up: do you work on agile team(s)? If so, how would you attempt to answer, "how much usage/adoption is good? how much is bad?" I feel like I'm pulling numbers out of my ass when we're looking at brand new stuff. It's much easier when you're trying to improve upon something you already have metrics for. – noluckmurphy Aug 23 '10 at 14:37
Usage or adoption, in itself it not necessarily the right measure. What business problem is your product trying to solve? What is the reason the product exists in the first place? eg. A co. I work for has goal of reducing errors on customer applications. They redesigned a form and 50,000 per month switched from paper. The error rate didnt really go down though, customers were still not providing all the information needed. Usage is not a problem here, the design of the form is. – Nathan-W Aug 23 '10 at 19:58
Spend less time on X, more on Y. 2. Win more Z.

Some of these metrics are outside the boundaries of the app. However, I realize I need to align my measurements with these intrinsic (to the customer's process) goals. Ultimately everything we do should be helping those two things

noluckmurphy

Aug 24 '10 at 13:20

@Kyle With agile teams look to the stories that you're implementing. Each story should be delivering business value. Each of those stories should have acceptance criteria. Ask your business owner what they would consider success/failure for that bit of business value. You might also find googling around "Customer Development" and "Lean Startup" gets you some useful ideas. – adrianh Aug 24 '10 at 14:33

score 3 · Answer 3 · answered Aug 18 '10 at 11:00

I agree with Lisa's comment about running reports that measure functionality usage.

The BBC News website (popular UK news website) just did a site redesign and a few weeks later a JavaScript overlay appeared over the page asking people what they thought of the changes, so a questionnaire is one way to do this. P.S. I really like the way Stack Overflow or UXExchange notify users with banner messages right at the top of the page, which might be a preferable approach to using overlays / popups.

The recent gmail changes have had mixed reviews, and there's a gmail blog which talks about new features and requests feedback on recently released features, which users can comment on, so you could introduce a blog on your site. Dropbox have this also. And facebook.

You could even do something like display "I like this change / I don't like this change" next to the new or changed UI (a bit like facebook newsfeed items) to see what people think of it. I know this clutters the UI, but you could set it up so that once it's clicked on, it never appear again. I haven't seen this anywhere before in the context of measuring success of new features but it doesn't seem a bad idea for certain new features if done sensibly.

If you're going to use a popup user feedback survey - DO NOT MAKE IT ASK TOO MANY QUESTIONS. Otherwise users won't complete it. (BBC take note !) — PhillipW, Aug 23 '10 at 09:01

score 2 · Answer 4 · answered Aug 22 '10 at 20:40

Like Adrian says, define your measures from the outset. Then you will probably find a standard analytics reporting tool like Urchin will capture the data you need, if your site/app is set up to pass the correct data.

Clickstream is easy, but if you want events or statuses or outcomes (ie you want to record key points along the user experience) you do need to do a little bit of thinking or programming.

Metrics we tend to measure are thing like: - there's 2 links on this page that go to the same place (positioned top of screen, side of screen for example) but which one did users actually click? - how many times did a data input error occur, which field and what was the error (usually its mandatory fields not completed)

score 0 · Answer 5 · answered Jun 05 '15 at 19:39

In my experience, we almost always know the feature is useful to people because we know the purpose of the feature and the overarching business problem we are solving. To me, the communication about how to extrapolate the value of the feature is lacking. I don't mean a how-to guide, but an explanation of the concept and how you are solving their problem. Try coding in some in-app messages that point to buttons and features and explain their purpose. The challenges with this are all the filters you have to apply to keep from bugging users over and over and over with the same messages. Once it is right though, it is gold for your product team. You may even consider prompting for feedback on click events.

How do you track new feature/product adoption?

5 Answers5