So you talked to your users, understood their needs, helped the product manager prioritize features, and built a feature to help skyrocket your product metrics.
After months of sleepless nights, countless bug fixes, and several energy drinks, your team finally pushed it to production. But you know your job doesn’t end there.
You need to talk to your users, understand how they’re using it, and gauge if your new feature made any difference at all.
A quick way to do this is to track your company’s sales and revenue. If you built the feature to up-sell users to a paid plan, you can quickly open up your company’s sales and revenue numbers, and see how it correlates to your feature’s usage.
But what if your feature isn’t directly connected to a pricing plan?
At the end of the month, quarter, or year, how can you be sure the features you shipped had an impact on your business?
The 3 Step Process To Measure The Success Of A Feature
1. What’s The Goal?
Before you even start building your feature, go back to your drawing board and ask yourself why you’re building it.
If you don’t actually know what you’re trying to achieve, it’s also hard to measure success. Try it, you'll see what I mean.
— Avinash Kaushik (@avinash) December 9, 2014
Sure, your users asked for it. But that’s not why your product team prioritized the feature and set out to build it.
You spent weeks talking to your users, took them to coffee shops, and jumped on demo calls even if it was late in the night. You took your learnings from these meetings, tied them together, and came up with a hypothesis to define your users’ pain points.
And only after receiving sufficient data to suggest your hypothesis could move the needle, you finally set out to build the feature. Right?
Right.
Whether you’re building games for teenage kids or selling internet software to Fortune 500 companies, every feature you build is tied to a bigger business goal. The features you ship not only help your users solve their pain points, but also help grow your business by increasing:
- Acquisition
- Engagement
- Retention, and
- The conversion of a free user to a paid user.
When you define the goal early on, it keeps you focused on the right questions. And more importantly, it doesn’t let your feature get watered down.
Takeaway: Establish what your feature is trying to accomplish for your company.
2. Define The Metrics
Once you’ve defined your feature’s goal, it’s time to think through the metrics you’ll be using to track your feature.
The internet is filled with posts telling you the exact KPIs and metrics you must track for your product. Unfortunately, you can’t take a one size fits all approach with KPIs and metrics.
Depending on the goal of your feature and the industry you’re in, the key metrics you track will vary. After all, not all features are the same.
For some companies, say a business product, 50,000 paying customers might help them achieve unicorn status, while the same number might not be considered large enough if you’re a consumer product.
Andrew Chen talks about this in his recent blog post and gives examples to explain why DAU/MAU, a KPI popularized thanks to Facebook’s success, need not be your north star metric even if you’re a consumer product.
Of the several examples he shared, two of my favourites are:
- If products like AirBnb’s and Booking’s value were measured based on their DAU/MAU numbers, they might not exist today. These products are used once or twice a year and yet they’re multi-billion dollar companies today.
- Similar to the travel industry, e-commerce stores aren’t used everyday by customers either. And yet, there's plenty of multi-million dollar e-commerce companies and at least a hand full of companies worth more than a billion.
Now that we know we can’t simply open up an article titled “Key Metrics Everyone Should Track” and decide to use them for your new feature, let’s talk about how you can come up with metrics that matter to you.
There’s no silver bullet to define the right metrics. But the good news is, there are questions that you can ask yourself to help you define them.
Teach a man to track a metric, and you feed him data. Teach him to choose the right metric, and you feed him intelligence for a lifetime.
Questions To Ask When Thinking About Feature Metrics
A. What Do I Want To Learn?
A lot of us often start by asking the question “What do I want to track?”, when it should be “What do I want to learn?”. Asking “What do I want to learn” helps put the user at the centre and prevents us from focusing on events, which often leads us to vanity metrics.
When you avoid vanity metrics, you take a step closer to gaining insight on user’s behaviour and prevent false proxies that look good only on paper.
B. What Actions Should Users Perform?
Think through what actions your users must perform to successfully use your new feature. Ideally, you should list down all actions from the moment your users open your app all the way till they successfully use your feature. At this point, do not worry if they’re trackable, or the complexities involved in tracking them.
For example, if you’re building a photo sharing feature into your app, the actions your users will need to perform before they can share a photo will include: Opening the app, clicking on the upload photo button, opening the gallery, selecting the image from the gallery, editing the image, and adding a caption.
When armed with the knowledge of what actions your users must perform to successfully use a feature, it becomes a lot easier to find the depth at which your users use a feature and frictions that might hinder them from seeing any value.
C. When Will Users Find Value In Your Feature?
While we’d all like to see users derive value directly from a single feature, it’s often dependent on how they use other features in your product. This is especially true if the goal of your new feature is to increase engagement or retention.
For example, if you’re building a dashboard for your product, the dashboard will not help your users gain any value unless they extensively use other key features in your product.
Having a hypothesis on when you expect users to see value in your feature will help set the expectation for frequency of usage, the adoption rate, and help evaluate whether your feature is a strong driver of engagement, retention, or conversion to paid subscriptions.
Metrics can help understand your customer's behaviour and serve as a starting point for identifying issues. But when you’re backed by the right metrics for your feature, the result is a product team that knows the ins and outs of a customer’s behaviour and their needs.
Takeaway: Measure to learn. Not to track.
3. Gather Feedback After Launch
Once you've gone through the product development process and your features are out in the open, you’re probably breaking your head trying to understand why a user performed a certain action, but didn't completely use your feature.
While analytics and numbers can help you understand what your users are doing, talking to them will help you understand the WHY behind every action. Organizations have started to realize the importance of customer feedback and are jumping at every opportunity to talk to their users and understand how they’re using their features.
You probably have multiple tools set up to help you get feedback from your users already. But to ensure you get feedback that helps you learn about why your users used your feature a certain way, there are few key points that you might want to keep in mind.
Key points to keep in mind when gathering feature feedback
A. Know Why You’re Reaching Out
Before you even pen down the questions, think about why you’re asking them for feedback. Consider asking yourself what you want to learn about the customer once you’ve got their feedback.
Your questions could range between trying to understand why your users aren’t performing a certain action and to see if your users are using the feature to solve the right problem.
Ultimately, if you end up asking the wrong questions, you’ll have a bunch of information that will not be helpful to you or your organization, and there’ll be nothing you can do about it.
“If you don’t use the information you’re asking for, you’re wasting your customer’s time. You’ll have a whole batch of responses to look through and none of them will make a difference.”
~ Lars Lofgren, Former Dir. of Growth at Kissmetrics
B. Ask The Right Users
No matter how well you frame the question to avoid any bias, if you ask the wrong questions to the wrong person, you will most likely not get any response.
Before asking for feedback, make sure to segment users based on their feature usage and ask them relevant questions. You don’t want to send an email asking “What works well about X” to a user who hasn’t logged in to your product for more than a month, right?
C. Avoid Yes/No Questions
Your users might surprise you when you start with open-ended questions. Open-ended questions help you understand what they’re thinking and remove the chances of you leading them to an answer.
A great way to keep the question open-ended and avoid falling into the trap of confirmation bias is to ask questions based on their recent usage.
For example, a simple "When you tried to use X, what were you trying to achieve?” is a great question since it’ll make the user talk in detail about how they tried to use your feature, giving you context and more information about your user. On the other hand, a yes/no question or a multiple choice question wouldn’t have given you this insight.
Takeaway: Customer feedback informs if your users love your feature, or if it's just another nice to have feature.
Conclusion
When you sit down to look back and analyze what your team achieved in the last few months, you don't want to know just the number of features you pushed to production, or the number of bugs you squished. You want to know how your feature moved the needle for your business.
Sure, when you see your analytics tool and read the customer feedback, you’ll have a pool full of data.
But when you know what your feature is trying to achieve, have the right metrics to give insights into your user’s behaviour, and ask the right questions to your customers, you’ll fall in to a pot full of gold that’ll turn your report from a “We pushed two big features last quarter” to “Our last feature helped increase the retention by 9%”.
