Experiment

How To Easily Measure Your Product Performance

Yahia El Mejjad

• 6 min read
How To Easily Measure Your Product Performance
Illustration by Hatim El Kaddaoui/Motion Graphics Designer

During a job interview with Revolut last year for a Product Owner role, the hiring manager asked me to solve this problem:

“There are X screens to go through in our app to be able to open an account, but only a certain number of users who downloaded the app make it to the last screen and open an account: X% drop at the first screen, Y % at the second screen, Z % at the third, etc. How do you go about solving this problem?”

Revolut is a very data-driven bank, so unsurprisingly, this is the kind of product metrics-driven approach that helps create better features, and this is what I’m going to talk about in this article.

In 9 Rules For An Actionable Product Dashboard, I laid down the process for creating a product dashboard and tracking product metrics. Now, I’ll explore how to define your product metrics to measure product success.  


Let’s step back a bit

Why Is Your Customer Using Your Product In The First Place?

IT products are essentially automation replacing manual processes that would otherwise take ages to complete (or even never complete). Think about all the services we use daily, from food delivery services to banking apps.

Example of a Customer Journey to order a Pizza

Today
I go to the Deliveroo app, which gives me access to a wide range of restaurants. I can choose the best pizza at the best price, and most importantly, I can do so easily, with one click from the comfort of my sofa. I place my order and the pizza is delivered within one hour.

Back in the days  
I would need to call pizza restaurants around, ask for the menu, and maybe go out and pick up the pizza myself. It takes time, effort, and the result is never fully guaranteed.

In fact, your customer uses your IT product to be able to get more (quantity), better (quality), cheaper, and quicker. And these 4 parameters are what, as product teams, we want to maximise for our clients. I will talk about the price and duration elements in another article and discuss how to track quantity and quality here.

Product metrics are specifically tracking qualitative and quantitative performance. The end goal is to understand whether your customer is getting the full value of your product (i.e. receiving more and in better quality) by tracking a few key product metrics. And as a result, you are able to make informed decisions on your next actions or Product features.

Is Your Customer Getting The Full Value Of The Product?

Today’s products have at least two elements in common. First, product use case(s) are made of a succession of phases executed in tandem by different parties (the parties’ involvement in the phases differ depending on the product):

  • Party 1: Your customer or app user
  • Party 2: Your apps and back-end automations
  • Party 3: Your Customer Support team (internal)
  • Party 4 (for B2B products): Your customer’s teams (external)
  • And so on

The other common trait is that product use case(s) aim at maximizing an output key metric in comparison to its input value. Example:

  • B2C example - Deliveroo makes sure all the Pizzas ordered (i.e. input) on their app are processed properly throughout the phases, passed to restaurants, and all delivered on time (i.e output)
  • B2B example - Traceability Supply Chain solution providers make sure all barcodes printed (i.e. input) are attached to items and scanned, those items are shipped and all delivered to physical stores (i.e output)
  • B2C example - Revolut makes sure all account opening attempts (i.e. input) successfully go through the important onboarding steps, and are all fulfilled (i.e output)
💡
In other words, I know that my customer is getting the full value of the product when a given metric is maximised throughout the different phases in comparison to its entry value

This can be translated on a graph:

In an ideal world (the graph above), the key metric is maximised throughout the product phases, meaning that if, for instance, we have 1M pizzas ordered on Deliveroo (phase 1), we get exactly 1M orders passed to restaurants (phase 2) and 1M orders fulfilled at the end of the chain (phase 6).

The reason why this chart is almost never flat is because of friction:

  • Human friction (human intervention in different processes)
  • Tech friction (bugs)
  • Missing scenarios that need to be implemented

And our goal in product teams is to implement features and improvements to remove these three barriers so that our graph looks more like a flat line: In fact, the less friction in the processes we have, the straighter the line and the more value the Customer is getting. And by the way, in practice, it’s a never-ending circle: the more features or phases we add, the more friction we generate and the more we need to remove friction.

A typical example of a product performance graph would look like this:

Due to friction, the key metric is losing value throughout the 6 phases in this example: -20% from Phase 1 to Phase 2, then -11% from Phase 2 to Phase 3, etc.

Build Your First Product Sucess Graph

Let’s assume for simplicity that your product is a food delivery app.

1/ Identify the important phases

Start by identifying the main phases where your customer gets value from your product.

Your food delivery app product use case is completed in five important phases: Order accepted - Order in preparation - Order ready - Order picked up for delivery - Order delivered.

2/ Your first product metric

Choose your “maximisible” metric. This is your first product metric and the base of all the other metrics.

In our food delivery app example, the metric we want to maximise is the number of orders.

3/ Product performance periodicity

Choose a periodicity that is relevant to your product performance analysis (e.g. daily, weekly, monthly).

4/ Et voilà!

Your graph is a big high-level hint at your product performance

Delivery app. Product performance measured in April

The question you want to ask yourself now is: How is each phase performing so that I can identify improvements? This can be answered by defining the rest of the metrics.

Define The Rest Of Your Product Metrics

We’ve established that your customer gets the full value of your product when the graph is a flat line. When applied to our delivery app example, this condition is fulfilled if:

Metric 2 = Success rate of accepted VS placed (phase 2 VS phase 1)
Metric 3 = Success rate of in prep VS accepted (phase 3 VS phase 2)
Metric 4 = Success rate of completed VS in prep (phase 4 VS phase 3)
Metric 5 = Success rate of picked up VS ready (phase 5 VS phase 4)
Metric 6 = Success rate of delivered VS picked up (phase 6 VS phase 5)

are all equal to 1 (meaning that input equals output). In practice though, these rates will almost never be equal to 1 but will tend towards 1. In some cases, given the complexity of the phases and the product, the customer would want a rate to only reach X% in a given period and to improve over time.

💡
Since not everyone is familiar with rate metrics (e.g. "66% of placed orders are accepted"), you can use simple numbers to indicate that: "only X numbers of orders placed are accepted", OR even better: "accepted orders are Y short". The idea is that your metric should be very easily understandable by your audience

With this, you’ve defined your 5 metrics or so to track in your Product dashboard.

Conclusion

You now have mapped the 5 Product metrics PLUS a very high-level graph showing your product performance. Go back to 9 Rules For An Actionable Product Dashboard for more information about how to create a product dashboard that works for you.