Don't let a metric become a data science.

As a business or function grows, it becomes important to measure this growth across various dimensions.

However, what's equally important is to avoid diving into the abyss and chasing perfection. It's very tempting to continue crunching numbers, way past the point of diminishing returns.

There are two guiding principles here:

Begin with the end in mind Realize that data is a means to support your decision making, not make it. Gather your ideas before diving into the data.
80/20 principle 80% of the value is arrived at with 20% of the work. The first iteration might be the best. Avoid letting a metric become a data science. Do not keep swimming, do no pass go, do not collect $200.

For example, let's say you're selling Pickleball (I'm addicted) paddles that cost you $25 to manufacture and stock. You set up some e-marketing campaigns to kickstart your sales. Perhaps you are paying for Instagram and YouTube ads, targeting the demographic that doesn't have the time or energy to learn tennis.

It's crucial to know how much you're spending on ads in order to make each sale. This is your conversion cost.

How much does it cost per impression (viewer)?
How much does a click cost?
What's the drop-off rate?
How many impressions translate into a "conversion" (purchase)?
How many clicks on average convert into a "conversion"?
Or best/simply: how much is a conversion $?

Data allows you to ask the right questions about your growth.

In this simple example, we can directly see a relationship between spend and profit. A story begins to emerge which directly influences our business strategies.

However - metrics & data aren't always as simple as Pickleball -- let's dive into a more complex, real-world example that might eat up our time.

Let's imagine you're investigating customer churn in your complex developer focused SaaS. You have a product that customers love, and they tend to stick around for a while. Although - you've noticed that there's a weird little subgroup, ones that suddenly terminate their subscription without notice.

Is it onboarding? Is it support? Did they go to a competitor?

And so, the discovery begins.

You're excited to find that you have hoards of data available to help in your analysis.

Data Type	Quantitative	Qualitative
Product Usage	% of features "used" call/storage volume/throughput ...	Did the customer understand the feature?
Product Maturity	- completed the in-product training? - Product Certification completed?	- were your customers champions? - Did they love your product? - Would they evangelize it internally? cross-functionally? publicly?
Customer Engagement	- # of emails from customers? - Message sentiment analysis %	- Does our CSM/AM have a good relationship with the customer? How happy is the customer, generally?
Technical Support	- # of support tickets - response & resolution times	What are the kinds of questions that come in?

Wow - there's loads of data here. Where do you start?

The biggest mistake that can be made is to simply dive into the data, without a clear hypothesis.

Let's play out what that would look like:

Normalize all the data
Plot it a million times, across a bunch of different dimensions
Try to make sense of the random clusters of users in your plotted data sets, thinking it means something
Get lost in standard deviations
Find yourself using linear regression to try to predict things that don't make sense
...?
Cry and weep
Find infinite visible patterns
Find no visible patterns.
Realize you've made a mistake

Repeat after me: Do not start with the data; start with a hypothesis.

We can't find what we need by mindlessly diving into data, expecting to miraculously find some silver bullet pattern that explains things.

Instead, let's begin with some educated guesses about what might be causing customers to walk away:

Were they onboarding/onboarded well?
Did they make it to "production" / "go live" ?
Were the use cases too "edgey" and caused them to spend too much time fighting the product?

What data would support or invalidate this hypothesis?

Always start with a hypothesis and look for data that invalidates or supports this hypothesis. Once the hypothesis is there, attach some metrics that are relevant to the hypothesis. It's helpful to measure proxy targets in addition to the main goal.

For example:

You hypothesize that customers are churning because the product is too complex.
This is evident in an abnormally high amount of support tickets. You can see a relationship between churned customers and higher than average submission of technical tickets.
The support tickets on average have faster resolutions, suggesting the user is struggling to understand the product instead of running into bugs and/or feature limitations.

Solution: Building an in-product tutorial will reduce time-to-value, resulting in overall product confusion, faster time to integration and an overall happier time.

After forming your hypothesis, identify which metrics can help validate or invalidate it. It’s not about tracking all available data; it’s about tracking the right data.

Metrics to watch

Number of support tickets created
duration from sign-up to "activated" / "in production"
Number of calls with customer success engineers

Note - the key word is watch, not target.

As we test our hypothesis, the main thing to note is how this will affect churn. But of course, churn can take a long time to play out in practice, and our results might not be evident until late. So we can also measure our proxy metrics to build an intuitional understanding of where we think we're heading.

A key warning is letting your metrics be just that, metrics. As per Goodhart's law, the moment the metric becomes a target, the metric ceases to be a good measure.

For example

duration from sign-up to "activated" / "in production"

It might seem natural to set a KPI or official target that aims to reduce this time. A well-meaning team may start optimizing to hit that target at any cost, rather than genuinely improving the customer's understanding and value from the product. This could lead to:

Rushing the onboarding process to reduce the measured time without genuinely enhancing the user's experience.
Pressuring customers to complete steps without ensuring they fully grasp them, leading to superficial activation but no real understanding.
Encouraging quick adoption of features without proper consideration if it's beneficial for long-term use or customer satisfaction.

Summary

The key takeaway is to use the metric as a guide and feedback loop rather than a rigid target that must be hit at all costs.
Data is used to validate and monitor strategies, not create them. Do not get stuck trying to make sense of data in a vacuum.
Do not let your metrics become KPIs