🐣 How to measure leaky KPIs

What to do when users go off-platform

Here’s another 60-second read: one insight, 300 words, super fast 🏎️ Read online here or download PDF​

Many products struggle with leaky buckets 🪣 But not in the way you’d expect.

For some products, the magic moment happens off-platform. That means that you can’t see whether users succeed.

This is often the case for marketplaces where users have the opportunity to take the action on another platform. For example:

• Job search pages → but the person applies elsewhere
• Hotel booking sites → where the person books directly with the host
• Comparison sites → where the final decision happens on the provider’s site

This leads to under-counting your success. You might conclude that a product test didn’t work earlier in the funnel, without knowing if it actually impacted the end.

The result is that you can under-predict success. You lose visibility at the end of the funnel.

So, how do you track it?

There are two ways:

​

1) Track via springboard UI

This is where you capture answers just before they bounce.

📌 Example: LinkedIn’s ‘Did you apply?’

Now, LinkedIn has a small blue module on job ads that asks if you applied (given that this action happens on another company’s site).

There’s a few things I like about this execution:

1. It’s super simple, just a Y/N question (increases the likelihood someone will engage
2. It’s visible very high up in the module’s hierarchy (more eyeballs = more conversions)
3. It stands out with a block colour (important especially on a busy module like this one)

​

2) Follow up via email

If the user’s already gone, catch them in their inbox.

📌 Example 2: Otta’s ‘did you get the job?’

​Otta is a job search site that was recently renamed Welcome to the Jungle. I used it to find my last full-time role back in 2021.

What was cool is that they asked me if:

• I’d got an interview
• I’d heard back from my interview
• I’d got an offer

Why it works:
✅ 1 question per email
✅ Clear subject lines
✅ No fluff

This is their only chance to get the data in the door on whether their users reached their magic moment. So they want to increase the conversion rate from receiving an email to answering a question.

​

To conclude: it won’t be perfect, but at least it’s something

Next time you’re measuring success, ask yourself:

Most metrics only tell part of the story. If you’re limited, try layering on user interviews to understand the onward journey in more detail.

Seen any clever ways to do this?

Lmk — I’d love to include more examples!

Thank you SO much for reading (all the way to the bottom, wow look at you go).

See you next week,

Rosie 🕺

​​