Issue #17: Validating Ideas CAN Be Easy (F.A.K.S)
Or Things We Need to do Before Worrying About A/B Testing
Hola Friends 👋
Last March, I opened my calendar to other Product Managers out there for a free 30 minute speed mentoring session! It was so much fun and I’m thinking of doing it again. But while I sort out my availability, I thought I’d share some of the popular questions I got (and my answer, of course).
I’m starting a new series in this newsletter called F.A.K.S
aka Folks.Asked.Kax.Series 😂
So on toToday’s FAKs!
Q: How can we experiment and validate our ideas if we can’t A/B test?
A/B testing is probably the best way to validate an idea. It’s basically science telling us that our idea is good or bad. Hard to contest that.
Realistically speaking, A/B testing is not easy. It’s expensive. You need specific tooling and expertise to do it well. It’s definitely an investment every Product Team should make.
But it is NOT the only way to experiment. And honestly, I don’t think it should be used for every kind of testing we need to do.
***
5 years ago, when I was in a team building a Messaging product, a Product Analyst joined our team. He was probably one of the smartest data people I’ve ever met. But his first few months with us had a lot of friction.
He was frustrated that we weren’t A/B testing at all.
My Tech and UX counterparts and I tried to explain to him the limitations of our context. But for our new analyst, we were not doing our jobs well enough to make A/B testing a reality for us.
There was a huge fight. 😂 (I’m laughing now but back then, you can cut the tension with a butter knife).
At the end of the day, we were both wrong - We all limited ourselves by thinking that experimentation was A/B testing alone.
Experimentation, at the end of the day, in the context of product building (we are not curing cancer here, folks!), is a way for us to gain confidence about whatever it is that we’re trying to do. It’s for us to learn more, make good decisions, and/or mitigate risks about an idea.
Simplifying the problem for ourselves - it was really about how we can get data and data about what.
It was our UX lead who found the answer. And from then we followed this process for our experimentation, which has been enriched by trainings we’ve been part of, and practices we’ve copied from other teams:
00. Flesh out the Idea
Sounds simple enough but this is pretty important. It’s important to get everybody on the same page of what the idea may look like. No need to make it detailed though.
Example (resemblance to any real life product is purely coincidental)
An AI assistant that sends daily workout recommendations at the best hour, to one’s Apple Watch based on the user’s daily routine, mood, and many excuses.
01. Map out your Assumptions and Prioritize the Riskiest one
Every idea we have, comes with many assumptions. Things we think to be true. And these assumptions can be about many things, but can be categorized into 3 groups:
Feasibility - Can we build it?
Viability - Is it worth building for the people we want to build it for?
Usability - If we build it, can people use it?
02. Define any Action that can prove or disprove your riskiest assumptions
The goal of this exercise is to know if the assumption is true or not true. Framing it this way opens up a lot of options for activities to find out more. Any actions count as long as these actions can have an observable output.
03. Define your measure for success for your assumption. What is the behaviour you want to observe?
Tip: it has nothing to do with your product outcome yet
At this point, we declare the behaviours that we want to see and to what extent we want to see them (usually in a way that we can count) for us to be able to say “ok, this is good!”
04. Evaluate the results and learn!
After taking the action, it’s time to see if the behaviours we wanted to see have been observed in the way we expected them . And to understand why things happened in the way they did. Both are equally important to make an informed decision afterwards about the idea and how to move forward.
05. You proved/disproved your assumptions, Now what?
Now that the team knows a lot more about the idea than in the beginning. Is this the “no-or-no-go” moment?
Kind of. At this point, for this example, we’ve learned a couple of things:
That there’s appetite for the idea (Viable)
But there’s no capability to build it (Not Feasible)
And this can lead to many things: a reframing of the idea, new assumptions to test, and a whole lot potential.
So maybe it’s a no-go for the original idea until there’s more capabilities in the team. But it’s a go to build something that can give personalized recommendation to the users.
And without a single A/B test done. YET.
TL;DR
Experimenting with your ideas is really about trying out different ways to validate whether:
You can build your idea
You should build your idea
It's worth it to build the idea
It's about gaining confidence, with observable data, so you and your team can make informed decisions about your priorities and next steps.
So if you can’t do an A/B test yet - no sweat.
There’s a whole lot of ways you can do to validate your idea just the same. And the learning afterwards is what we’re really after in the first place.
What about you, dear reader, how often do you test your ideas?
💙
Kax
Did you find this post valuable? Know other people who might find this post valuable too? Sharing is caring! So feel free to forward this to your friends 🙏
Got questions, feedback, violent reactions? Or other learnings you want to share with other human beings? Leave them in the comments below👇
Hey there! Did this post resonate with you and you’d like to know more about how you might:
Become more confident and build your own toolkit to become a high-impact Product Leader
Increase your impact in your product organization in a way that is aligned with your strengths and your values
Prepare for your path to Product Leadership and create a strategy to get there
Or do you have a specific problem that you want to tackle and are looking for guidance on how you might solve them? For example - stakeholder management, defining the next step in your product career, or even public speaking and story telling.
If you're curious to explore how I could support you in your tech career, feel free to book a free discovery call with me.
You can also write me at hello@kaxuson.com or read more about my coaching here
I like your usage of tables and real examples to help support the rest of the post. It's great!