When I first ran growth experiments I was an idiot.
It hurts to admit, but I’d make these painful mistakes ALL THE TIME. In fact, now see 90% of companies making the same mistakes today when running A/B tests.
Mistake #1: Confusing a hypothesis with a prediction
The best article I’ve read in 2018 is called “That’s not a hypothesis”. It’s written by the VP of Growth at Patreon, Tal Raviv, and dissects the difference between a hypothesis and a prediction.
We cover the article in episode 5 of the Growth TL;DR podcast with Tal Raviv along with a few other FAQs for companies running experiments.
Check out Tal’s article here if interested.
Mistake #2: Vanity metrics as success metrics
Let’s pretend you’re a B2B software company. You’re focusing on user activation rate so you run an A/B test targeting invited user emails.
If you pop the champagne 🍾 when your variation B has a higher email click-through rate than variation A … you are celebrating a vanity metric.
It doesn’t matter if more people click your email. It matters if more people activate as a user. If you don’t have a large enough sample size then DON’T RUN THE DAMN EXPERIMENT!
Pick other ways you can meaningfully move your metrics.
Mistake #3: Falling into the trap of “success theater”
When I ran an experiment, and it succeeded, I would write an article, do a presenation, and shout from the rooftops how cool I am. :facepalm:
However, when the experiment failed, I wouldn’t share it.
That’s what we call “Success Theater.” We are theatrical when we win. We hide behind the curtain when we lose. And that’s the biggest mistake.
Running experiments is about learning. We learn more when we fail than when we succeed. Share everything and don’t let your ego get in the way of being wrong.
Check out this article by John Cutler about Success Theater for more context.
Experiment Template
Next week, I’ll share the exact Google Doc experiment template we use to run experiments at HubSpot. Here’s a teaser of the document:
If you’re impatient (like me) and don’t want to wait until next week just do the following.
- Share this article on Twitter or LinkedIn.
- Tag me when you share it (Twitter is @sjtousley and here’s my LinkedIn)
- I’ll give you the Google Doc immediately
Adios. Talk to you next week.