How early-stage startups can use data effectively

It is a commonly held belief that startups can measure their way to success. And while there are always exceptions, early-stage companies often can’t leverage data easily, at least not in the way that later stage companies can. It’s imperative that startups recognize this early on — it makes all the difference.

In this piece, I draw on my experiences using data to take Framer from seed round to Series B. More concretely, I’ll describe what to (not) focus on, and then, how to get real results.

There are good and bad ways for startups to use data. In my opinion, the bad way unfortunately is often preached on saas blogs, a/b test tool marketing pages, and especially growth hacker conferences: that by simply measuring and looking at data you’ll find simple things to do that will drive explosive growth. Silver bullets, if you will.

The good way is comparable to first principles thinking. Below the surface of your day to day results, your startup can be described by a set of numbers. It takes some work to discover these numbers, but once you have them you can use them to make predictions and spot underlying trends. If everyone in your company knows these numbers by heart, they will inevitably make better decisions.

But most importantly, using data the right way will help answer the single most important – but complex – question at any moment for a startup: how are we really doing?

Let’s start with looking at what not to do as a startup.

• Common pitfalls
  • Don’t measure too much
  • A/B tests are anti-startup
  • Understand your calculations
  • Don’t just use quantitative data
  • Use the real numbers
• Make a plan
  • Define a high-level product funnel
  • Define a metrics plan per stage and implement it
  • Get your whole team on board
• Perfect setup
  • Decide on infrastructure
  • What to build, ideally
• Typical problems
  • Non-measurable data
  • Defining what results are “good”
  • Turning data observations into actionable learnings
• Discovering data
  • Visualization
  • What to look for
  • Analyzing your splashes
  • Look for trends that are growing structurally
  • Look for trends that are growing exponentially
  • Look for trends that are dropping of cliffs
  • Look for predictors: correlations
  • Deciding what to work on

Technically, it’s easy to measure everything, so most startups start out that way. But when you measure everything, you learn nothing. Just the sheer noise makes it hard to discover anything useful and it can be demotivating to look at piles of numbers in general.

My advice is to carefully plan what you want to measure upfront, then implement and conclude. You should only expand your set of measurements once you’ve made the most important ones actionable. Later in this article, I provide a clear set of ways to plan what you measure.

To make decisions based on data you need volume. Without volume, the data itself is not statistically significant and is basically just noise. To detect a 3% difference with 95% confidence you would need a sample size of 12,000 visitors, signups, or sales. That sample size is generally too high for most early-stage startups and forces your product development into long cycles.

While on the subject of shipping fast and iterating later, let’s talk about A/B testing. To get reliable measurements, you should only be changing one variable at a time. During the early stages of Framer, we changed our homepage in the middle of a checkout A/B test, which skewed our results. But as a startup, it was the right decision to adjust the way we marketed our product. What you’ll find is that those two factors are often incompatible. In general, constant improvements should trump tests that block quick reactionary changes.