Awareness, Successful usage, and Outcomes: An opinionated framework for project metrics

Before starting on a new project, teams should write down target metrics that they expect to hit if the project is successful. A helpful format is to measure and set goals around the following three metrics: are people aware of the project, are they using it as expected, and is it producing business outcomes.

As part of creating a data-driven culture, teams kicking off a new project or feature should start by making estimates for the impact the project will make and indicating how they will measure that impact. For people who haven’t set project metrics before, this can be daunting: there are so many possible ways to measure success! Should all metrics be in-product metrics, or are customer surveys allowed? What if the project is trying to impact something long-term like renewal rates - will the team have to wait a year to determine if the project was successful? Is it best to have a single north-star metric, or does that generalize away too much nuance?

A good policy is to select three metrics for each project: awareness, successful usage, and outcomes. These three metrics give the team a lightweight “funnel” to frame their thinking about the results of the project, and provides a balance between immediate, actionable insights and quantifying long-term strategic impact. As an example, when for our district app store project at Clever we set three target metrics: number of monthly visitors, number of monthly leads requesting price quotes, and eventual revenue resulting from those leads.

Awareness - do people have a chance to use the project?

People can’t use something they don’t know exists, so as a first metric, teams should set a goal for the number of people who see the project. Awareness - usually the number of people who visit a page or see a new feature per month - is typically the most straightforward for teams to measure. While awareness alone is definitely not sufficient for calling the project a success, it’s certainly necessary! Forecasting the awareness metric helps the team get a sense of what level of marketing or promotion they will need to do, and upon release provides an immediate indicator of whether the project has a chance to hit its other goals.

Successful usage - are people using the project like we thought they would?

All projects have an intended behavior that they hope to support and facilitate; as a second metric, teams should measure the number of people who are using the project as intended. What “successful usage” looks like is highly specific to the individual project: for an e-commerce feature, this might be the number of people who engage with the feature and then add an item to their cart, or for an social network this might be the number of people who complete a post using the new feature.

Forecasting the successful usage metric helps the team get a sense of what they expect the funnel to look like from a user seeing the feature to engaging with it, and from initial engagement to “completing” their usage. In the planning phase, this funnel can also be used to check that the team picked the right target for awareness. After release, the team can inspect how people are progressing through this funnel and spot if there are issues preventing users from using the project successfully.

Outcomes - is the project providing value to our business?

It’s great when a project is actively used, but usage alone is rarely the intended impact of a project. Whether the goal is new sales, customer retention, a reduction in support tickets, or decreased cost, projects are funded because of the the outcomes they aim to produce for the business. As a third metric, teams should articulate what they expect it to mean for the business if the project is successful. If a team can’t clearly state what the business impact of the project would be, they probably aren’t ready to start building.

While awareness and successful usage can usually be measured in real-time with in-product instrumentation, business outcomes often take longer to materialize and more work to analyze. It’s not uncommon for teams to need to run a survey or pull a report 6 months after the feature was released to really understand the business impact of the project. This is why teams can’t just use business outcomes as their sole metric - while outcomes are ultimately what the project is trying to deliver, awareness and successful usage provide the real-time measures that the team can use to course correct after launch.

Forecasting the outcomes of a project encourages the team to look critically at their targets for awareness and successful usage and ensure they are appropriate given assumptions around conversion rate, average deal size, etc. A metric and forecast around outcomes also gives an opportunity to check whether the expected impact of the project is worth the investment.

For a team looking to pick a set of target metrics, a good place to start is forecasting the number of people they need to see the project, the percent of those they expect to use it successfully, and the resulting impact on the business. Metrics around awareness, successful usage, and outcomes provide a concise, balanced set of high-level targets that will help the team shepherd the project from proposal to release to long-term success.