You've spec'd out, wire-framed, and maybe even implemented a simple version of your website or mobile app. Maybe you even have a launch plan in place. You know that data and KPIs are a thing, but you don't know where to start?

I had the pleasure of addressing this very topic last Thursday when I guest lectured a class at Cornell Tech. The course is simply titled, Product Management, and is co-taught by Greg Pass (ex-Summize, ex-Twitter, and Cornell Tech's "Chief Entrepreneurial Officer"), and an old friend Leland Rechis (ex-Twitter, ex-Google) with whom I had the pleasure of working with at Etsy.

My lecture covered the basics of the data behind product development: how to measure and instrument your product, how to define performance metrics, and how to build effective dashboards to iterate and improve.

Read on to learn more.

The data behind your product

Broadly speaking, there are three types of data behind your product: behavioral data which measures how people interact with the product, marketing data that describes how people first discover and re-engage with your application, and database data that catalogs properties of your product (and business): users, content, inventory, revenue, etc. Fundamentally, behavioral data and database data aren't all that different (theoretically the latter could be constructed by the former). But these types of data are logged by fundamentally different technologies so we'll treat them differently.

Behavioral data

For most web or mobile applications, behavioral data falls under two forms: page-level analytics that reflect URL-based browsing patterns of the web, and event-based analytics that are more general and can measure your product at a much finer granularity.

In the early days of the web, everything was a page view. Ajax wasn't a thing, and for the most part, clicks on a website resulted in a new URL that would load in your browser. These are the assumptions behind web analytics. Page views are just that, sessions (or visits) are defined as a contiguous set of page views (see Google's definition of a visit), and unique visitors are defined over the time period you're measuring. Today, Google Analytics is the most popular web analytics tool (and it's free).

If your web application is javascript intensive, chances are many actions take place that aren't in 1-1 correspondence with page views. For example, starring a Tweet is an ajax-based action that doesn't require a page refresh. If your product is a mobile app, page views and URLs don't exist, and everything must be thought of (and measured as) an event. There are many event-based logging systems on the market today, for example the open source package Snowplow, or Mixpanel, which offers a free version for low-volume accounts.

Event-level logging generally has format of (event name, key / value attributes). For example, an event on a search page might have the name "event", and properties such as ("query": "bananas", "page": 1).

Marketing data

As you start promoting your product and getting users and traffic to try it out, download it, register, etc., you'll quickly find that the context in which people arrive to your product will have big impacts on how people use your product and engage with it.

Generally speaking, users who find your product via highly specific search keywords will have higher engagement than, say, someone who clicked through to your product via a Facebook post from an old high school friend. One of Etsy's incredible strengths was the long-tailed nature of their massive marketplace; they sell things that aren't available anywhere else on the web, and when users search for something like "American flag shorts" on Google, they arrive straight to Etsy in a ready-to-buy context. From a data perspective, time on site and conversion rates were very high for search traffic.

Contexts and common referrer sources include: organic search, social media, email, third party referrals (press, etc.), and direct traffic (people opening your app directly or typing in your url to their browser bar). If you're using Google Analytics for your web application, you can encode links to your website in a way that they're easily trackable via URL parameters—see Google's URL builder. Referrers are a bit trickier for your mobile application, but there have been some efforts in standardization.

Beyond referrer sources, user information can also have big impact on engagement: location, age, gender, and device type (mobile, desktop, or browser type). Understanding how each of these attributes correspond to product success can impact both future product development and related marketing efforts.

Your database

Revenue, users, inventory, or content: your database is the definitive source of your core business operating data. And you'll definitely derive many of your top-line metrics from your production database. But regardless of whether your database is SQL, no-SQL, or whatever the latest trends happen to be in database design, you'll find that the format in which your application reads and writes data for operating your product will be substantially different from the format required for reporting and analytics.

I won't even scratch the surface of data warehousing in this post. But I will say this: if you find yourself running scripts with hundreds of lines of code to build analytics for your product, you may want to step back. At best your script will be extremely fragile, and more likely it will be buggy and/or wrong. Break the problem into smaller steps, build intermediate database tables corresponding to sub-metrics that go into solving the problem in an effort to reducing complexity.

Key Performance Indicators

Taking a step back for a minute, data is a measurement of reality. The ultimate goal of reality is to measure each one of your users, along with their happiness and success brought by using your product. Intuit has a program called "follow me home" in which employees would actually go to homes of real users and watch them use their software in a realistic context.

While programs like this as well as more general user testing studies are effective user research tools, they're not effective tools for measurement at scale, much less any sort of web scale with hundreds of thousands or millions (or more) of users.

Let's consider perhaps the most basic measure of your product: unique users. If you're unable to attract many users to your product, then you're most likely not meeting your product goals. Yet this relationship isn't bi-directional. It's very possible to have a successful press effort, social media flurries, or other traffic spikes that lead to tons of visitors, but none of which really engage with your offering.

The goal of an effective key performance indicator, or KPI, is to find a metric (or two, or maybe three) that bidirectionally correlate to the success of your product.

Effective Dashboarding

Let's dive right into an example of a real-world dashboard with real-world KPIs.

Etsy's Facebook Gift Recommender made shopping for your friends a breeze!

While at Etsy, Leland and I worked on a product called the Facebook Gift Recommender. Conceptually, the product was simple: connect with Facebook, and we'd suggest presents for each of your friends based on their likes on Facebook.

The goals for the product were to (1) drive revenue and sales to Etsy, (2) generate incremental visits to the Etsy from external referrers, and (3) create a super-simple, one-stop place for people to shop for their entire family.

These product goals were reflected in the KPIs of the dashboard below:

The three top-level KPIS for the Facebook Gift Recommender: 30-day revenue, 30-day non-internal visits, and # of friends shopped for per connection. — The three top-level KPIS for the Facebook Gift Recommender: 30-day revenue, 30-day non-internal visits, and # of friends shopped for per connection.

One thing you may also notice about the above dashboard is the hierarchy of information. The KPIs are clearly shown at the top of the dashboard, and information flows to a more granular level as you go down.

In an effective dashboard, less is more. You're trying to tell a simple story that everyone on your team can understand. This is in contrast to a deep data dive that's exploratory in nature and seeks to answer something anomalous or unearth possible opportunities that aren't yet understand.

Defining your KPIs are only as good as your ability to socialize them within your team and company. Making your dashboards easy to access and highly visible. Dashboards and goals are of little use if they're hidden. Once you're at a point where everyone has completely internalized your KPIs, discussions on how to improve your product will become much more focused—whether that be the checkout conversion funnel or your search ranking algorithm—whatever the data says.

Hang your dashboard on a TV on the wall. Put your dashboard up on the homepage of your internal admin site. Send out daily or weekly emails with screenshots of the latest dashboard numbers.

At my current startup an IRC bot named "metric man" shows up every day at 10am, announces himself on Slack, and beckons everyone to check out our latest numbers.

Whatever works!

The big picture

Measuring your application is an exercise in good engineering. Yet defining the right metrics and KPIs is more of an art than anything else. Think creatively and pick measures that reflect your beliefs, support your product, and tell things as they really are.