Measure What Matters in Experience Design

“Do not solve the problem that’s asked of you. It’s almost always the wrong problem.”

– Don Norman

Designers have often been told to focus on outcomes, not features, so that they solve the right problem instead of building the wrong thing. While this rule has been accepted and practiced within experience design, it is too often forgotten when it comes to digital analytics. Just as building a feature to address the wrong problem will surely fail, tracking the wrong metric will prove meaningless.

Focus on UX goals to drive analytics measurement plans, rather than tracking superficial metrics. Here are a few tips on how to identify the core goal of a design to meaningfully measure it:

Always Ask “Why?”

When determining what to measure on an ongoing basis, it is imperative to fully understand the ultimate goal of the design to be sure you choose an appropriate metric. If you don’t clearly define the goal, it is impossible to determine the best method to track performance or user experience.

What do we really want to know? What action could we take if we knew that number or rate? There are countless things to track, and, without honing in on the true goal of measurement, no metric is likely to provide a meaningful or actionable insight.

Define Goals, Then Decide on Metrics

Analytics data by itself is not noteworthy, but rather is the means of getting insight into the current experience. While determining what to measure for a task or feature, don’t get bogged down by the knowledge of which metrics exist in your analytics tool or what you think is possible to measure. Think top-down, from goals to metrics, instead of bottom-up, from metrics to insights.  A good way to approach this process is to go sequentially through the following steps:

  1. Ask: what is the goal of the feature or design element in question? What do you want to ensure about the user experience of that feature or functionality? Only once this big-picture goal is defined should you look for concrete metrics that relate to that goal.
  2. Once this true goal has been identified, determine the behaviors that act as a signal for that goal. For example, whether people engage with the content, or whether they reached an appropriate page and if they match your target audience. These signals need not specify a method of measurement but should instead be behavioral cues that would tell an observer whether the goal has been reached.
  3. Depending on the number of signals you identify, you may narrow the list to only the strongest signal(s) to simplify your measurement plan and avoid splitting your focus between too many alternate ways of identifying the same behavior.
  4. Define ways of measuring the signals. For example:
  • use a scroll heatmap to gauge how far down the page most people scroll and presumably read
  • use event tracking to count interactions with in-page widgets or photo elements
  • look at the referral sources or search keywords that brought people to that landing page — are those appropriate given the content and purpose of that page?
  • track the rate of return visits, even specifically returns from those who have bounced, or look at the overall frequency and recency of your audience to assess loyalty

These more specific metrics will get you much closer to evaluating the true performance of the UI than a surface-level metric chosen without fully reflecting on the deeper goal.

Track the User Experience

Don’t lose sight of the target of many redesigns and new features: to make a task easier or help save people’s time. These experience goals should remain at the forefront when choosing how to measure the success of a design.

When calculating returns to report on the usability ROI of a new design, these performance-focused metrics will show insight into how the design benefits your customer. Emphasis on these measures will, in turn, create more loyalty and drive more business.

Focusing on the high-level, experience goals rather than jumping directly to what may be simple to track is critical to meaningful measurement. If these deeper goals are not considered during metric selection and data reporting, significant statistics will be overlooked. Good design is not about getting more clicks or more page views, so why would we measure success that way?

Harley, Aurora. Translating UX Goals into Analytics Measurement Plans. Nielsen Norman Group. May 14, 2017. https://www.nngroup.com/articles/ux-goals-analytics/