Ask a Marketer: Is Test Velocity a Good Metric for Website Experimentation Programs?

We use our expertise to answer your question.

Experiment velocity is the quantity or volume of tests run over a period of time; it’s also a metric that can show the growth (or lack thereof) achieved in an experimentation program. While velocity is a popular metric because it's easy to track and prove, is it a good metric to measure success of an experimentation program? Yes and no.         

Pros of Measuring Velocity in CRO

Using velocity, you can determine how many experiments you can get up and running, measure testing bandwidth against available resources, and look for ways to improve CRO-related processes. 

In short, measuring velocity helps answer these questions:

• How many tests could you run vs the number of tests you’re currently running?
  • Do you have the appropriate resources to run more tests?
• Are you getting the desired ROI from the current volume of tests?
  • Would adjusting the velocity improve your ROI?

Cons of Measuring Velocity in CRO

While measuring test velocity has its benefits, it isn't the metric you should rely on as your primary key performance indicator (KPI) for an experimentation program. Velocity is easy to measure, track, and prove — which is why it’s so tempting to use as your primary KPI, but doing so will lead to poor test quality.

With velocity, there is a mistaken association of quantity of test results equating to quantity of insights. Inconclusive tests without depth of metrics won’t tell you much, and only running tests based on velocity (or, “for the sake of running tests”) may cause problems like: 

• Assuming you’re learning from test results, which is not necessarily the case
• Focusing too much on only doing small things that are easy to test
• Perverse incentivization to engage in unethical testing practices

When to Measure Velocity in CRO

Test velocity can be used as a benchmark, but it shouldn't be your sole metric to determine the success of your website experimentation program. However, it is a metric you should use to help plan for testing bandwidth and determine if you have the resources (and need) available to increase your testing speed.

If you think you need to increase your program’s test velocity to achieve certain results and meet company goals, take a pause and first: 

• Understand that increasing velocity requires more resources (time, people, brain power, etc.)
• Make sure the quality of tests you’re performing (as well as their results) doesn’t decrease by increasing the number of tests you run
• Determine the efficiency of your current programs and processes — do they allow for this velocity increase to happen? Will your efforts be hindered? Or will optimizing these programs and processes actually provide better results than increasing velocity?

When you're trying to determine what metrics to use to measure success, let our eCommerce digital marketing experts help. Contact us today!