Is it OK to run CRO A/B testing for a longer period of time than you anticipated? We use our expertise to help answer your question.
While you should always pre-plan your A/B tests to help determine its statistical viability, sometimes you need to run the experiment longer than you anticipated. And this is OK! If you can’t glean insights from the test you’re running, you should feel comfortable extending the testing period until there is something you can learn and take action.
Here are a few reasons you may find yourself running a test longer than initially planned:
- Your test isn’t statistically significant. In short, “statistical significance” means that your test results most likely occurred from a factor other than chance. If your A/B test doesn’t have a large and/or diverse enough sample size of data to provide actionable insights, then it’s likely not yet statistically significant. If this is the case, then you can’t yet fully determine if the results are telling your future or actually giving you a data-driven blueprint for website optimizations. Our advice here: Just be patient and wait for the test to become statistically significant AND have a large/diverse enough data set to make sound, data-driven decisions.
- Your test wasn’t set up correctly to begin with. Sometimes, your test just isn’t set up right, so it won’t pull in the data you need…or it won’t pull in any data at all. To combat this, make sure you always conduct a “setup check” of your test after 24 hours of launching it to make sure you set it up correctly. If no data is being recorded after this 24-hour period, end the test, figure out where you went wrong in setup, and try again.
- Your test was only planned for a short period of time. A/B tests should run for at least two weeks before you have collected a large (and diverse) enough sample size to provide truly actionable results. If you’ve planned an experiment to only run for five business days, chances are you won’t have the data you need to take action. Run your test for at least two full weeks (including weekends!) so you can help make sure you’re not wasting your time on an experiment that won’t give you the results you — or your boss — are looking for.
