We’ve all heard the phrase “learn from your mistakes”. In Conversation Rate Optimisation (CRO) it’s no different. Mistakes can and will happen, but the important thing is to learn and not repeat them.

CRO is a complex process and all the information can be overwhelming. Creating hypotheses, setting tracking, analysing data, etc. can lead to mistakes.

In this fourth of our five blog series, we’ve focused on five common CRO mistakes and how you can avoid them.

1. Not following a CRO process

It is very important for CRO to have a process and to follow it. The CRO process is like the scientific method, if you skip a step, you might end up with unwanted or unsatisfactory results. As we explained in a previous blog, the steps of the CRO process with Cohaesus are:

  1. Exploratory phase
  2. Data research and analysis
  3. Create hypotheses and prioritisation
  4. Design and development
  5. Set up the test
  6. Monitor results and learnings

Your CRO effort should follow similar steps and it is important to consider the statistical significance of your experiments as well. Directly jumping into testing without analysing the website’s performance, or backing up tests with data-based hypotheses can lead to errors. By following a process you avoid making assumptions and mistakes.

2. Stopping a test too soon

As we mentioned before in our previous CRO blogs, we recommend running a CRO test for a minimum of two weeks. The duration of the test will depend on your sample size – the bigger your sample size, the more accurate your conversion values will be. Your sample size is directly related to the volume of traffic to your website.

The goal of implementing a CRO process is to understand your audience and gather enough information to improve user experience. Stopping a test too soon might result in misinterpreting data points or inconclusive results.

3. Given up on a “loser” experiment

You can say that a “loser” experiment means that the variant you created had a negative impact on conversions or, in other words, did not perform better than the control.
However, the concept of ‘loser’ does not exist in CRO; all experiments are winners in a way. You can learn invaluable insights from every test you do on your website. In each test, you get closer and closer to a better understanding of your audience.

Don’t give up on your test! If you have a variant with a negative uplift you can take the opportunity to answer some questions like:

  • Why did the control perform better than the variant?
  • What can the results teach me about my audience?
  • Are there any positive results based on the conversions tracked for this experiment?
  • What hypothesis can I create based on these results?

4. Running too many tests at the same time

Running tests at the same time, even if they’re on different pages, can lead to cross-pollination. The concept of cross-pollination of users is similar to its equivalent in the natural world. The user comes to a page on your website, sees a test A then navigates to another page and sees a test B and so on, until finally converting. In this case, you have to consider two questions:

  1. Which test was responsible for the conversion? (Attribution)
  2. Did one test affect the results of the other? (Interactions)

It is hard, if not impossible, to tell which test was responsible for the conversion if they have the same goals. If they have different goals, you might say one helped the other but you cannot say that test A was responsible for the conversion. Let’s say for example that you have a test running on the homepage and the goal of this test is to direct visitors to the product page and the goal for the test running on the product page is to complete a purchase. In this case, you could say that the visitor completed a purchase due to the test on the product page and the visitor went to the product page due to the test on the homepage. Can you really be sure of that?

That’s when the second question comes in. What if the test on the homepage was enough to convince the visitor to purchase? In that case, when seeing the product page it doesn’t matter if the visitor saw the control or the variant, as he/she was already convinced to purchase. Unfortunately, there is no way to know that. What can you do now?

You have three options:

1. Assume that the tests will not affect each other

You cannot say for sure that they will not affect each other but neither can you say for sure that they will. You can assume that two tests running at the same time, on two different pages, with two different goals will not affect each other but you might be wrong.

2. Mutually exclusive tests

This option is available in many CRO tools. You can set tests to be mutually exclusive, which means that the visitors will only be part of one test.

3. Multivariate tests

In a multivariate test (MVT), you can test different versions of your variant at the same time. It makes sense to run an MVT when you have tests with the same goal or flow. For example, if you have a green call-to-action on the homepage with the copy ‘Donate’, you can have a variant in red with the same copy, a variant in green with the copy ‘Donate now’ and a third variant in yellow with the copy ‘Take action’.

It is important to bear in mind that multivariate tests take longer to produce results and it might not be recommended if a page or website has low traffic. In an A/B test, the traffic is split in half, with 50% of the visitors seeing the control and 50% seeing the variant. In a multivariate test, this percentage decreases according to the number of variants.

5. Only optimising underperforming pages

It is a mistake to assume that CRO should only be done in underperforming pages. The theory that you have to only fix what is broken does not apply in CRO. Everything can be improved, even pages that are doing well. Something good can be great if you invest the time to optimise. In fact, it is much easier to improve something that is already performing well.

There is no magic formula in CRO. However, if you follow a process, learn from your mistakes and make it better next time you are going in the right direction. Learning is the word that defines CRO. Hopefully, we’ve given you an insight into a few mistakes that can be made and helped you to avoid them.

In our fifth (and last) blog, we are going to give you 7 simple A/B test ideas to get you started with optimisation.