A/B testing mistakes that can cost you big
A/B testing is an essential tactic for improving the optimisation of your PPC advertising. A/B testing or split testing is the process of running tests to compare performance of different versions of a web page in an effort to find the best version for maximum conversion optimisation. While A/B testing commonly refers to landing pages, it is also frequently used with ad copy or display ad creatives but can apply to various elements of your digital marketing.
Those experienced in PPC advertising will know that A/B testing for PPC ads can improve CTR, quality score and provides the advertiser with beneficial data for understanding what messages resonate with their audience. When carried out effectively on ad copy and landing pages, it can increase conversion rates and improve your profits. While A/B testing is extremely valuable, the process is not as easy as it may appear. A/B testing does not automatically equal better results and when carried out poorly it can waste time or actually hurt your campaign.
There are a plethora of damaging A/B testing problems advertisers need to avoid but the following 18 mistakes are some of the most common.
1) Failing to test all relevant elements of your campaign
In PPC advertising the most common features to test are within the ad copy. It is easy to produce multiple text ad variants for the search network quickly. Advertisers frequently focus on the headline, body text and display link testing. On the display network, the A/B testing tends to focus on ad creatives, copy and CTA. While ad copy tests are an obvious choice and can influence CTR, A/B testing can go much further and help you to refine almost every aspect of your campaign into a highly optimised and perfected strategy. Landing page tests are vital for improving conversion rate optimisation and can go far beyond the usual suspects of titles, visuals, and content.
Advertisers should also look to test elements like:
- Your landing page call to action
- How your call to action is displayed e.g. style and position of button
- The position of your conversion form
- The length of your form and the questions it asks users
- The style and number of sections in your checkout system
- The style of your checkout system for mobile
- The mobile experience layout of your landing page
- Full navigation vs partial vs no navigation on conversion landing pages
- The funnel flow e.g. the number of steps required by users in the funnel journey (soft sell vs direct sell)
Take a look at the length of this form for a white paper from Adobe:
That is a lot of questions for a white paper and a good opportunity for Adobe to try some A/B testing with shorter forms to improve their user experience and conversion rates. You could also A/B test your strategic approaches for different networks to see which is best for your company’s conversions, such as dynamic search ads vs standard text ads or rich media ads vs standard display ads. A/B testing such as this goes much further than advertisers might think.
Even the optimisation of keywords is a kind of spilt testing and an essential one. By Optimising and pruning your keyword list to find the most effective variations, you are also performing multi variant testing.A/B testing can also be used on your ad extensions to see which are most effective for improving ads, such as A/B testing your callout extensions. You could even A/B test different product feeds if you have the time, along with different bidding strategy approaches, various remarketing lists, and various audiences.
With elegant audience targeting available in the forms of demographic, geographic and behaviour targeting, advertisers can experiment with different ads and landing pages for different audience types to really perfect the ideal messages for each group or to choose to target the most responsive audience only. As you can see A/B testing has the potential to enhance your PPC advertising on every level. Split testing the ad copy and landing pages alone is only scratching the surface.
When you start to spilt test these elements within different audience groups your data can become highly detailed. The best advertisers know to move beyond basic ad copy A/B tests and look to all the PPC advertising factors that could be improved by testing their validity.
2) Testing multiple elements at once
This is one of the major stumbling blocks in A/B testing. It is tempting to advertisers because it seems to save time but by changing multiple elements in each variant, it is much harder to analyse which is the feature that is making the difference.
Take the example of a simple text ad. It is a good idea to test multiple versions but if each version has a completely different headline, description, and display link it would be harder to compare. It is the same problem as the metaphor of comparing apples to oranges.
Advertisers can use multiple variations of the same element in their testing but should try not to change too many different features at once e.g. only change a visual or only the headline to be able to make the most accurate comparisons.
Unfortunately, there is a trade-off between accuracy and efficiency. By only testing one feature at a time and trying to test every element of your campaign for greatest optimisation, you will greatly extend your testing time. However, while trying to strike a balance, advertisers should avoid testing too many elements at once for the sake of cleaner data.
3) Only focusing on minute changes
While it is good to focus on relevant comparisons in you’re A/B test, it is useful to not become obsessed with only minute changes.
Small changes can make a big difference to conversions and sometimes they are just really pointless.
In the past, some PPC marketers became fixated on the colour button test. It is a popular trope of A/B testing and has been greatly overused in recent years. While very small changes, such as a simple colour change can be influential, it is easy to get bogged down in testing every tiny, minute detail of your landing page and ad creatives. Often more drastic changes will prove necessary to make an impact. For efficiency, you will need to prioritize the most important elements to A/B test.
Often more substantial variations are required to avoid wasting time and to make any significant impact on your data and strategy. When choosing what to A/B test always make sure it has some real value and is in line with your objectives.
The following example of an A/B landing page test from Neil Patel shows how taking a chance and making a big change can be beneficial.https://72gpf1za5iq428ekh3r7qjc1.wpengine.netdna-cdn.com/wp-content/uploads/2016/05/image26-9.png While technically, multiple elements were changed, in this case, it was worth it and they did focus primarily on visual changes. Advertisers need to be willing to make significant variations at times and not get obsessed with minute tests for no reason.
4) Only focusing on radically different changes
Just as only implementing tiny changes can be a mistake, so can only implementing large changes. Sometimes large differences between ad or landing page variations are required but often it will be easier to compare smaller changes.
It has become an exaggerated motif that minor changes can make a major difference but we must acknowledge that sometimes they can.
Not only can a small change for an A/B test help improve CTR and/or CVR, it is also fairly easy to implement and at the very least could elevate ad fatigue.
According to research conducted for the Science of Lead Generation and reported on by Hubspot, landing pages with submit buttons, which actually contained the wording “Submit” tended to have lower conversion rates than those that used other wording. It was theorized that the reason was to do with the emotionally loaded commitment of the word submit. In this case, a small tweak made a big difference, as you can see from the chart below: https://blog.hubspot.com/hs-fs/hub/53/file-23152353-jpg/blog/images/submit-resized-600.jpg?t=1499098697457&width=531&name=submit-resized-600.jpg
Honestly some marketers have become sick of the colour button test mantra and who could blame them but some people really do just hate orange, so wouldn’t you rather eliminate something so simple and inconsequential if it could mean more profit?
Below is a perfect example of the button colour A/B testing approach.
optimisely-AB-example-AB-tehttps://instapage.com/wp-content/uploads/2016/08/optimisely-AB-example-AB-testing-mistakes.pngsting-mistakes.png
Although more significant variations will often be required, advertisers should not forget the value of small tweaks in optimisation.
5) Non-goal led testing AKA testing for the sake of it
Our two previous mistakes may seem like contrary advice but they are both symptoms of a greater error: testing for the sake of it.
Whether it is only running minute colour change tests or major variation differences, both can be wrong. Equally, both can be performed well but only when the A/B tests are done to meet the objectives of your campaign.
Testing for the sake of it, because you have been told to by PPC blogs, is not a tactic. All A/B testing is done to improve optimisation for your business goals and gather relevant data.
Advertisers should never run a test just for the sake of it. The solution is to base test proposals off existing data. Consider where you would like to make more conversions. Where in the process are users abandoning you? Try to address if there is a specific problem at a point in your conversion funnel with you’re A/B tests.
A/B tests are a good way to optimise by removing friction points in the conversion process. It is unlikely, in most cases, that a simple colour change is going to be as useful as testing areas that remove friction points, like your form questions. In highly visual businesses colour can make a huge difference and if you have optimised for everything else, a colour tweak test could hold value. Always consider which test will hold most value for your goals.
6) Not following data
A/B testing requires a premise and a decision on your different ad or strategy or landing page variations. The problem is these will be based on some subjective, personal belief on what will resonate with users. It is therefore natural that advertisers will already have an underlying belief about which variation should work.
This can lead to confirmation bias. Advertisers may misread, misinterpret or actively ignore the results of their A/B test in favour of their own opinion. It is essential that if you wish to run effective A/B testing you follow the evidence of the data not so called ‘gut instinct’.
Confirmation bias is not limited to your own instincts but also beliefs about what will work based on PPC industry insights. You should never let the advice of PPC experts influence how you interpret data. You may find that your PPC advertising A/B testing reveals some surprising results. WordPress published a compilation of interesting results advertisers found through A/B tests and some were not as they expected, such as more questions in forms actually working to improve conversion rates. This is why it is always a good idea to test your theories.
7) Using before and after tests instead of simultaneous A/B tests
It is vital to always run your A and B variations simultaneously to ensure fair results. If you choose to run your ads sequentially, rather than simultaneously, external factors can compromise the test results.
As you can see from the following chart, traffic quality varies from day to day on every website, whether it is organic or paid traffic:
wp-content/themes/quicksprout/oc-guide/images/08-06.png
Even when you have fairly stable conversion rates it won’t be exactly the same amount every day and the same goes for CTR. Many different factors can affect the number of visitors to your website, hence to ensure a truly fair test the variations must run together.
8) Not allowing long enough for tests
Judging results too early is probably the most common of all A/B testing mistakes, which can be costly for your campaign. You need to look at a wide enough window of time to eliminate other factors on the test e.g. day to day traffic variation, seasonal changes, local and national events. There could be a vast number of reasons that variation B won, so you need to be sure it genuinely was more popular over time. A bigger window gives more accurate data and levels out exterior factors.
The following graphs from Conversionxl show how intense the effect of overly short tests can be. The first is the results of an A/B test reviewed after only two days:
conversionxl.com/wp-content/uploads/2015/09/11-568×199.jpg
This next graph shows the results after 10 days and as you can see the results are completely different:
conversionxl.com/wp-content/uploads/2015/09/2-568×178.jpg
The larger your sample or the more samples you have, the more your results stabilise. PPC Hero’s own test confirms this.
The graph below shows a multi variant test over time and as time progresses the results stabilise revealing the true benefit of the variations in action: https://www.ppchero.com/wp-content/uploads/2013/03/CRO-Test.png
9) Over testing
There are so many elements you need to test and it is important to allow enough time for accurate test results but on the other hand, over testing is just not practical for the average business.
At the end of the day we need to remember the average PPC advertiser is not working for scientists, you are working for businesses. This means that profit is more important than being irrefutably sure about which variation of ad or landing page is best. As long as conversions and profits are improving for you that is what really matters.
If you really want honest data where you know element x, y and z were best and for exactly what reasons, then you are going to spend all you time testing instead of implementing. The best thing to do is to prioritise your most important elements to test and set clear time windows for reviewing and stick to them.
10) Not having a testing plan
To solve the problems of over testing and cutting tests too short you can develop an A/B test plan. This will make it easier for you to know when to carry out a test, which elements are most important to your campaign and should therefore come first and your test time windows.
By laying out a clear plan at the start of your campaign it will prevent you from becoming overwhelmed with all the tests and help you keep track of progress. This is just like having a plan for the different stages of your campaign, such as introducing additional networks, areas, keywords, remarketing and expanding your audience. You could plan a test at the start of each of these expansions to ensure you always use tried and tested methods.
While having an A/B testing plan is beneficial, don’t let it make you too rigid. As the campaign runs and data unfolds, you may need to change your plan with what is proven to work for you.
If you are short on resources and time, some of the main element you will want to prioritize in A/B testing are:
- your visuals
- your ad copy
- your call to action
- your landing page visuals and copy
11) Don’t forget to set ads to rotate evenly when A/B testing ad copy
If you are A/B testing elements in your PPC ads then you must not forget to rotate your ads evenly for accurate results. With all the different optimisation options in AdWords, it is easy to forget this very simple step, which could jeopardize the validity of your test.
It is likely you will have set your ad rotation to match your business goals but a fair comparison with a new ad variation must be shown evenly for the same keywords to obtain accurate data. Most optimisation settings favour ads already in your account which have shown to perform well so far.
You have two options when it comes to rotating ads evenly, you can either rotate them evenly indefinitely or rotate them evenly for 90 days before AdWords optimises them again. We recommend using indefinite even rotation for A/B testing, this way you can review the results when you are ready and make your own strategic decisions accordingly.
Just head to your settings tab, scroll down to ad rotation and expend. Then click edit next to your chosen optimised rotation and switch to rotate evenly indefinitely:
12) Over Optimising for micro conversions
Micro conversions are great. They help move users along the conversion funnel and are especially handy for highly priced items and contract based services. They can also help attract users to your mailing list where you can acquire even more data from them. While micro conversions are very useful for tracking user behaviour, they can be a distraction to advertisers carrying out A/B tests.
For some industries, micro conversions might be completely essential to your funnel process, in which case it is well worth A/B testing elements of the micro conversion. However, for many industries the real value is your actual transactional conversion. When you have many elements to test you need to have the right priorities on which are most important and for many that will be a transactional conversion.
Focusing on optimising the micro conversion elements is not just taking up time that could be spent working on your macro conversion, it could actually harm your conversions.
You may have users who wish to convert straight away. If the micro conversions have been highly optimised by A/B testing they can become too central to the user experience on your page and therefore a distraction. If it becomes a necessary hoop to jump through before macro conversion, it becomes a friction point to your profits.
An A/B test on micro conversions will only show you how to make that conversion more successful, it cannot be so easily compared to your transactional conversion.
13) Using faulty testing software:
Reliable A/B testing software is a helpful way to easily implement changes and tests and monitor them, yet just 44% of companies use A/B testing software tools.
If you have yet to find a suitable software but would like to speed up your testing methods take a look at this helpful list from Crazy Egg.
Despite their benefits, you need to be aware that the additional code from A/B testing software could slow down pages on your site. Page speed is a vital part of good user experience and a crucial factor to be considered in CRO. If some pages in the test are slower this will unfairly impact your results.
You can check all is well by running a simple A/A test before beginning your A/B landing page tests. This will show users the same page, so it isn’t useful for your optimisation data but it can reveal any discrepancies caused by software slowing down pages. Conversion rates should be very similar in these tests, so if one of the identical pages is very different it should be cause for investigation.
Don’t forget that even without fancy third party software you can easily test different landing pages organically before you use them in your campaigns with Google Analytics Content Experiments.
Watch the following video guide on how to implement a Content Experiment:
www.youtube.com/watch?v=TGrujIh2H0I
14) Only looking at conversion rates
While improving conversion rates is a main goal of A/B testing it is not the only relevant KPI. Conversion rate increase is only positive if you can maintain reasonable CPA and ROI. If you are making a loss on conversions due to keyword bids being too high it is not as valuable.
In you’re A/B testing for keywords, copy, landing pages and everything else, keep an eye on how these elements work together and on your ROI to ensure these are the best results to gain business profit.
This is particularly relevant if you are A/B testing different product or package prices. If the lower price converts better but you are making a loss on your advertising efforts then this is not a successful test.
15) Having over the top expectations
If you have read a few case studies and advice blogs on A/B testing, you have probably encountered some incredible stories. Tales of tiny button colour tweaks resulting in 105% CVR increase are common but they lead to bad cases of over the top expectations.
While it may seem every blog on A/B testing sings it praises and has experienced miraculous results, the fact is the majority of tests will have resulted in small improvements or no improvement at all.
Big wins make the most compelling case studies and that is why industry leaders use them but in reality improvements are made over the long term little by little and with no small amount of hard work and patience.
Sadly, these big win stories can leave advertisers feeling disillusioned with A/B testing. You should still use A/B testing even when the results are less than you hoped. Over time little improvements could bring improvements to profits and poor results are still very valuable for you to learn from. You can analyse weak results to know what is not working, which is very useful in the quest for what will work.
The problem has been vocalised brilliantly by Optimisely when they wrote an article on why A/B test success stories are bad for you. Remember that big wins often happen when there were many things wrong with a campaign previously, so the only direction to go is up. When your campaign is already good the improvements are smaller.
16) Giving up too fast
Neutral results and lack lustre, minor improvements can be disheartening. If you have put considerable work and planning into your tests only to yield mediocre results it can leave you feeling fed up. It is important not to give up too quickly. When the first couple of tests did not produce what you hoped this is not a reason to pull the plug on all A/B tests.
Disappointing and neutral results are still valuable. If you do not continue you cannot learn from your mistakes. Repeated neutral results might mean it is time to A/B test with some radical new changes. Remember A/B testing requires many tests, sufficiently large windows for appropriate sample sizes and above all patience.
17) Just implementing what has worked for others
One of the biggest mistakes impatient businesses can make is heavily researching A/B testing studies and then skipping the actual tests and just implementing elements that have worked well for other companies.
What works for one business may not work for you, so don’t add extra fields to your enquiry form just because it led to 15% conversion uplift for someone else.
Even within the same industry, what works for some may not work for your customers, therefore advertisers should not cut corners and hope for the best but run their own proper tests and follow real data. Otherwise you could make a costly change that does not work or actually lowers conversions.
18) Thinking of the process as A/B testing rather than A, B, C, D and E testing
A/B testing traditionally means comparing two versions i.e. two pieces of ad copy or two landing pages. When it comes to gathering the best possible data and crafting the best campaign, it should really be thought of as multi variant testing.
Multi variant testing is now often mistakenly used synonymously with A/B testing in digital marketing and it simply means to compare several versions of your ad or page rather than just two.
Comparing two arbitrary versions rarely goes far enough. By comparing multiple versions of your ad or landing page you can gather far more data at once.
Marketers are often torn between the values of A/B vs multi variant testing. By comparing more versions, test can take longer and in the case of major version changes it can become harder to identify which element is making the difference.
However, by making small changes to each variant it is easier to pinpoint exactly which elements are most popular and which combinations of elements make a winning recipe for success. By having more choice, you also have a higher chance of offering a successful ad.
In the table below you can see how A/B testing can be easily transformed into multi variant testing on a display ad. instapage.com/wp-content/uploads/2016/08/optimisely-multivariate-example-AB-testing-mistakes.png
Testing multiple versions together is more efficient. It also casts a wider net and by adding more message variants you can appeal to a bigger audience and have a greater chance of catching the right audience for your product or service.
Of course, multiple variant testing is much easier on small changes like ad headlines than checkout system styles.
Final thoughts
This may seem like an intimidating list but if you can avoid the most common mistakes in A/B testing you can save yourself a great deal of trouble and costly incorrect decisions. This will help A/B testing bring you useful results and a better conversion rate.