Learn about the promises and pitfalls of A/B testing and how our Lead Web Engineer created a new Craft CMS plugin to harness its power.
3minutes remaining
A/B testing isn't a new concept, but it's not used as often as it should be. This article will explore what A/B testing is, when it is best employed, what tools work for different scenarios, how to interpret the results, and finally, will introduce our solution to facilitate simple A/B testing in Craft CMS (Craft).
About A/B testing
A/B testing, also known as split or multivariate testing, is a way of comparing customer interaction with two or more different digital experiences to optimize a website or app. Some examples include testing different messaging, different images, and even different funnels to lead a customer to the checkout.
Direct marketers pioneered A/B testing by creating two different mailings and seeing which one had a better response. Online, we can do that even more easily today by diverting some web visitors to see one experience and some to see another.
When creating an A/B test, a marketer should answer the following questions:
What are the goals of the test? For example, determining which messaging leads more users to request a demo.
What is going to be tested? The more different variations tested simultaneously, the longer it will take to gather data.
How long do I want the test to run, or how much data do I want to collect?
How will I interpret that data at the end? Do I want to simply pick the option that drives better results, or use statistical analysis to make sure my results are statistically significant?
What A/B testing tool should I use?
A/B testing tools come in three levels of complexity:
Homegrown – You can ask your web developer to set up two different versions of a page and divert half your users to one version and half to the other. Using Google Tag Manager or some other tracking, you can gather results. This is ideal if you only want to use A/B testing occasionally and are comfortable doing your own statistical analysis or using Google Tag Manager’s options.
Simple tools – As discussed further below, simple tools like Imarc's new A/B testing plugin allow you to change elements on a web page but require more work to create entirely different web experiences or funnels. This is ideal if you want to do simple A/B testing regularly, but don’t have a big budget. With tools like this, you must also do your own statistical analysis or use what Google Tag Manager supplies.
Enterprise options – Full-featured tools that allow you to set up complex tests and target based on results, like VWO or Optimizely. This is ideal for companies with large marketing organizations (and budgets!) that want to continually run complex A/B tests. Complex analysis with user breakdowns and statistical analysis are also included.
A/B testing best practices
The most important thing is to test, test, test, test! In our experience, marketers do not use testing enough, or systematically enough, or follow through on the results often enough. Testing can be done occasionally, but it’s better to identify key pathways in your site, and test and tune them frequently to optimize the flow.
It’s also important to monitor tests in progress. Most often, tests don’t reveal much of a difference between two options, but if one option quickly performs better than another, stop the test early and use that option.
Once you make a tweak based on test results, test again – markets change, users change, and content gets stale.
“We tend to test it once and then we believe it. But even with a statistically significant result, there’s a quite large probability of false positive error. Unless you retest once in a while, you don’t rule out the possibility of being wrong.” – Kaiser Fung, Founder of the applied analytics program at Columbia University & author of Junk Charts, as told to the Harvard Business Review
The smaller the effect of a test, the more likely the result is wrong or not significant. Retesting prevents wrong results from persisting, so it's important to include frequent tests in your overall strategy.
A desire to optimize
Imarc wanted to test our Imarc.com call-out messaging to optimize for conversion. To implement this, we experimented with a third-party Craft plugin we hadn’t used before for A/B testing. Unfortunately, this plugin changed the content of the website on the server, rather than on the browser, so any kind of caching like Blitz or CloudFront made it fail to deliver alternate versions. Code also had to be added to any part of a page we wanted to test, and then deployed, so it required developer intervention for any new test.
Instead, Imarc built our own A/B testing plugin. Tests can be created and administered in the Craft back-end, but the website is changed through JavaScript. Using a standard CSS selector, the test targets an HTML element on a page or pages and replaces the innerHTML for that element.
This implementation gives tests a great deal of flexibility. The easiest kind of test to create through our tool is changing a piece of text or an image, but it can also be used to change a URL on a link and send users down different funnels, though this also requires a more robust Google Tag Manager set-up.
Once an end-user encounters a test, the version they see is recorded in a cookie, so each user will always encounter the same experience, but different users will see different versions. We also provide ways to test the tests by previewing each option, to make sure they are working.
Tracking is done through Google Tag Manager custom dimensions, which track the handles the CMS editor assigns to the tests.
Experimenting on ourselves
Imarc created an A/B test for the main call-to-action (CTA) on the Imarc.com homepage. One version said "See our work," directing users to our portfolio and another said "Say hello," sending them to the contact page.
The test went live on March 12 and concluded on April 8, 2024, with the launch of our rebranded website.
Unsurprisingly, "See our work" outperformed "Say hello" by a factor of four because people visiting our homepage typically want to learn more about us before engaging in a conversation. We will run additional tests on our rebranded website shortly, so stay tuned for more results!
Now that we have tested this tool, we are excited to roll it out to our client partners on the Craft platform who also want to test their messaging. Further, we are more ready than ever to help our non-Craft partners choose and implement a solid A/B testing solution that meets their needs.
–
At Imarc, we love building and optimizing websites. If you do too, let's connect.
