In
the marketing
field, we often hear the term “AB testing.” AB testing is a method of comparing and verifying two things, and is one of the methods often used in Internet marketing.
In this article, we will explain the basic knowledge of AB testing, the reasons for doing it, and the risks in an easy-to-understand manner.
What is AB testing?
AB testing is a test that compares multiple things. It may be easier to understand if you think that it is called an AB test because it compares “Which is better between pattern A and pattern B?”
This is a method often used in Internet marketing, in which data such as the number of accesses and closing rates of sites and advertisements are compared, and the one that is producing results is adopted.
In order to collect more accurate data, it is desirable to keep conditions other than the comparison target as similar as possible, such as the period and frequency of ad delivery. In addition, we will compare designs and wording that are not significantly different.
AB testing may be conducted to improve the results of
your email newsletter
.
Why do AB testing?
Reasons for conducting AB testing include:
Let’s take a look at each reason.
Improve the accuracy of your homepage and increase your profits
The first homepage, advertisement, and landing page (LP) you create can be said to be a collection of hypotheses. You won’t know the results until you actually try it out. By renewing the main visual, improving the content of headings and text, and changing the posted images, results may be better than now.
If you conduct AB testing and improve each of these elements one by one, the accuracy of your homepage will improve, inquiries and orders will increase, and your revenue may increase.
Possibility to improve results without incurring costs
Just because your homepage or LP isn’t producing results, it’s costly and time-consuming to rebuild the entire thing. Moreover, just because you renew doesn’t necessarily mean you will get results.
As mentioned above, there are cases where even just a few changes to the elements on your site can improve results. AB testing can lead to increased profits without the need for costly renewals, and it is also possible to improve the operational efficiency of your homepage or site.
Understand the patterns that will produce results
By conducting AB tests and verifying the results, you will accumulate knowledge. You’ll come to understand things like, “This design is good for the main visual,” “If I use this content to appeal, I’ll increase my contract rate,” and “If I use this banner, I can attract more traffic through advertising.”
This kind of know-how is a company asset and can be greatly applied to future marketing measures.
Benefits of AB testing
The benefits of AB testing include:
We will explain the benefits of each.
Multiple patterns can be verified simultaneously
Although it is called an AB test, it does not necessarily compare and verify two things. Sometimes three or more patterns are verified simultaneously.
By comparing and verifying them all at once, you can find more effective measures in a short time. Also, depending on the pattern settings, it is possible to analyze which part of which pattern should be improved to make it even better.
Can be implemented at low cost and in a short period of time
As mentioned above, typical AB testing does not compare and verify items with significantly different designs and wording.
Therefore, even if the verification results show that improvements are needed, all you need to do is make a few changes to the original version. Another advantage of AB testing is that it can be effectively improved at low cost and in a short period of time.
Points to note when conducting AB testing
AB testing has many benefits, but before implementing it, you need to pay attention to the following points.
We will explain the precautions for each.
Testing is performed at the same time and at the same time, focusing on one element.
In AB testing, it is important to verify multiple patterns at the same time. If the timing of the test is off, the inflow route and motivation of users will change, making it difficult to obtain highly valid verification results.
It is also advisable to limit the comparison to only one element. If you compare two or more factors at the same time, you won’t know which factor has an effect on the result.
Ensure a certain number of samples
In order to obtain reliable results from AB testing, the number of users and
(CV) must be large to a certain extent. For example, in the case of website verification, a CV count of 2,000 or more is required to obtain highly valid verification results.
However, please note that the appropriate number of samples will vary depending on the original number of views. The number of samples required also differs when the number of users is limited, such as when verifying an email newsletter.
The test will take place over 2 weeks.
In AB testing for the purpose of improving web pages, it is ideal to test continuously for two weeks or more.
However, for websites with a small number of views and CVs, it may not be possible to obtain a sufficient number of samples within about two weeks. On the other hand, for websites with a high number of accesses, it may be possible to obtain more than enough samples in a short period of less than two weeks.
Also, the appropriate test period for verifying the effectiveness of email newsletter distribution varies. Decide on an appropriate test period depending on the number of samples and users required.
Understand that this does not necessarily mean good results.
In AB testing, new things do not necessarily yield better results. There are often cases where the conversion rate has dropped because of changes to the elements of the homepage, or where the number of accesses has decreased because of new advertisements. Please keep in mind that there is a risk that the results of AB testing may decrease.
However, the same can be said for cases such as completely renewing a homepage or LP, or distributing completely new advertisements. It’s less risky and less costly, so let’s start with AB testing and make small improvements.
Please note that the results of AB tests are easily affected by changes, and may not always yield the expected results. You may get undesirable results, results that deviate from your purpose, or results that may not be helpful at first glance.
These results may not be useful for solving current problems. However, it is true that by conducting AB tests, you can get closer to the optimal solution. Let’s appropriately accumulate results that are not helpful and continue to conduct AB tests.
4 types of AB tests
AB tests used to improve web pages can be broadly divided into the following four types.
1. Test to switch elements/appearance of the same URL
2. Multi-page test
3. Redirect test
4. Multivariate testing
It is important to choose the appropriate test depending on the content you want to analyze.
AB testing tools include Google Optimize (ended September 30, 2023). Efficient improvements can be made by using heat map tools such as Microsoft’s “Clarity” in combination.
For verification of email newsletter distribution, we use a different type of AB test.
From here, we will explain the characteristics and flow of each type.
1. Test to switch elements/appearance of the same URL
There is a way to change only specific elements or appearance without changing the URL. This is a common method of AB testing used to verify web pages, and the advantage is that it is easy to prepare for the test because it does not require rewriting the source code.
JavaScript technology is used to allow different users to access different web pages even with the same URL. If you use
JavaScript
to register each pattern you want to compare as a script, you will be able to access web pages with different elements and appearance depending on the user’s device and browser type.
2.Multi-page test
Multi-page testing is an AB test that is used when the elements you want to compare and consider span multiple web pages. In multi-page testing, we compare not only the patterns of each page, but also
the link
destinations.
Since you can also compare the conductors within a page, it is also possible to find an effective structure that leads to CV. In addition, by controlling so that only specific conductors can be moved, comparisons can be made more simply, and the verification results will be easier to understand.
3. Redirect test
A redirect test is an AB test in which users who visit the web page under test are redirected to another URL. For example, it may be used in web pages where users do not enter halfway, such as an application flow.
However, please note that it is time-consuming as you will need to prepare a separate page to redirect to.
4. Multivariate testing
A multivariate test is a test that switches and combines multiple different elements on the same page and compares many variations. By analyzing the results of each pattern, you will be able to understand which patterns are more effective and which elements have the greatest effect.
Although it is more complicated than a general AB test, it is a convenient method because it can be verified in one go when there are multiple elements that you want to compare and consider.
4 basic steps for AB testing
AB testing basically proceeds as follows.
1. Clarify the purpose of AB testing
2. Create a hypothesis based on actual data
3. Prepare the data measurement environment and conduct the test
4. Analyze the results, implement improvement measures, and run the PDCA cycle
Let’s look at it step by step.
1. Clarify the purpose of AB testing
First, clarify the purpose of AB testing. When you want to verify multiple issues or changes, make a plan to conduct AB tests starting from the areas that have the greatest impact on improvement.
2. Create a hypothesis based on actual data
Create a hypothesis for each test. For example, write down a hypothesis based on the improvement, such as “Click-through rates will increase if you change the button to red.”
When formulating your hypothesis, use actual data such as banner click rates and page exit rates. Utilizing data increases the likelihood that a hypothesis is judged to be valid and allows for more accurate verification.
3. Prepare the data measurement environment and conduct the test
In order to accurately measure the indicators to be verified, we will conduct AB tests after preparing the data measurement environment.
You can build tests on your own using JavaScript, cookies, etc., but it is not only time-consuming, but it also becomes difficult to handle when there are a large number of samples, so it is better to use AB testing tools, etc., which will be described later, to perform them efficiently.
4.Analyze the results, implement improvement measures, and run the PDCA cycle
Analyze and verify the results by comparing them with the hypotheses. Let’s come up with specific improvement measures and implement them. It is also important to continue the PCDA cycle of improvement and verification, rather than just making improvements once.
Scenarios and methods in which AB testing is performed
AB testing is mainly performed in the following situations. Let’s introduce each method.
3 recommended AB testing tools
AB testing can be carried out efficiently by using tools. Introducing recommended AB testing tools.
1.SiTest
SiTest is a tool that can be used not only for AB testing, but also for
heat map
analysis and input form optimization. Let’s use it when you want to analyze and improve your site using various methods. Although it is a paid tool, there is a free trial, so you can check how it feels before you use it.
SiTest
2.Juicer
Juicer is a tool that allows you to conduct AB tests intuitively using a visual editor. The basic plan is available for free. However, it does not support multivariate testing, so it is not suitable when there are multiple comparison elements.
3.Optimizely
Optimizely is a tool that allows you to perform AB testing across multiple pages. There are many major languages that can be used for test design, such as JavaScript and
CSS
, and they are easy to use. Try using it when you want to measure results from multiple perspectives.
Use AB testing to conduct effective analysis
AB testing is a method often used in Internet marketing that compares multiple different methods and adopts the one that produces results. By conducting AB tests on homepages, advertisements, e-mail newsletters, etc., you can select content and methods that will yield better results.
AB testing has the potential to increase profits and increase knowledge. However, be aware that there is a risk that the results may temporarily decrease when conducting AB tests.