Guide For A/B Testing

What is A/B testing?


A/B testing is a method used to create and test variations of a web page to find out which one performs better. Creating variations of your web page is a great way to find out which changes will work better with your target audience.


For an A/B test, the traffic split could be 25% and 75% between variation and original.

A/B tests can be used to run experiments on different parts of the site to determine which changes would work and which won’t. You can also keep creating variation till you notice a maximum gain in results and adapt that variation.

A/B testing results are most conclusive when gradual testing is done with minimal elements.

Why should you A/B test?

A/B testing is the most effective way to test and decide the best variation that gives your website better conversion possibilities. By running A/B test you can pick out the elements that your website visitors are missing out on and identify efficient replacements for it. It will help you get the most out of your marketing efforts. From finding what is best for your landing page to optimizing the website content itself, A/B testing is indispensable. It helps you find what works best with your audience

What are the goals you should A/B test?

Anything from the headline of your content, images in your blogs, call to action buttons, graphics of your website and any other element that are small and can be changed on a web page must be A/B tested.


Websites usually have

CTAs Headlines Graphic contents that are relevant to your sales goals

But be sure, to not spend months and months testing every element of your website. So, be wise in choosing the elements to test - this saves you time and dime simultaneously.

Other than elements on your website you can test the following events you perform


In general, emails contain the same set of content to analyse. For instance, the first metric to we track in an email would be its CTR (click through rate). And the other metrics we pay attention to in our email marketing are conversion rate, growth rate, list through rate and email sharing or forwarding rate.

How to A/B test your ads?

But in ads, there are fewer things you need to concentrate on, especially if they are text ads. In a text ad you can test the headline to see if they sound intriguing. Since, text ads have only less than 7 seconds to convert visitors. So likely, in a text ad, you would analyse the main headline or the offer itself.

Why you should A/B test your offers?

Testing different offers must be considered under A/B testing. It is necessary to make sure that each group is offered the same type of promotion every time. If group A is shown discount option and group B with free gifts option, every time they login they must be targeted with the same type of promotion. And they must be tested if group A and group B has the same set of traffic.


You may also test concurrently. Once you might want to test version C landing page for newsletter D and vice versa. Testing materials like this will help you figure out what works really well. In scenarios, where the reports may seem so close to decide on which one to choose - A/B testing will help.

Start My Free 30 Day Trial

Sign up for a 30-day free trial or speak with our CRO expert so we can learn about your goals.

Why should you run heatmap before A/B testing?

By running heatmaps you can get insights into user behavior that lets you frame effective hypotheses of your A/B test. Visual representation of data offers better insight. Heatmaps in conjunction with A/B testing is a great way to make changes based on logic and data instead of blindly testing and changing elements.

By running heatmaps, before A/B testing, you can see how your visitors are actually using your website.

What are the other features to use before A/B testing?

Form analytics and Polls & Feedback are the other features that will help you gather data around what to analyse on your website by A/B testing.

Form Analytics: Helps you figure out how visitors are interacting with your webform.

Forms help you increase conversions through stats on time taken, hesitation points, refocus and corrections.

Polls & Feedback: Collects visitor opinions instantly.

Polls helps you understand your audience and their opinions. You can create different types of poll such as radio buttons, checkboxes, short and long answers.

How much traffic should you A/B test?

This seems to be a common question in CRO space. Well, if you don’t have a million visitors coming to your website or thousands of conversions happening every month then A/B testing your website may not be an ideal way to proceed. Having low traffic and running an A/B test for it might take months to get certainty. But overall, A/B testing a website short on traffic can be a little tricky but not futile. Roughly a website must have 1000 visitors every month or 50 conversions per week.

Websites getting 100 or more visitors/month can focus on micro conversion- no. of visits, page clicks, scroll percentage etc. Websites getting <100 visitors/month can focus on macro conversion- sign ups, purchases etc.

Refer: How to use A/B test calculator?

How many days should you run an A/B test?

It is recommended to run A/B tests at least for seven days even if you have a statistical winner earlier. Running A/B tests for a week will account for A/B testing user personas, advertising and user device variants. Always run a test to statistical significance 95% or more.

Refer: How to use A/B test calculator?

What is statistical significance?

Statistical significance is the measure of accurate interpretation of data and the actions that you take based on these interpretations. Only by leveraging your knowledge upon analytics and understanding the context, you can interpret accurately and take decisions.

Most analytics tools suggest 95% significance as the best test case scenario

The lower your significance rate is, the sooner you get the reports. But the risk that you impose by implementing the output is higher.

Observational error is the difference between collected data and the actual data. This mostly depends on the sample size that you test and duration of the test which you are running.

The observation error can at most be 5%.

How to use A/B test calculator?

By using A/B test calculator you can get the sample size you should analyse and how optimal test duration.

Sampling calculator: Shows you the size of the sample you should test

Duration calculator: Shows you the no.of.days you should run the test

Significance calculator: Shows you the statistically significant variation

A/B Testing summary.

By A/B testing, you can experiment on different parts of your website in order to optimize it continuously.

Run gradual (testing single/minimum elements at a time) A/B testing instead of radical A/B testing (testing multiple elements at a time).

A/B testing is the only way to check all the conversion possibilities of your website or the marketing efforts that you make.

All virtual elements of your marketing material can be tested.

You may also test concurrently.

In scenarios, where data from the reports may seem too close to make a decision, A/B testing will help.

With heatmap data if you run an A/B test, you will be able to find out the reason why the variation outperformed your existing web page.

Roughly a website must have a minimum of 1000 visitors every month or 50 conversions per week.

It is suggested to run A/B tests, always, at least for seven days.

Lower your significance rate, the sooner you get the reports but the risks that you impose by implementing the output is higher.

The observation error can be 5%.

Zarget and A/B Testing

With Zarget‘s A/B testing, you can create variations of your web pages and test their impact without hurting your business bottom line. To optimize your funnel and track all the clicks, A/B test. To split your traffic and test their performance, A/B test. Fortunately, Zarget ‘s a/b testing tool comes with an intelligent visual editor, that will help you make changes on your website/web pages without seeking any developer’s hands. It also offers chrome plugin, for easy access to the tests you are running on your website. And if you are struggling to see how your customers behave beyond the login screens, Zarget has a way out! It allows you to run tests beyond the login screens.

Exciting! Isn’t it?

Start testing with Zarget and get better business insights.

Glimpse into the future of A/B testing

Future of A/B testing is more inclined to predictive analytics and all that jazz. With the emergence of new technologies, machine learning is winning higher votes which mean that most of the manual works will be automated in future. Once this comes into existence the machines and intelligence algorithm will start coming up with personalization rules on their own.

Fret not, humans will still be needed for qualitative analysis.

Apparently, the technologies will keep emerging, but the methodologies will remain the same. And one fact of testing with new technology is, there will be the handful of methods to slice and dice the data in different ways. Which makes the process of testing as flawless as possible. Right now, for most of the sites with A/B testing it takes too long and thus with the new technology testing will become as easy as pie.

Verdict: One thing that most of our CRO experts are trying to materialize for A/B testing is Accuracy and a good deal of Automation.

Start My Free 30 Day Trial

Sign up for a 30-day free trial or speak with our CRO expert so we can learn about your goals.