What is A/B Testing in Email Marketing? The Essential Guide

You’re no longer just a “marketer” or just a “business owner.”

As it turns out, you’re also a freaking data scientist!

Scientist GIF

That’s right.

Modern marketing requires that you move past your intuition and lean into the numbers more than ever. 

Conversion rate optimization (CRO) is here to stay, and this couldn’t be truer for email marketing.

Put it this way. If you master how to effectively A/B test your email campaigns, you’ll increase traffic and sales from every email you send.

But how do you do that without getting overwhelmed, wasting time, or hiring actual data scientists, you ask?

Here’s the good news. A/B testing doesn’t have to be intimidating. 

Providing you have the right approach and tools, testing can be pretty fun. 

This guide will break down everything you need to know about A/B testing your email campaigns for market-leading open rates, clickthrough rates, and conversions.

What is A/B testing in email marketing? (and why it matters)

Email A/B testing, also known as email split testing, is an experiment where two or more versions of the same email are shown to subscribers at random, such as versions A and B (or variations 1 and 2).

Then the statistical analysis is used to determine which email variation performs best so you can send the “winning” email to the rest of your email list for the best results.

Here’s why email split testing is crucial:

Most email marketing tools will measure your open rates, clickthrough rates, and sometimes your conversion rates.

And every time you send an email to your list, you “hope” these numbers go up. 

This is a costly game to play. 

Your customers often aren’t aware of how they respond to your marketing messages, so how can you be so sure?

A/B email testing replaces “hoping” for data-driven decisions that will incrementally improve your metrics across the board.

For example, Yesware, a SaaS tool for salespeople, wanted to see if shortening the preview text on promotional emails would increase open rates.

A/B testing preview text

The result: the variant saw a 16.4% increase in open rate.

A/B testing in email marketing example — Yesware

They didn’t stop there. Yesware tested 3 more campaigns with shortened preview text and saw increases in open rates from 16.5% to as high as 33.3%.

Who would have thought tweaking your preview text would have such an impact? That’s the power of A/B testing your emails.

If you’re wondering how to run these tests for your email campaigns, Enchage makes this a pinch with automated A/B testing for email broadcasts and flows

5 questions to ask before starting an A/B test email campaign

Before you start poking around your email marketing software to figure out how to run split tests, you need a plan.

Below are 5 questions for you to answer to give your email campaigns the best chance of success

1. What’s your goal?

The first step to A/B testing an email is to determine your goal. This is easy as there are 3 main KPIs to measure in email marketing — the aforementioned open rate, clickthrough rate, and conversion rate.

Take the average of your last 5-10 email campaigns for each metric. Better yet, if you’ve been running specific campaigns or automations for years, then average out all your historical data.

Pick one KPI to track for your experiment, then set a goal for that metric. For instance, if your average click-through rate is 3.1%, then set a goal of 5%. Here’s what it would look like:

KPIAverageGoal
Open rate24.5%30%
Clickthrough rate3.1%5%
Conversion rate0.56%1.5%

Conversion rate is the most important metric that could make a real dent in your business. However, if your open and clickthrough rates are low, we suggest you start with them. After all, it’s hard to get people to buy from your emails if they don’t open them. 

Got your data points? Good. Onto the next step.

2. What’s your hypothesis?

Here’s a quick refresher from science class.

A hypothesis is a proposed prediction to a particular scenario containing independent variables that can be measured against the forecast.

In the context of email marketing, the variables are any elements in your campaign that can be changed to influence your results. 

The variable you choose to hypothesize depends on your campaign goals. Below are examples of how variables influence your A/B test KPIs.

Elements that influence open rates:

  • Subject line
  • Sender name
  • Preheader text
  • Preview text
  • Emojis

Elements that influence clickthrough rate:

  • Imagery
  • Copy
  • Call to action
  • Design
  • Emojis

Elements that influence conversions:

  • Offers
  • Promises
  • Timing
  • Copy
  • Emojis, again!

There’s a lot to unpack here, which we will cover in greater detail later. All you need to know for now is how to assign variables to your goals to formulate your hypothesis statement.

For instance, “by shortening the preview text, open rates will increase.” 

Now you have your hypotheses with a defined, measurable variable to test your theory.

But one more thing…

It’s crucial that you only test one variable at a time — otherwise, your data will be unreliable.

Say you want to test version A email with a button CTA against version B email with a text link CTA. After a few hours, you find that version B had a better open rate.

What if, at the same time, you tested version A email with an image against version B email without an image?

Now it’s impossible to determine that the button CTA made all the difference in the test. 

It’s best to test the button CTA first, then when you have a winner, test the image variable, and so forth. 

3. What’s the sample size?

If you run an A/B test for a campaign sent to 20 email subscribers, you won’t have enough data to draw reliable conclusions.

So what’s the minimum number of subscribers required to make your email split tests statistically significant?

Thankfully, Evan Miller has created an excellent (and free) sample size calculator tool to save you from taking a dreaded statistics lesson.

A/B test data significance calculator

Unless you have a good grasp of statistics, the terminology is likely unfamiliar to you.

Let’s briefly review the elements in the sample size calculator and how it relates to your A/B test.

Element 1. Sample size

The sample size is the number of subscribers you need to reach per variation in your email A/B test. This number is the result you need to carry out your tests.

Element 2. Baseline conversion rate (BCR)

The BCR is your current conversion rate and the starting point for your sample size calculation. 

In the context of an email split test, your BCR is either your open rate, clickthrough rate, or sales conversion rate.

Element 3. Minimum detectable effect (MDE)

The MDE is the slightest possible change in your primary KPI that your test can detect with any statistical confidence. In other words, the MDE measures experiment sensitivity.

The lower the MDE, the more subscribers you need to reach in your test to account for small changes. 

Let’s run through a quick activity.

Open up the calculator and set your BCR at 3% for your clickthrough rate.

As you likely know, a percentage increase in email marketing can lead to significant results. So let’s make the MDE 1%.

A/B test data significance calculator

You’d need a sample size of 4,782 subscribers for each variation in your split test. That’s 9,564 subscribers total.

If you decreased the sensitivity of your experiment and changed the MDE from 1% to 2%, your sample size drops to 1,245 per variation, 2,490 total.

A/B test data significance calculator

When doing any type of A/B testing in digital marketing, you’re going to be working with tight MDE figures. So technically, you’d need thousands of folks on your email list to achieve any statistical significance.

But what if you don’t have a massive email list? Is A/B testing useless?

Well, there’s no getting around it. The more data you have, the better.

Whiteboard Gif

Another approach is to use the 80/20 rule, also known as the Pareto principle.

The Pareto principle suggests that 20% of your efforts yield 80% of your results. So when running an A/B test on an email campaign with over 1,000 subscribers, sample 20% of your list. 

10% for version A, 10% for version B, and the winner gets sent to the remaining subscribers.

In Encharge, we call this “Distribution.” Here’s how that test would look like in Encharge:

Email A/B testing distribution

However, if your list is under 1,000 subscribers, you flip the equation around. Your sample is 80% of your list, with 40% for version A, 40% for version B. And the winning email goes to the remaining 20% of your list.

Email A/B testing distribution

The Pareto principle is a good place to start with your email A/B testing. 

But as you grow your list and you have access to more data, we suggest using Evan Miller’s free tool to calculate accurate sample sizes for your experiments with an MDE ranging between 1-5%.

4. What’s the time window?

How quickly subscribers act on your emails depends on a variety of factors, including:

  • Different timezones
  • Push notifications
  • Subject lines
  • Whether they were online or in their inbox when you sent the email

That’s why you need to set a time window to account for these variables in your email A/B test. 

The longer your time window, the more accurate the results.

As a rule of thumb, wait at least 2 hours before picking your winner. Or better yet, wait an entire day. 

The smaller your broadcast audience is, the longer you’ll need to wait. We see too many businesses run an A/b test with a list size smaller than 2,000 subscribers and try to determine a winner in less than 3 hours. It is impossible to have a significantly viable test with that small number of subscribers in such a short timeframe.

Another thing to consider when defining the time window is the type of the test. If you are testing for email clicks, you want to wait longer than testing for opens. People usually take longer to click an email. Also, the clicks will always be fewer than the opens, so you need to be more patient to gather enough data.

With that said, nothing beats your firsthand data. So as you run more email split tests and understand your analytics, adjust your time window accordingly.

5. What time will you send your email?

When using the A/B test tool in Encharge, we’ll send your winning email automatically. Meaning if you sample 20% of your audience, 10% for version A and 10% for version B, we’ll automatically send the winning variant to the remaining 80% of your list.

A/B testing in email marketing campaign

So if you want to send your winning email at a specific time, work backward from your time window.

For example, if you plan to send your winning campaign at 7 am and your time window is 3 hours, you’ll schedule your sample emails for 4 am. If your time window is 24 hours, schedule your sample emails for 7 am on the previous day.

What to split test in your email campaigns

With today’s email marketing tools, you can basically test anything for your campaigns. This is where you get to be creative!

In this section, we’re drilling down into what you can test in your email A/B experiments, including examples to help you get started.

A/B testing in email marketing — things to test

1. Test subject lines

When looking at an email inbox, the subject line is the most prominent element and one of the first things you’ll notice. Subject lines are emphasized with darkened or more heavily formatted font to make it stand out.

Inbox example

According to OptinMosnter, 47% of email recipients will open your email based on your subject line. At the same time, 67% of users will mark your message as spam solely based on your subject line.

Your email open rates are dependent on your subject lines, so you need to a/b test the heck out of them! 

But what should you test? Below are some ideas.

Length

How many characters of your email subject line a recipient will see will depend on:

  • The device they’re using
  • Browser
  • Email client

Generally speaking, you want to keep your subject lines relatively short to accommodate these different environments. 

Studies show that the optimum subject line length is 61-70 characters.

Subject Line Length and Average Read Rate graph

But guess what? You’re the scientist now, and while these statistics can provide a starting point for your hypothesis, nothing beats your own data. 

Experiment: split test two subject lines that deliver the same message but make version A shorter than version B long to see which one performs best.

Emojis

It’s not 1997. Emojis are now a commonly accepted form of communication, so why not drop them in your email subject lines?

Just look at how emojis stand out in an inbox.

Another benefit of emojis is replacing emotive words with them — saving you some character space in your subject line.

However, do emojis live up to the hype? Do your customers respond positively to them? Only one way to find out. A/B test.

Experiment: test two variants of an email subject line. Variant 1 without emojis and variant 2 with emojis. 

Personalization

Adding your recipient’s name to the email subject line has been shown to improve open rates. 

The idea is that by adding your subscribers’ name to messages makes the communication sounds more personal.

Even though this is a more common strategy for marketers, you still don’t see too many personalized subject lines in your inbox.

Personalization in subject line example

Experian reported that personalized promotional emails have a 29% higher open rate.

But how does this study compare to your results? Time to test.

Experiment: A/B test your subject lines with and without your recipient’s name. So version A with name and version B without the name. You could take it a step further and also test the first name versus first name and surname.

You can dynamically insert names into your subject lines in Encharge using email personalization.

Attention-grabbing words

Words like “free,” “deal”, “promotion”, “only now” can have a positive impact on your open rates.

In his article Gregory Ciotti, content writer at Shopify, researched the 5 most persuasive words in the English language:

  • You
  • Free
  • Because
  • Instantly
  • New

E-performance, a Swiss importer of electric motorcycles, uses several attention-grabbing words in their subject lines. As you can see, the subject line starting with “Promo” has generated a 10.4% increase in open rates.

Try including these words in your subject lines and track the performance of your emails.

A/B testing in email marketing example — e-performance

Word order

The sequence in which you place words in your email subject line can change how the recipient interprets your message, thus impacting your open rate.

Take these subject lines as an example:

  • Upgrade by August 15 to get 40% off
  • Get 40% off if you upgrade by August 15

It’s the same messaging with a different ordering of words.

The benefit (getting 40% off) is placed at the beginning of the subject line in the second version. As English speakers read left to right, this puts the benefits front and center and could potentially increase the open rate.

But that’s just a hypothesis.

You know what to do. A/B test it.

Content

If you’ve ever tried sending a newsletter trying to promote multiple prices of content or various products, then you know how hard it is to write a subject line.

How do you provide enticing context to subscribers in 6-10 words? 

That’s where split testing will help. 

Instead of cramming a reference to each different piece of content in your subject, you could run an A/B test to see what type of content your audience resonates with.

Experiment: test two subject lines where version A summarizes all the content in your email and version B describes a singular piece of content.

Capitalization

What stands out more; free, Free, or FREE?

The capitalization of words has a different effect on different people. So don’t sleep on this seemingly insignificant adjustment, as the difference between Free or FREE in your email subject could be the difference of thousands of dollars.

Experiment: A/B test two email subject lines, each using different forms of capitalization. Version A is lowercase, and Version B is capitalized. Then test the winner against all caps. 

Symbols and numbers

Like emojis, using special characters and numbers in your email subject line can break the pattern of words and catch your reader’s eyes as they scroll through their inbox.

Just be sure to make your non-standard characters relevant to your content. 

Emotions and other human psychology triggers

The language you can use in email subject lines to influence your recipients to open is endless. 

For example, humans struggle to resist the fear of missing out. So you could test words that communicate urgency like “today only” against “tonight only.”

Here are some other emotional triggers you could experiment with:

  • Pain points
  • Greed
  • Vanity
  • Curiosity
  • Mystery
  • Funny
  • Direct

For its Million Dollar Year program, Dow Janes, a financial education company for women, used curiosity to entice its members to open their emails and share their wins throughout the program. 

The 2nd subject line variation generated a 42% increase in opens, leveraging curiosity and a bit of mystery. Imagine if you receive an email like this — you’d most likely open it to find out what you did.

A/B testing in email marketing example — DowJanes

Be sure to check out these 6 subject-line frameworks to help you skyrocket your open rates.

Experiment: A/B test two or email subject lines with the same message but with different emotional triggers in the copy. 

2. Test sender information

When you receive a text message, a phone call, or a physical letter, our level of trust and urgency to respond to the communication depends on who it’s coming from.

Spam from your local real estate agent will be received differently than a letter from your local council.

The same goes for email. 

Recipients scan their inboxes and open messages based on who’s trying to reach them.

So you must test your sender name and even your email address to get the best results. 

Here are some ideas for experimenting with sender information:

Experiment: split test two emails containing the same content but using different sender names. For instance, Version A is your company name, and Version B is your personal name.

3. Test preview text

The preview text, also known as preheader text, is the description that appears alongside your email subject line and sender name. It provides a preview or summary of what to expect in the email message.

Example preview text

Typically, the preview text is automatically pulled from the first sentence of your email copy. But in most email marketing tools, you can write edit the preview text to your liking. 

Preheader testing can lead to a 30% increase in email open rates — so don’t overlook it.

Preview text is an extension of your subject line. So you can do all the same testing, including length, unique characters, emotional triggers, personalization, content summaries, and so forth.

Experiment: A/B rest two emails with the same sender information and subject line, but with different preheader text. For instance, Variation 2 has a shorter copy, while Variation 1 has a longer copy.

4. Test images

Humans are visual communicators, and how you place imagery in your emails will influence clicks, conversions, and overall engagement. 

Below are examples of how you can A/B test images in your emails.

  • Image vs. no image
  • Animated GIF vs. still image
  • Stock image vs. original image
  • Text on image vs. no text

Action: split test two emails with the same content but version A without images and version B with images.

5. Test email design

The layout, colors, and typeset of your email message are all factors that can influence your engagement and conversions.

Email design

You’ve likely been using the same email template for a long time, but now’s the time to test different variations to see if you can achieve better results.

Here are some examples for A/B testing email design :

  • Vibrant colors vs. pastel colors
  • Garamond font vs Arial font
  • Plain text template vs. visual template

Experiment: test two emails with the same message, but version A is plain text, and version B is an email template.

6. Test email copy

An email message merely used to be a digital letter. All words. 

While emails have evolved to be a rich visual experience, your words still matter. You just need to test to see what resonates with your recipients. Let’s look at some examples.

  • Longer vs. shorter text
  • Positive tone vs. negative tone
  • Personalisation vs no personalisation 

Experiment: A/B test two emails with the same design and call to action, but version B addresses recipients’ first names, and version A doesn’t include personalization.

7. Test call to action

Your call to action is the purpose of your email. 

Do you want subscribers to click through to a landing page, respond to your message, read a piece of content, complete a survey?

This is often the money-making action and one you need to split test thoroughly for the best results.

Here are some examples:

  • Button vs. text
  • Vague copy vs. specific copy
  • PS vs. no PS

Paul Jarvis, author, and founder of the analytics tool Fathom, used to include his CTA as a PS at the end of his plain text emails.

CTA in a PS

Experiment: split test two emails with the same content. Version A has a button for CTA, and version B uses a text link for CTA. 

Tips for running effective email marketing A/B tests

At this point, you know how to plan your email A/B test and what elements to test in your campaigns.

However, we have a couple more tips to help you run your split tests end-to-end for the best results.

Tip 1 – Use the right tools

Getting your tool stack right is not only going to save you time but will also help to optimize your results. Here’s our recommended tech stack for running email A/B tests.

Email marketing software

Your email marketing tool should have A/B testing automation built-in. Without it, you’ll have to track your campaigns manually, which is a time suck. So be sure your email software can:

  • Test two or more variations of an email message
  • A/B test email flows/automations as well as broadcasts
  • Determine a winner after a pre-determined time window and automatically send the winning email to the rest of your list
  • Analytics to measure open rates, clickthrough rates, and APIs to track sales conversions

Encharge has all these features, including editing preheader text and adding dynamic personalization through tags for more experiments. 

Headline analyzer

Your email subject line is one of the most critical elements to run A/B tests on, as it will influence all your KPIs.

If you don’t have a team of in-house copywriters and wordsmiths, you could leverage a headline analyzer tool like CoSchedule’s Headline Analyzer to optimize your subject lines. 

Database

Depending on how many emails you’re sending, tracking your studies can quickly get out of control.

You need a database to see your research at a glance so you can easily make iterations. 

We suggest using a tool like Airtable to record and organize your email growth experiments. You can also invite collaborators to get feedback on campaigns.

Tip 2 – Prioritize your A/B tests

CRO specialists once pushed the idea to “test everything.” More always equaled better. 

Prioritize your A/B tests Gif

However, experts have since walked back this claim as A/B testing can take an excessive amount of time, costing you money. So prioritization is essential.

But with so many elements to test in a given email campaign, where the heck do you start?

There are 3 popular prioritization methods used in CRO, including PIE, ICE, and PXL. To ensure you don’t glaze over these fancy acronyms, we’ll focus on PIE to help you get started. 

PIE (potential, important, ease) was created by Chris Goward at Widerfunnel. You apportion a score to the following variables to determine priority:

  • Importancewhat’s the significance of the element you’re testing? For example, is testing a slight change in your preview text having as much impact as changing your sender name?
  • Confidencehow confident are you the test will succeed? For instance, changing the word order of your subject line is proven to be more effective than changing the word order in your body copy.
  • Easehow easy is it to create the A/B test? For example, changing the color of your CTA button is easier than designing or curating the perfect image.

For every element you want to A/B test, apply the three questions in the ICE framework to help you grade and prioritize which test you should try first.

Tip 3 – Build on your learnings

Some of your A/B tests will result in a positive increase in conversions, some will see a decrease, and others won’t have any noticeable impacts.

You must learn from your tests to apply your findings to future campaigns for the best results. 

Use a combination of your email statistics, prioritization framework, and database of records to review each A/B test and make incremental improvements over time. 

Put on your lab coat and start A/B testing your emails today

A/B testing is about reducing your bias and intuition and taking a data-driven marketing approach. Sure, split testing email campaigns can be overwhelming at first, but once you do a couple of experiments, you’ll be hooked.

Plus, with email marketing tools like Encharge, your A/B testing efforts are automated — making the process more accessible than ever.

So before you send your next email, pause. Develop a quick hypothesis, sample, and element to test. We suggest starting with your subject line. Then test away. 

Which variation won? What did you learn? 

Rinse and repeat. Trust the science. And voila… 

Your open rates, clickthroughs, and conversions are bound to go up.

Start A/B testing with Encharge

If you’re thinking about trying out A/B testing, but you’re not sure where to start, take a look at Encharge.

Encharge supports A/B testing of both email newsletters (broadcasts) and flows.

To A/B test a broadcast, simply enable the A/B test feature when you are creating your broadcast. 

You can run a simple A/B test or ask Encharge to determine the winner based on opens or clicks after a number of hours or days — then Encharge will send the winning variant to the rest of your audience.

You can test as many variants as you. Test different email subject lines, from email addresses, or even completely different email content; it’s up to you.

A/B testing in email marketing with Encharge

With Encharge, you can even go beyond emails and test whole different flows. Our A/B test step allows you to put people into different buckets in a flow, essentially creating completely different customer journeys. The possibilities are endless.


Sign up for a free 14-day trial with Encharge and unlock your A/B testing creativity.

4.8/5 from 228 reviews
4.9/5 from 158 reviews
4.9/5 from 155 reviews
4.91/5 from 154 reviews

Meet your new marketing automation platform

Customer messaging tools don’t automate workflows outside your product and marketing automation tools are bad at behavior emails. Encharge is the best of both worlds — a marketing automation platform built specifically for startups and digital businesses.

“Encharge helped us visually redesign our onboarding flow resulting in a 10% increase in our trial activation rate."

camille-photo
Camille Richon
Founder Payfacile
See why Encharge is different
Use Cases
Marketing automation
Create user journeys that convert, onboard, and retain customers.
Lead nurturing
Nurture email leads into trial users and customers.
User onboarding
Boost product activation and guide your users to value faster
Trial conversion
Smart marketing automation and behavior-based emails to double your trial conversion.
Success Stories
Landbot automates the onboarding for 80,000+ users while saving 320 hours/month
Samdock reduced the cost and time spent on acquiring a new customer by 77% with Encharge
Aito reduced the time for implementing behavior-based emails from hours to minutes
laptop
SaaS
Convert, onboard, and retain more customers. Grow your MRR with marketing automation for SaaS.
meeting
Agencies
Build advanced marketing automation flows for your clients and grow your agency faster than ever before.
document-2
Content creators
Bring your subscribers in Encharge and nurture them into customers with automated email sequences.
opening-times
Subscription businesses
Create targeted messages and automation flows for recurring-revenue businesses.
automated-logistics
Productized services
Automate your services and focus on the important parts of your business.
Flow builder
Create remarkable user journeys with a robust and easy to use visual flow builder.
Broadcasts
Send targeted one-off newsletters to your whole audience or a segment of people.
Behavior emails
Send targeted emails when people do or don’t do something in your app.
User segments
Create user segments with the market’s leading segmentation for SaaS.
User profiles
See the people behind the actions and access the full view of your customer data.
Email personalization
Get your users to act with highly-personalized emails.
Email editor
Design beautiful mobile-ready emails without any HTML skills.
A/B tests
Drive email engagement with A/B tests for your Broadcasts and your Flows.
Lead scoring
Identify interested users and best-fit customers and proactively reach out to them.
Website tracking
Track page visits and form submissions on your website in real-time.
verified
Free email verification
Free email verification for all your contacts in Encharge, on all plans.
Transactional emails
Send emails like password reset, payment receipt, and single sign-on link from your app.
Facebook Logo
Facebook
HubSpot Logo
HubSpot
Calendly Logo
Calendly
Typeform Logo
Typeform
Slack Logo
Slack
Intercom Logo
Intercom
Mailchimp Logo
Mailchimp
Salesforce Logo
Salesforce
Zapier Logo
Zapier
Pabbly Logo
Pabbly
Integrately Logo
Integrately
Stripe Logo
Stripe
Chargebee Logo
Chargebee
Chargify Logo
Chargify
Recurly Logo
Recurly
Paddle Logo
Paddle
Twilio Logo
Twilio SMS
Webhooks Logo
Webhooks
Segment Logo
Segment
API Logo
API
Google Analytics Logo
Google Analytics
SyncSpider logo
SyncSpider
KonnectzIT logo
KonnectzIT
ThriveCart Logo
ThriveCart
+38
More
Blog
The resource for startups who want to grow their revenue with marketing automation.
logo-facebook
Facebook Group
A community for all things Encharge-related.
Knowledge base
Learn how to use Encharge.
window-dev
Developer Docs
Tools for developers.
thumb-up
Feature Requests
Request new product features.
Product Updates
Latest Encharge updates and news.
Affiliate Program
Earn up to 30% recurring commission when you refer customers to Encharge.
b-check
Encharge Experts
Get help from trusted Encharge consultants and service providers.
books
Resource Library
Get access to all of our eBooks, webinars, blueprints, cheatsheets, and more.
artificial-brain
Generate Subject Lines with AI
Create unique subject lines with the only free AI email subject line generator based on GPT3.
logo-slack
Slack community
Talk to the Encharge team in real-time.