You’re no longer just a “marketer” or just a “business owner.”
As it turns out, you’re also a freaking data scientist!
That’s right.
Modern marketing requires that you move past your intuition and lean into the numbers more than ever.
Conversion rate optimization (CRO) is here to stay, and this couldn’t be truer for email marketing.
Put it this way. If you master how to effectively A/B test your email campaigns, you’ll increase traffic and sales from every email you send.
But how do you do that without getting overwhelmed, wasting time, or hiring actual data scientists, you ask?
Here’s the good news. A/B testing doesn’t have to be intimidating.
Providing you have the right approach and tools, testing can be pretty fun.
This guide will break down everything you need to know about A/B testing your email campaigns for market-leading open rates, clickthrough rates, and conversions.
Contents
What is A/B testing in email marketing? (and why it matters)
Email A/B testing, also known as email split testing, is an experiment where two or more versions of the same email are shown to subscribers at random, such as versions A and B (or variations 1 and 2).
Then the statistical analysis is used to determine which email variation performs best so you can send the “winning” email to the rest of your email list for the best results.
Here’s why email split testing is crucial:
Most email marketing tools will measure your open rates, clickthrough rates, and sometimes your conversion rates.
And every time you send an email to your list, you “hope” these numbers go up.
This is a costly game to play.
Your customers often aren’t aware of how they respond to your marketing messages, so how can you be so sure?
A/B email testing replaces “hoping” for data-driven decisions that will incrementally improve your metrics across the board.
For example, Yesware, a SaaS tool for salespeople, wanted to see if shortening the preview text on promotional emails would increase open rates.
The result: the variant saw a 16.4% increase in open rate.
They didn’t stop there. Yesware tested 3 more campaigns with shortened preview text and saw increases in open rates from 16.5% to as high as 33.3%.
Who would have thought tweaking your preview text would have such an impact? That’s the power of A/B testing your emails.
If you’re wondering how to run these tests for your email campaigns, Enchage makes this a pinch with automated A/B testing for email broadcasts and flows.
5 questions to ask before starting an A/B test email campaign
Before you start poking around your email marketing software to figure out how to run split tests, you need a plan.
Below are 5 questions for you to answer to give your email campaigns the best chance of success.
1. What’s your goal?
The first step to A/B testing an email is to determine your goal. This is easy as there are 3 main KPIs to measure in email marketing — the aforementioned open rate, clickthrough rate, and conversion rate.
Take the average of your last 5-10 email campaigns for each metric. Better yet, if you’ve been running specific campaigns or automations for years, then average out all your historical data.
Pick one KPI to track for your experiment, then set a goal for that metric. For instance, if your average click-through rate is 3.1%, then set a goal of 5%. Here’s what it would look like:
| KPI | Average | Goal |
|---|---|---|
| Open rate | 24.5% | 30% |
| Clickthrough rate | 3.1% | 5% |
| Conversion rate | 0.56% | 1.5% |
Conversion rate is the most important metric that could make a real dent in your business. However, if your open and clickthrough rates are low, we suggest you start with them. After all, it’s hard to get people to buy from your emails if they don’t open them.
Got your data points? Good. Onto the next step.
2. What’s your hypothesis?
Here’s a quick refresher from science class.
A hypothesis is a proposed prediction to a particular scenario containing independent variables that can be measured against the forecast.
In the context of email marketing, the variables are any elements in your campaign that can be changed to influence your results.
The variable you choose to hypothesize depends on your campaign goals. Below are examples of how variables influence your A/B test KPIs.
Elements that influence open rates:
- Subject line
- Sender name
- Preheader text
- Preview text
- Emojis
Elements that influence clickthrough rate:
- Imagery
- Copy
- Call to action
- Design
- Emojis
Elements that influence conversions:
- Offers
- Promises
- Timing
- Copy
- Emojis, again!
There’s a lot to unpack here, which we will cover in greater detail later. All you need to know for now is how to assign variables to your goals to formulate your hypothesis statement.
For instance, “by shortening the preview text, open rates will increase.”
Now you have your hypotheses with a defined, measurable variable to test your theory.
But one more thing…
It’s crucial that you only test one variable at a time — otherwise, your data will be unreliable.
Say you want to test version A email with a button CTA against version B email with a text link CTA. After a few hours, you find that version B had a better open rate.
What if, at the same time, you tested version A email with an image against version B email without an image?
Now it’s impossible to determine that the button CTA made all the difference in the test.
It’s best to test the button CTA first, then when you have a winner, test the image variable, and so forth.
3. What’s the sample size?
If you run an A/B test for a campaign sent to 20 email subscribers, you won’t have enough data to draw reliable conclusions.
So what’s the minimum number of subscribers required to make your email split tests statistically significant?
Thankfully, Evan Miller has created an excellent (and free) sample size calculator tool to save you from taking a dreaded statistics lesson.
Unless you have a good grasp of statistics, the terminology is likely unfamiliar to you.
Let’s briefly review the elements in the sample size calculator and how it relates to your A/B test.
Element 1. Sample size
The sample size is the number of subscribers you need to reach per variation in your email A/B test. This number is the result you need to carry out your tests.
Element 2. Baseline conversion rate (BCR)
The BCR is your current conversion rate and the starting point for your sample size calculation.
In the context of an email split test, your BCR is either your open rate, clickthrough rate, or sales conversion rate.
Element 3. Minimum detectable effect (MDE)
The MDE is the slightest possible change in your primary KPI that your test can detect with any statistical confidence. In other words, the MDE measures experiment sensitivity.
The lower the MDE, the more subscribers you need to reach in your test to account for small changes.
Let’s run through a quick activity.
Open up the calculator and set your BCR at 3% for your clickthrough rate.
As you likely know, a percentage increase in email marketing can lead to significant results. So let’s make the MDE 1%.
You’d need a sample size of 4,782 subscribers for each variation in your split test. That’s 9,564 subscribers total.
If you decreased the sensitivity of your experiment and changed the MDE from 1% to 2%, your sample size drops to 1,245 per variation, 2,490 total.
When doing any type of A/B testing in digital marketing, you’re going to be working with tight MDE figures. So technically, you’d need thousands of folks on your email list to achieve any statistical significance.
But what if you don’t have a massive email list? Is A/B testing useless?
Well, there’s no getting around it. The more data you have, the better.
Another approach is to use the 80/20 rule, also known as the Pareto principle.
The Pareto principle suggests that 20% of your efforts yield 80% of your results. So when running an A/B test on an email campaign with over 1,000 subscribers, sample 20% of your list.
10% for version A, 10% for version B, and the winner gets sent to the remaining subscribers.
In Encharge, we call this “Distribution.” Here’s how that test would look like in Encharge:
However, if your list is under 1,000 subscribers, you flip the equation around. Your sample is 80% of your list, with 40% for version A, 40% for version B. And the winning email goes to the remaining 20% of your list.
The Pareto principle is a good place to start with your email A/B testing.
But as you grow your list and you have access to more data, we suggest using Evan Miller’s free tool to calculate accurate sample sizes for your experiments with an MDE ranging between 1-5%.
4. What’s the time window?
How quickly subscribers act on your emails depends on a variety of factors, including:
- Different timezones
- Push notifications
- Subject lines
- Whether they were online or in their inbox when you sent the email
That’s why you need to set a time window to account for these variables in your email A/B test.
The longer your time window, the more accurate the results.
As a rule of thumb, wait at least 2 hours before picking your winner. Or better yet, wait an entire day.
The smaller your broadcast audience is, the longer you’ll need to wait. We see too many businesses run an A/b test with a list size smaller than 2,000 subscribers and try to determine a winner in less than 3 hours. It is impossible to have a significantly viable test with that small number of subscribers in such a short timeframe.
Another thing to consider when defining the time window is the type of the test. If you are testing for email clicks, you want to wait longer than testing for opens. People usually take longer to click an email. Also, the clicks will always be fewer than the opens, so you need to be more patient to gather enough data.
With that said, nothing beats your firsthand data. So as you run more email split tests and understand your analytics, adjust your time window accordingly.
5. What time will you send your email?
When using the A/B test tool in Encharge, we’ll send your winning email automatically. Meaning if you sample 20% of your audience, 10% for version A and 10% for version B, we’ll automatically send the winning variant to the remaining 80% of your list.
So if you want to send your winning email at a specific time, work backward from your time window.
For example, if you plan to send your winning campaign at 7 am and your time window is 3 hours, you’ll schedule your sample emails for 4 am. If your time window is 24 hours, schedule your sample emails for 7 am on the previous day.
What to split test in your email campaigns
With today’s email marketing tools, you can basically test anything for your campaigns. This is where you get to be creative!
In this section, we’re drilling down into what you can test in your email A/B experiments, including examples to help you get started.
1. Test subject lines
When looking at an email inbox, the subject line is the most prominent element and one of the first things you’ll notice. Subject lines are emphasized with darkened or more heavily formatted font to make it stand out.
According to OptinMosnter, 47% of email recipients will open your email based on your subject line. At the same time, 67% of users will mark your message as spam solely based on your subject line.
Your email open rates are dependent on your subject lines, so you need to a/b test the heck out of them!
But what should you test? Below are some ideas.
Length
How many characters of your email subject line a recipient will see will depend on:
- The device they’re using
- Browser
- Email client
Generally speaking, you want to keep your subject lines relatively short to accommodate these different environments.
Studies show that the optimum subject line length is 61-70 characters.
But guess what? You’re the scientist now, and while these statistics can provide a starting point for your hypothesis, nothing beats your own data.
Experiment: split test two subject lines that deliver the same message but make version A shorter than version B long to see which one performs best.
Emojis
It’s not 1997. Emojis are now a commonly accepted form of communication, so why not drop them in your email subject lines?
Just look at how emojis stand out in an inbox.
Another benefit of emojis is replacing emotive words with them — saving you some character space in your subject line.
However, do emojis live up to the hype? Do your customers respond positively to them? Only one way to find out. A/B test.
Experiment: test two variants of an email subject line. Variant 1 without emojis and variant 2 with emojis.
Personalization
Adding your recipient’s name to the email subject line has been shown to improve open rates.
The idea is that by adding your subscribers’ name to messages makes the communication sounds more personal.
Even though this is a more common strategy for marketers, you still don’t see too many personalized subject lines in your inbox.
Experian reported that personalized promotional emails have a 29% higher open rate.
But how does this study compare to your results? Time to test.
Experiment: A/B test your subject lines with and without your recipient’s name. So version A with name and version B without the name. You could take it a step further and also test the first name versus first name and surname.
You can dynamically insert names into your subject lines in Encharge using email personalization.
Attention-grabbing words
Words like “free,” “deal”, “promotion”, “only now” can have a positive impact on your open rates.
In his article Gregory Ciotti, content writer at Shopify, researched the 5 most persuasive words in the English language:
- You
- Free
- Because
- Instantly
- New
E-performance, a Swiss importer of electric motorcycles, uses several attention-grabbing words in their subject lines. As you can see, the subject line starting with “Promo” has generated a 10.4% increase in open rates.
Try including these words in your subject lines and track the performance of your emails.
Word order
The sequence in which you place words in your email subject line can change how the recipient interprets your message, thus impacting your open rate.
Take these subject lines as an example:
- Upgrade by August 15 to get 40% off
- Get 40% off if you upgrade by August 15
It’s the same messaging with a different ordering of words.
The benefit (getting 40% off) is placed at the beginning of the subject line in the second version. As English speakers read left to right, this puts the benefits front and center and could potentially increase the open rate.
But that’s just a hypothesis.
You know what to do. A/B test it.
Content
If you’ve ever tried sending a newsletter trying to promote multiple prices of content or various products, then you know how hard it is to write a subject line.
How do you provide enticing context to subscribers in 6-10 words?
That’s where split testing will help.
Instead of cramming a reference to each different piece of content in your subject, you could run an A/B test to see what type of content your audience resonates with.
Experiment: test two subject lines where version A summarizes all the content in your email and version B describes a singular piece of content.
Capitalization
What stands out more; free, Free, or FREE?
The capitalization of words has a different effect on different people. So don’t sleep on this seemingly insignificant adjustment, as the difference between Free or FREE in your email subject could be the difference of thousands of dollars.
Experiment: A/B test two email subject lines, each using different forms of capitalization. Version A is lowercase, and Version B is capitalized. Then test the winner against all caps.
Symbols and numbers
Like emojis, using special characters and numbers in your email subject line can break the pattern of words and catch your reader’s eyes as they scroll through their inbox.
Just be sure to make your non-standard characters relevant to your content.
Emotions and other human psychology triggers
The language you can use in email subject lines to influence your recipients to open is endless.
For example, humans struggle to resist the fear of missing out. So you could test words that communicate urgency like “today only” against “tonight only.”
Here are some other emotional triggers you could experiment with:
- Pain points
- Greed
- Vanity
- Curiosity
- Mystery
- Funny
- Direct
For its Million Dollar Year program, Dow Janes, a financial education company for women, used curiosity to entice its members to open their emails and share their wins throughout the program.
The 2nd subject line variation generated a 42% increase in opens, leveraging curiosity and a bit of mystery. Imagine if you receive an email like this — you’d most likely open it to find out what you did.
Be sure to check out these 6 subject-line frameworks to help you skyrocket your open rates.
Experiment: A/B test two or email subject lines with the same message but with different emotional triggers in the copy.
Also, check out: 11 Email Subject Line Tester Tools to Increase Your Engagement
2. Test sender information
When you receive a text message, a phone call, or a physical letter, our level of trust and urgency to respond to the communication depends on who it’s coming from.
Spam from your local real estate agent will be received differently than a letter from your local council.
The same goes for email.
Recipients scan their inboxes and open messages based on who’s trying to reach them.
So you must test your sender name and even your email address to get the best results.
Here are some ideas for experimenting with sender information:
- Brand name vs. personal name
- First name vs. full name
- Generic company email address (info@company.com, support@compamny.com, etc.) vs. personal email address (jessica@company.com)
Experiment: split test two emails containing the same content but using different sender names. For instance, Version A is your company name, and Version B is your personal name.
3. Test preview text
The preview text, also known as preheader text, is the description that appears alongside your email subject line and sender name. It provides a preview or summary of what to expect in the email message.
Typically, the preview text is automatically pulled from the first sentence of your email copy. But in most email marketing tools, you can write edit the preview text to your liking.
Preheader testing can lead to a 30% increase in email open rates — so don’t overlook it.
Preview text is an extension of your subject line. So you can do all the same testing, including length, unique characters, emotional triggers, personalization, content summaries, and so forth.
Experiment: A/B rest two emails with the same sender information and subject line, but with different preheader text. For instance, Variation 2 has a shorter copy, while Variation 1 has a longer copy.
4. Test images
Humans are visual communicators, and how you place imagery in your emails will influence clicks, conversions, and overall engagement.
Below are examples of how you can A/B test images in your emails.
- Image vs. no image
- Animated GIF vs. still image
- Stock image vs. original image
- Text on image vs. no text
Action: split test two emails with the same content but version A without images and version B with images.
5. Test email design
The layout, colors, and typeset of your email message are all factors that can influence your engagement and conversions. Choose software with an intuitive UX design tool for easy iterations and testing.
You’ve likely been using the same email template for a long time, but now’s the time to test different variations to see if you can achieve better results.
Here are some examples for A/B testing email design :
- Vibrant colors vs. pastel colors
- Garamond font vs Arial font
- Plain text template vs. visual template
Experiment: test two emails with the same message, but version A is plain text, and version B is an email template.
6. Test email copy
An email message merely used to be a digital letter. All words.
While emails have evolved to be a rich visual experience, your words still matter. You just need to test to see what resonates with your recipients. Let’s look at some examples.
- Longer vs. shorter text
- Positive tone vs. negative tone
- Personalisation vs no personalisation
Experiment: A/B test two emails with the same design and call to action, but version B addresses recipients’ first names, and version A doesn’t include personalization.
7. Test call to action
Your call to action is the purpose of your email.
Do you want subscribers to click through to a landing page, respond to your message, read a piece of content, complete a survey?
This is often the money-making action and one you need to split test thoroughly for the best results.
Here are some examples:
- Button vs. text
- Vague copy vs. specific copy
- PS vs. no PS
Paul Jarvis, author, and founder of the analytics tool Fathom, used to include his CTA as a PS at the end of his plain text emails.
Experiment: split test two emails with the same content. Version A has a button for CTA, and version B uses a text link for CTA.
Tips for running effective email marketing A/B tests
At this point, you know how to plan your email A/B test and what elements to test in your campaigns.
However, we have a couple more tips to help you run your split tests end-to-end for the best results.
Tip 1 – Use the right tools
Getting your tool stack right is not only going to save you time but will also help to optimize your results. Here’s our recommended tech stack for running email A/B tests.
Email marketing software
Your email marketing tool should have A/B testing automation built-in. Without it, you’ll have to track your campaigns manually, which is a time suck. So be sure your email software can:
- Test two or more variations of an email message
- A/B test email flows/automations as well as broadcasts
- Determine a winner after a pre-determined time window and automatically send the winning email to the rest of your list
- Analytics to measure open rates, clickthrough rates, and APIs to track sales conversions
Encharge has all these features, including editing preheader text and adding dynamic personalization through tags for more experiments.
Headline analyzer
Your email subject line is one of the most critical elements to run A/B tests on, as it will influence all your KPIs.
If you don’t have a team of in-house copywriters and wordsmiths, you could leverage a headline analyzer tool like CoSchedule’s Headline Analyzer to optimize your subject lines.
Database
Depending on how many emails you’re sending, tracking your studies can quickly get out of control.
You need a database to see your research at a glance so you can easily make iterations.
We suggest using a tool like Airtable to record and organize your email growth experiments. You can also invite collaborators to get feedback on campaigns.
Tip 2 – Prioritize your A/B tests
CRO specialists once pushed the idea to “test everything.” More always equaled better.
However, experts have since walked back this claim as A/B testing can take an excessive amount of time, costing you money. So prioritization is essential.
But with so many elements to test in a given email campaign, where the heck do you start?
There are 3 popular prioritization methods used in CRO, including PIE, ICE, and PXL. To ensure you don’t glaze over these fancy acronyms, we’ll focus on PIE to help you get started.
PIE (potential, important, ease) was created by Chris Goward at Widerfunnel. You apportion a score to the following variables to determine priority:
- Importance – what’s the significance of the element you’re testing? For example, is testing a slight change in your preview text having as much impact as changing your sender name?
- Confidence – how confident are you the test will succeed? For instance, changing the word order of your subject line is proven to be more effective than changing the word order in your body copy.
- Ease – how easy is it to create the A/B test? For example, changing the color of your CTA button is easier than designing or curating the perfect image.
For every element you want to A/B test, apply the three questions in the ICE framework to help you grade and prioritize which test you should try first.
Tip 3 – Build on your learnings
Some of your A/B tests will result in a positive increase in conversions, some will see a decrease, and others won’t have any noticeable impacts.
You must learn from your tests to apply your findings to future campaigns for the best results.
Use a combination of your email statistics, prioritization framework, and database of records to review each A/B test and make incremental improvements over time.
Put on your lab coat and start A/B testing your emails today
A/B testing is about reducing your bias and intuition and taking a data-driven marketing approach. Sure, split testing email campaigns can be overwhelming at first, but once you do a couple of experiments, you’ll be hooked.
Plus, with email marketing tools like Encharge, your A/B testing efforts are automated — making the process more accessible than ever.
So before you send your next email, pause. Develop a quick hypothesis, sample, and element to test. We suggest starting with your subject line. Then test away.
Which variation won? What did you learn?
Rinse and repeat. Trust the science. And voila…
Your open rates, clickthroughs, and conversions are bound to go up.
Start A/B testing with Encharge
If you’re thinking about trying out A/B testing, but you’re not sure where to start, take a look at Encharge.
Encharge supports A/B testing of both email newsletters (broadcasts) and flows.
To A/B test a broadcast, simply enable the A/B test feature when you are creating your broadcast.
You can run a simple A/B test or ask Encharge to determine the winner based on opens or clicks after a number of hours or days — then Encharge will send the winning variant to the rest of your audience.
You can test as many variants as you. Test different email subject lines, from email addresses, or even completely different email content; it’s up to you.
With Encharge, you can even go beyond emails and test whole different flows. Our A/B test step allows you to put people into different buckets in a flow, essentially creating completely different customer journeys. The possibilities are endless.
Sign up for a free 14-day trial with Encharge and unlock your A/B testing creativity.
