Email Marketing A/B Test

Objective

To determine which email messaging would be most effective in driving traffic to the Chicago History Museum Halloween Ball microsite, we conducted a A/B split test. We took a sample of 20% of our intended mailing lists, and sent two versions of the email, one version to each half of the sample. 

Responsibilities

Assisted with copywriting, editing, structure and formatting, and deploying both versions of the emails.

Original Copy

The copy we received had only been copy-edited for punctuation and grammar:

Join us as Chicago’s top talent unites for Boo: A Halloween Ball, a night of terrifying tales, music, and craft cocktails!  Steppenwolf’s Jessica Thebus resurrects Chicago’s scariest stories in our exhibition Chicago: Crossroads of America, along with music by DJ NIX and devilishly good cocktails by Danny Shapiro of Scofflaw, Justin Cochren of Moet Hennessey, and Paul Bastien of Drumbar. Costumes are encouraged for this upscale, spooky affair. (This event is 21+; ticket price includes 2 beverages.) Hurry! Tickets are going fast!

Editing Process

1. Separate the copy for easier reading.
2. Create a short, straight-forward subject line that clearly states the purpose of the email.
3. Emphasize the name of the event and create a subheading that sums up what the event features.
4. Insert time and date of event, which is crucial information.
5. Include chance to win free airline tickets as an extra incentive to attend the event.

Version A  

This version was our most concise, focusing solely on the experiences and amenities the event would provide. The copy was meant to be skimmable, offering up just enough detail to encourage email recipients to click through and read more information about the event. The event details listed on the website would then further motivate ticket sales.

Version B 

This version emphasized the “who” rather than the “what” as the primary motivation for attending. Chicago’s local talent — all big names in their industries — would encourage readers to explore the event details and activities further on the website after the call-to-action.

Hypothesis

Both emails contained identical subject lines, images and the same call-to-action, but different event details. Version A will be more successful. It is concise, and offers up just enough details to give readers a sense of the event’s experience without resulting in fatigue. In Version B, the list of information might become overwhelming. The details about the experiences and the amenities provided at the event may have gotten lost in the long list of local celebrities’ names, whom some of our audience may not even be familiar with.

Results

Used .082 (Remainder click rate) to calculate the potential click rate for the total email, which would equal the total sends (27678) times the click through rate for your copy (.082) = 385 clicks vs. their click rate (.058) times total sends (27678) = 272

The difference between your potential clicks (385) and theirs (272) is 113. 113/272 = 41.5%, assuming their copy is the incumbent in this test.

Because both emails contained identical subject lines, their open rate was identical. But Version A has received significantly more many click-throughs than Version B. Thus, we sent Version A to the remaining 80% of recipients.

In the end, our audience cared more for details about the event experience, than the names of those that would be at the event.

I also helped design and build the Boo: A Halloween Ball branded microsite which promoted and sold tickets for this event.