How do you discover what works best in email marketing? You run lots of tests. You can test subject lines, layouts, messages and the placements of links.
But email marketers rarely use the most powerful test in their arsenal—the email holdout test.
When you conduct an email holdout test, you simply stop sending email to a small percentage of your email file.
Kevin Hillstrom explains email holdout basics:
- You’ll need about ten percent of your audience to measure this well.
- Randomly sample 5% of your customer audience. Treat these customers normally.
- Randomly sample 5% of your customer audience and do not send e-mail marketing messages to this audience for a period of three months.
- Keep customers in each of these two segments for a period of three months.
- At any time during the test period, you are free to analyze the test results. Subtract the difference between the mailed group and the holdout group. This is the incremental value of sending the email. Run a profit and loss statement on this incremental difference.
- Measure the incremental difference between the mailed group and the holdout group. You can look at the impact on telephone demand, online demand, e-mail demand, paid search, natural search, affiliate marketing, and banner ads. You may find that email marketing drives incremental business to some of these channels. This is the true value of e-mail marketing. You are likely to be very surprised by the outcome of the test!