Roy C. Jones, CFRE
My job today is about “doing the math”… testing, reading results, retesting, back testing and beating the control. In 1977 as a senior in high school if someone had told me that “you will be a MATH consultant, teacher and mentor for full time profession” I would have clicked my platform shoes heels together and struck a John Travolta pose and told them… YOU ARE NUTS!
It would take me another 20 years after graduating from high school before I realized my career had brought me full circle back into a classroom to “learn the math.” I had learned the basics of direct response fundraising “on the job” raising money for political candidates and advocacy organizations in Washington, D.C. My college experience had been studying political science and preparing for a career in public service.
As I excelled in my chosen profession I found that I became more marketable because I was not afraid to ask for money. Politicians and advocacy group leaders hated to call, write or pitch in person for money and I often stepped up and became the closer. In the 1990’s I started procuring direct mail fundraising services in addition to my major gift work.
I joined the Direct Marketing Association (DMA) and the Direct Marketing Association of Washington, D.C. (DMAW). I quickly discovered that I was surrounded by scores of mathematicians, analysts and results oriented strategists. I looked around and said… “I need a math tutor…”
The next thing I knew I was setting in a classroom in Chicago with a calculator in hand and listening to Robert Weinberg teach direct mail math. I only spent two days with Robert Weinberg through the DMA in his “Direct Marketing Math and Finance Seminar” but the lessons changed my life forever. We discussed how to assess a single test for response rate and average gift and, in short, how to “BEAT THE CONTROL”. We looked at how to determine the difference between a successful test and a failure.
Mastering the basic mathematical concepts you need to calculate response rates, cost-per-donor, breakeven, and lifetime value of a donor, and avoiding the risk of potentially major – and often preventable – losses. The direct marketing “arithmetic” was no different than the my high school algebra and trigonometry class, but what was different and invigorating to me was that I was helping leaders, politicians and advocacy organizations raise funds to make the world a better place. Math was now becoming exciting to me.
Conducting the A Successful Test. There are two basic types of testing, A/B split testing and the Taguchi Method. In the case of direct mail, A/B split testing is done by sending out two letters, sometimes simultaneously, to test one element of the letter, the offer, the format or the list. Perhaps letter “A” is testing one headline and letter “B” is testing another. The letter with the headline that generates the most responses becomes what’s referred to as the control. Other elements of the letter, such as the offer, can also be tested against the control.
A/B split testing can only test one element at a time, while the Taguchi Method can test multiple elements simultaneously. Taguchi finds what combination of the different elements will have the strongest probability of pulling the most responses. According to Forbes, Dell, Inc. used The Taguchi Method for an e–mail campaign that had 11 different ad elements and 10,365 possible combinations. The Taguchi algorithm selected 18 of the combinations. Randomly selected e–mails were then sent to 18 test groups consisting of 2,000 people each. The response rate was analyzed and the most compelling features were used to create an e–mail that was sent to 150,000 customers. 1 Unfortunately, The Taguchi Method is expensive, costing up to $150,000 per project.
Analyzing Results A/B split testing is by far the simplest and cheapest way to determining package improvements, the following formulas are the stuff we all use every day to analyze campaign results:
Cost per donation – How much did each donation cost? Formula: Total cost of campaign divided by number of donations.
ROI (Return on Investment) – How profitable was the campaign? Formula: ((Total number of pieces multiplied response rate multiplied by the average donation amount) – total campaign cost) / total campaign cost
Number of Additional TEST Responses Required to Offset an Increase in Cost? Formula: ((Test CPM minus Control CPM multiplied by number of pieces mailed = Additional Cost) Amount of additional cost divided by the average gift = number of additional responses for the test to offset additional cost)
Lift in Response Rate for TEST to Offset an Increase in Cost over the Control? Formula: (Number of additional TEST responses required to offset the additional cost added to number of responses for the CONTROL) divided by the number of responses for the control= Percentage Lift in Response Rate to equal or beat the Control
For a shortcut, use this ROI calculator to determine future and current results of your promotions. The importance of tracking and testing cannot be overstated. It eliminates the guesswork; lets you optimize results from every marketing dollar you spend; and provides the data needed to justify your budget -or budget increase.
JUST FOR FUN! Let me make up a testing scenario for an A/B split test and let’s see if you can help me determine the lift in response required to beat the control. Did the response from the test get a high enough return to beat or exceed the control?
Package A: CONTROL
Control Total Cost Per Thousand w/Postage is $555/m.
Control Response Rate is 4.5%
Average Gift: $105
Total Pieces Mailed: 10,000
Total Number of Responses: 450
Total Income: $47,250
Total Cost: $5,550
Total Net: $41,700
Package B: TEST
Total Cost Per Thousand w/Postage is $1,550/m.
Control Response Rate is 5%
Average Gift: $110
Total Pieces Mailed: 10,000
Total Number of Responses: 500
Total Income: $55,000
Total Cost: $15,500
Total Net: $39,500
DID THE TEST BEAT THE CONTROL? Post your answer or send me an email at getroyjones@aol.com
References 1Kellner, Thomas, “Reengineer That Ad “, Forbes, 2005, 89-90
