How to set up Experimental Marketing

The Anatomy of a Marketing Experiment - How to Set it Up for Success

The methodology behind marketing experiments is rather straightforward:

1. Start by brainstorming potential things to test. You can test anything from design changes, landing page configurations, value propositions, hashtags on your social media posts, pricing and discounts, or even how exactly your CTA (call to action) is phrased.
   
   
2. Prioritize and pick the test that shows the most promise. Don’t worry if you are not certain what is the best choice. Getting feedback from the market fast is more important than spending months deciding what is the best possible thing to test. Execution speed beats ideation.
   
   
3. Come up with a hypothesis that can be verified. So a statement like “In the next quarter, we’re gonna be the best e-shop ever” is painfully not specific enough. A good hypothesis puts forward a metric that can be measured and an independent variable you want to change to see if it moves the metric. A better example would be “With the optimization of our web pages loading time rollout in April (independent variable), we are going to increase our checkout conversion rate by 5% (metric).” That is clear and easily verifiable.
   
   
4. Pick your target audience and split it in two (hence the other name for marketing experiments, “A/B tests”, one for each group). To one half, named the control group, show the normal default state of things before you started the experiment. To the other half, show the proposed change (improved loading time web pages).
   
   
5. Analyze the performance to determine the winner. Compare both groups by the target metric. In the example above, we decided on the conversion rate. If the test group had a better conversion rate than the control one, your experiment was successful.
   
   
6. Rinse and repeat. Experimental marketing is about the process, not about a one-off.

There are many possible caveats to how your experiment turns wrong for all the wrong reasons. 

But the three most common issues are:

• Too small sample sizes. Imagine you’re looking through Google reviews to pick a restaurant for your date. You have two contenders. Both have an average rating of 4.8 stars. One has 5000 reviews, the other 5. Which one would you pick? Exactly, the 5000 reviews one. The same goes for your own experiments. Having too small samples makes them untrustworthy. What is a good sample size, you ask? Depends heavily on the experimental design you set up. Either consult a data scientist or rely on tools that do the calculation of sample size heavy-lifting for you (more on this later).
  
  
• Unrepresentative samples. For example, running a focus group as part of a “youth online shopping habits” market research and picking participants aged 80 and above will obviously not work out. The same goes for marketing experiments. You have to make sure you are picking the right target audience for your experiments.
  

Test too many things at once. It is called an A/B test because it compares side by side two possible realities: A, the world as is, and B, the world as it could be. If you introduce too many additional variables (C/D/E/…) it becomes hard to tell what really drove the KPI metric up or down. Was it the loading time optimization or the hero visual change on the homepage? Maybe it was a new value proposition? Did you try and test new paid marketing campaigns on TikTok? Introducing too many independent variables makes the interpretation murkier.
‍
Case Study: A subject line worth 2 million dollars

Obama’s presidential campaign happened many presidential mandates ago, but the methods used will go down in history for their incredible success. 

The majority of the campaign funds were raised via email marketing, and to make the outreach successful, the campaign managers conducted experiments with the email’s copy to optimize it.

Concretely, multiple subject lines were shortlisted as the best contenders, and emails were sent out to smaller batches of the email list for an A/B test. The winning email of those experiments was then sent to the remainder of the email list. 

How impactful were those tests? 

According to Toby Fallsgraff, Email Director, and Amelia Showalter, Director of Digital Analytics, the difference between the best and the worst email equated to 2 million dollars in donations.

This is why testing pays off.

How To Set up a Marketing Experimentation Machine that Scales?

Many analytical tools can help you with the heavy-statistical-lifting needed to run and interpret marketing experiments. 

From Facebook Ads running machine learning A/B tests on advertising campaigns to VWO, Optimizely, Crazy Egg, and Unbounce that set up and analyze experiments for you. 

These tools are great and we highly recommend them.

But their main challenge is that they are hard to scale. 

The more you experiment, the more data you collect, the more you become interested in the big picture. When you direct your marketing efforts towards multiple growth levers, you want to make sure the omnichannel approach is working.

So tools like Unbounce can only inform you about the landing page experience, and Facebook ads can only offer insights about PPC advertising.

Understanding and running experiments on your end-to-end marketing funnel necessitate a tool that can collect data from all the different tools you use for every marketing touchpoint. 

Keboola can help with that. 

Keboola brings together data from advertising platforms, CRM systems, and other data sources to help you establish and showcase the true value of your marketing efforts. 

And you don’t need a single data scientist or engineer on your team. With its plug-and-play design, you can easily export data from all of the marketing tools you are using to analyze the big picture behind your marketing experimentation. 

Check why Keboola is trusted by over 95 marketing agencies globally.