Foundations: Lifecycle vs Channel Testing

Channel testing optimizes towards channel systems . Lifecycle testing optimizes towards audience behavior. You need both.

What Is Channel Testing: 

Channel Testing is another name for traditional A/B or multivariate testing. The test takes place in a single channel. For example: website traffic, email marketing list, paid social campaign, or direct mailers. 

The test group is a random sample of the channel’s entire audience. This means that the test is not focusing on a specific type of customer or prospect.

The test variant is successful if it drives a statistically significant lift in the desired metric, usually conversion rate or sales. The person running the test may decide to periodically re-test the variant to confirm the hypothesis is still valid.

What Is Lifecycle Testing:

There are a few key differences between Lifecycle Testing and Channel Testing:

The test group is a specific audience segment whose behavior we want to influence. For example: lapsed customers or first time buyers.

Lifecycle testing can take place across multiple channels, but only if a member of the channel audience is in the test group.

Instead of defining a single test and control experience, the test hypothesis may “come to life” in different ways across different channels. 

For example, you may try to “quarantine” a customer segment from clearance messaging by excluding them from clearance-focused email sends and by showing them full-price banners on the homepage. This could hypothetically increase the audience’s lifetime margin contribution.

The test variant is successful if it drives a statistically significant lift in conversion rate and sales for the test audience over a predetermined period of time. You’re tracking results for the audience, not for specific channel experiences.

Why Run Lifecycle Testing?

Two reasons:

First: every customer is in a different stage of their purchase journey. Some customers are very familiar with your brand and just need to see the right product. Some are much less familiar with your brand and its offering. They may need more education or even an incentive to make another purchase.

The majority of your audience is going to be prospects and casual customers, so you need to develop strategies to win more of them over if you want to grow revenue.

Second: we teach our customers how to behave, and most customers drift towards sales and promotions over time, absent any intervention from marketers. 

If we send every customer the same messages, most customers will have a read on your brand’s seasonal promotion cadence within six months, and it will be harder to get them to buy at full price.

If we limit ourselves to channel testing, the most aggressively promotional variant will win every time because it has the most urgency and broad appeal. But that doesn’t mean this is the best strategy for the long term health of your business.

Foundations is a series of posts where I explain some concepts that will become recurring themes in No Best Practices.

Subscribe for 2x/monthly dispatches from the front lines of digital transformation: