How to use A/B testing for ongoing site optimization

Over this past year, Sitecore Business Optimization Services (SBOS) has worked closely with Corporate Marketing to help us get started with A/B testing. Using what we’ve learned, our team can optimize and personalize effectively on a regular basis. Most importantly, we’re able to measure and react to results.

As an example, check out the results from a small tweak we made to an ebook landing page:

While a change like the one above might seem trivial, incremental changes like this add up. It’s quick and easy to test different imagery, copy changes, and design tweaks. With ample website traffic and a solid Engagement Value pyramid in place, you can quickly produce meaningful results, especially if you’re optimizing for value, not clicks.

While tweaking and testing existing pages or components can add value anytime, it’s also natural to introduce new features to your website or completely redesign existing ones periodically. In this post, we’ll explore some ways Sitecore's A/B testing can be used to validate more sizable changes.

Change is inevitable — embrace it

As businesses grow, strategies shift, and industries evolve, websites change significantly in response. At Sitecore we redesign Sitecore.com from the ground up every few years to align more closely with our evolving customer journey, brand, and content strategy.

In addition to full-fledged redesigns, some other examples of largescale digital projects include:

• Reworking entire sections of a website
• Introducing new features
• Building spin-off event sites
• Developing apps and other complementaryexperiences

On the current iteration of Sitecore.com, for example, we are in the process of reworking our Knowledge Center. In its present state, the Knowledge Center encourages our visitors to explore content by asking them to select their area of interest in a series of dropdowns.

Further down on the page, we present content in a topic-based format, starting with the basics and progressing to more advanced topics.

By redesigning this section of our website, we hope to improve our visitor’s experience while increasing consumption of our gated content associated with high-value goals on our Engagement Value pyramid. We determined that achieving this required more than incremental tweaks — a redesign was necessary.

Testing before investing

Projects such as our Knowledge Center redesign can grow large and often require budget, resources, and time to plan and execute properly. They also run the risk of following a trajectory all too familiar to folks who work in the digital space:

1. Project is planned, scoped, and estimated
2. Content and design mockups are produced
3. Mockups are passed off to the development team to be implemented
4. The project is launched
5. Time passes
6. The project does not perform as planned and is deleted or abandoned

We wanted to avoid this. Wouldn’t it be great to test the water before diving in?

This is where Sitecore’s optimization features can be leveraged in ways that are not often thought of:

• To validate assumptions about a largescale project before investing time and resources, or
• Testing during the implementation to keep the project on track while it is underway

Validate assumptions and concepts with lightweight tests

For the Knowledge Center redesign, we had a working hypothesis: If we surfaced recent and popular content up front, and enabled visitors to easily filter using their preferences, we would see more valuable content being consumed. If more valuable content was consumed, our value per visit would go up.

It seemed like a safe assumption to us. But instead of entirely redesigning the Knowledge Center to find out, we decided to test the first phase of the project while we continue to work on the overall solution.

The split A/B test we are running uses our current Knowledge Center homepage as the “A” variant. The “B” variant exposes the first phase of the redesign — including a listing of resources with the ability to search and filter. 

Preliminary results reveal that we’re on the right path, with users engaging higher in the “B” variant. And it required very little investment. With our assumptions validated, we continue the redesign confident it will be an improvement and worth the effort. 

Of course, this was only one test for one of our assumptions. But it shows how taking the time to run small tests for various aspects is a great way to garner feedback throughout the never-ending web design and development processes. We will continue to do so as we finish our redesign. 

Another aspect of testing on Sitecore that we are particularly excited about is the support we get from machine learning. Learn more about the difference this makes in “How Sitecore’s machine learning amplifies your test results.”

Derek Hunziker is the Digital Technology Director at Sitecore. Connect with him on LinkedIn.