A/B Split Test | Lars Birkholm Petersen

Category: A/B split test


Must have Marketplace modules for Sitecore Digital Marketing System

January 10th, 2014 — 3:03pm

If you are using Sitecore Digital Marketing System (DMS) to optimize the Customer Experience, then make sure to look at these great modules on Sitecore Marketplace, that are ready to use and will give you some nice additions to the DMS Out of Box capabilities.

Personalization Tracker
This module can be found as part of the SBOS Accelerator Module and gives you the opportunity to easily track personalization.

This means you can better analyze views and clicks on personalized content and see the effect based on either Engagement Values or a Specific Goal (e.g. how many have filled out a specific form, based on the personalized content).

Using the module is really straight forward, once installed you get the option of enable the Personalization Tracker when you configure personalization, checking the Enable Personalization Tracker below will automatically track views, clicks and value for the different conditions.

Enable Personalization tracker in Sitecore Digital Marketing System

Once data is start collecting, you can access data by opening the Personalization Tracker from the Sitecore Desktop or Page Editor. Select the page where you have configured personalization and you can see the data for each condition, based on:

  • Views
  • Clicks
  • Engagement Value (view contributed)
  • View Effectiveness
  • Engagement Value (click contributed)
  • Click Effectiveness
  • Personalization Tracker application

    You can also select specific goals, if you would like to see contributions to conversions by views or clicks.
    With these insights you can better make decisions on:

  • Which conditions work – so you can consider applying these on a larger scale and test variations of these to optimize outcome further
  • Which conditions don’t work (look for low views or high views but low view/click effectiveness), consider deleting those or change content (if they have high views)
  • Actual Conversions Report
    As a bonus in the SBOS Accelerator module you also get access to the Actual Conversions Report, which enables you to dive into specific goal conversions, when running an A/B or MV test in Sitecore DMS.

    The Actual Conversion Report is accessible through Engagement Analytics or by selecting a test in Sitecore Marketing Center (this also works with historical tests).

    Below are a screendump of the report, here you can see information about the test, such as how many experiments to date, experiments by device, experiments by browser, combinations (if you are running a MV Test) and the variations. For each variation you can see the value contributed, based on the Engagement Values configured in DMS.

    Report for Testing in Sitecore Engagement Analytics

    If you select a test variant, you are able to see all Goal Conversions contributed by that variant – and get insights into which goals and value have contributed to the total value of the variant.

    Actual Goal Conversions using Sitecore Engagement Analytics

    Experience Explorer
    If you are working with personalization in Sitecore, then this module is your new best friend! You can download the Experience Explorer from Sitecore Marketplace here.

    The Experience Explorer allows you to, on the fly, turn on different parameters (like referral information, GEO Information, site behavior etc.) or configure presets, which could be iPad visitors from Colorado, that has arrived on the website by specific PPC Campaign and have signed up for the Newsletter – how would the experience look like for them. The Experience Explorer will use those parameters and you can browse the site and see when a personalized experience is triggered.

    Explore the Visitor Experience by using Sitecore's Experience Explorer

    You are also able to see all the personalization conditions for the different components on a page and which are currently triggered and the hierarchy of those.
    This is a must have module!

    DMS Gutters
    This module, that can be downloaded from Sitecore Marketplace here, allows you to see which pages have personalized content or a test running. The module is intended for Power Users, that are working in Sitecore Content Editor.

    Simply install the module and two new icons will appear in the gutter, if a page have personalized content or a test is enabled.

    See which pages have personalization or test running using Sitecore DMS Gutters

    I’ll get back soon – as there are more to be released very soon!

    Comment » | A/B split test, Digital Marketing System, Personalization, Sitecore

    A/B split test with Sitecore Online Marketing Suite

    May 17th, 2010 — 7:41pm

    In my previous posts I have mainly focused on Behavioral Targeting, as I see this as one of the strongest actions to get more value of a website.

    Another and very used way to optimize for better conversions is A/B split testing, which let you test different content on your site and optimize for which gets most conversions!

    You may think, that you know which content is the best; on the front page, as teaser spots or as call-to-actions, but why not let your visitors decide?

    .. And yes you can have them both, just imagine what Behavioral Targeting with a twist of A/B split testing can do for your conversions.

    With Sitecore Online Marketing Suite, you can configure and run several tests on your site. But a good rule is only to run one test at a time, so the different tests won’t influence each other.

    It’s really easy to get started – and in a short while you won’t be guessing about what works best on your site, but you can recommend actions upon real test data.

    So first up you need to have some content to test, fortunately there’s a lot, like:
    - Front page
    - Spots on front page
    - Spots on pages
    - Call-to-actions like newsletter sign-up
    - Forms like contact us (is 5 fields better than 6 and how many more conversions do we get)
    - Design changes
    - Etc-

    Lets say I have a Camera brand site and I want a spot, where the user can click and see offers for pro digital Cameras.

    I can create different spot and test those, in my example I have created two spots – one red and one dark, see below:

    Red spot test variable in Sitecore Online Marketing Suite

    Dark spot test variable in Sitecore Online Marketing Suite

    In order to setup the A/B split test in Sitecore Online Marketing Suite, I open the Marketing Center, right click on the “Test lab” section and insert a “Multivariate test”, see dump below.

    Create A/B split test in Marketing Center

    I give it a name (in this example “Pro offer spots”) and just need to fill out a description and chose whether the test strategy should be “Random” or “Sticky”, meaning that random can show different test variables to same visitor, while sticky only shows the same test variable.

    Configure A/B Split test

    Then I add my test variables, by right click on the test I have just created (Pro offer spots) and insert a “Multivariate Test Variable”, see dump below.

    Insert test variable in A/B split test

    For each test variable I give it a name and select the data source, where my content is in the Sitecore tree. In the dump below I have created a “Red BG with best offer”.

    Select A/B split test variable from Sitecore

    I create all the variables I wish to test, in my example I have just created two – one with red background and one with dark background.

    Once the test is created, all we need to do, in order to use it, is to insert it into the page(s) where we would like to have the spot.
    This can be done by using the Page Designer, where I can add the rending for tests, in my example it’s called “Flexible Personalization Spot”. See dump below where I have added the rendering to the right column in a three column layout.

    Add A/B split test rendering i Page Designer

    With the rendering selected I open the Properties and select the “Pro offer spots” from the Test lab.

    Setting the rendering properties in Page Designer

    And voila – you have created a A/B split test on your website.

    Whats left is to wait untill there is statistical significance, look at the build-in test report and see which test variable performs best.

    With the test report in Sitecore Online Marketing Suite, tests is measured against your site goals – so instead of looking at how many clicks each test variable gives, you can see the outcome!

    For more on renderings and page designer, please read How to get started with Behavioral Targeting (Part 2).

    If you want to be inspired and see what other have tested, try to visit www.abtests.com, which has a lot of examples with outcomes.

    Feel free to comment, if you have anything in mind!

    2 comments » | A/B split test, Online Marketing, Sitecore

    Back to top