Automated Site Content MVT (A/B) Testing in the Streaming Era

Part 1: Content Rotation

In this series of posts I will describe a modern possibility for building the middleware and backend necessary to automate content rotation on a website. Ultimately, the point of this will be to aggregate data about how each content combination performed, with respect to a collected key performance indicator corresponding to a conversion. The benefits of adding a streaming platform (the design of which I can propose in a future post) for data processing to this setup come out in the ability to dynamically change the content on a website to maximize the chances of said conversion occurring. But first it’s necessary to create a system that can serve up those rotations, and it needs to be FAST!

Assumptions

JS Tag: What we’re assuming we have at this point (which is a big assumption as it’s less than trivial even for an experienced Javascript developer), is a Tag written in JS that the client adds to their header. That Tag would need to be able to switch out content on a website based on the response of your externally-facing content rotation server. Ideally, and this is hard to achieve, it would be quick enough to retrieve and place all of this info before the DOM has loaded fully– preventing a flicker of the page.

Management Apps: The second piece you’ll need here are administrative and reporting Apps. Users need to be able to tell the system which pieces of content they want rotated. As well, they’ll need to specify the different pieces of code or HTML that they want to comprise each of those content changes. Often times, the slickest way to do this is to give the clients a Chrome Plugin, or possibly load an iframe of their site, which allows them to click on what they want changed and see how it changes in real time using an editor. This plugin/app will need to then pass that info about what the client wants changed and how to do it on to the backend.

Server Design

Our most crucial starting invention in this workflow will be to invent a service capable of content rotation. Consider this high-level workflow:

  1. The user enters the client’s site, they should have your JS Tag already in all their page headers. The Tag will be setup to always send an ‘orienting’ call to your Content Rotation Service. And so this is indeed where we should focus our first efforts. The nested entries here all pertain to mission-critical topic of cache design
    1. The Content Rotation call needs to be the most lightning quick of this whole operation as causing at-all significant delays in the loading of a client site is absolutely devastating to your marketability and UX
    2. In the name of speed, caching is key; and there should be no case where one is going to disk. The culmination of this thought is that we should use a mix; first with of local caching of the pages we are trying to change and the content to insert. Then below that will be distributed caching (with a persistence backup, in case of crashes) such that all your Content Rotation servers have access to the same site alteration instructions
    3. The consequence of our caching needs will be that these servers can’t rely on any central object storage DBs that your management and reporting services regularly use. They will instead need their own storage caches, preferably only used by them as the ‘orienting’ call’s speed is king and shouldn’t compete with other consumers
    4. Lastly for caching, it will be necessary to invalidate the cache entries for page content for any pages that are edited or deleted. This need can be fulfilled by creating a messaging (or even notifications of new writes in Zookeeper, as you can see in one of my earlier posts Zookeeper as a Messaging Service?) service that which notifies all your Content Rotation Servers that they need to invalidate their local caches as well as updating the distributed entry
  2. Once your ‘orienting’ call has retrieved the possible content that we want to show on this page, and we’re still within the Content Rotation Server at this point, it will need to decide what to show. If it turned out we’re not interested in that page, then return nothing which will tell the JS Tag to let the page load as normal. Later on I plan to go into a much deeper discussion of different ways to decide this but at first I’d recommend a simple uniform random distribution (e.g. if you have 4 different content options, do a contentArray[Rand.nextInt(4)]). Having decided, send the chosen content back to the user’s page ASAP
  3. Now your Tag is ready to do its main thing, and switch out the page content before anyone is the wiser. The result will be that the user sees one of your possible combinations of content. Many services will only allow clients to swap out once piece of content at a time. I would recommend against this as an assumption in any of your low-level designs, front-end or back. While it may make sense to start with one piece of rotated content (i.e. A/B Testing) to achieve Minimum Viable Product. However, in the long run there will be SO MUCH more value to adding support for multiple points of content rotation– a technique referred to as Multivariate or MVT Testing

And that’s it! You’ve swapped out the content on a user’s site, the first major step in building a framework for automated optimization of client’s site/app. This will absolutely be the toughest part for the front-end developers; especially if you include the massively complicated to design setup plugin that will allow clients to specify what changes they want to test. However, even considering the bottleneck smashing toughness involved in the ‘orienting’ call, for the back-end team this is only the start. The difficulty will escalate throughout, with its first step-up occurring in Part 2. There we will discuss tracking the user’s actions (and most importantly conversions), and getting them into storage where they can be useful.

Share