Following our media partnership with Newsline Magazine recently, we found the need to re-arrange TeaBreak’s homepage. We decided to go for a ‘sectioned’ layout, i.e. the home page should feature selected sections or categories (like Politics, Business, etc.) along with their latest feeds. This will also allow us to easily scale and include sections like Newsline Magazine’s.
Once I sketched out the new layout – I wanted to compare it with the existing one and conduct an A/B test. I thought it will be interesting to give Google Website Optimizer a try. I am sharing my experience in this post.
Experiment Design
The experiment design is quite straight forward. I have two variations of the home page and I want to test which one converts better. The conversion is defined as a visit to one of the internal pages (e.g. search, archive, single post or read more).
Here’s the original layout:
And the new variation looks like:
Setting it up in Google Website Optimizer
I followed the instructions to setup an A/B Split testing on Google’s website optimizer. It was fairly self-explanatory. After setting it up it gave me three snippets of javascript code:
- Control tracking script: This snippet will go in to the original home page (i.e. the one without sections in our case).
- Conversion tracking script: This snippet will be placed on the conversion page(s). You can place the script on multiple conversion pages. In our case this is single post, search, archive and read more pages.
- Variation tracking script: Similar to the control script, you will place this on the variation of the page (i.e. the new ‘sectioned’ homepage in our case).
You also configure what % of traffic you want to go through the experiment; I set this to 100% which means that 50% of the traffic will get the original homepage and the other half will get the new version.
Now in terms of implementation a few interesting bits that I found out are:
1. Use different URLs for each of your variation:
It’s better to setup your variation under a different URL. For example in my case, I decided to use querystring “?version=variation” to flip between the original homepage and the variation. So in your code you check for this and flip between your variations.
2. Website Optimizer Javascript does the session partitioning:
Google Website Optimizer automatically assigns one of the variation to the visitor and redirects them (via Javascript) to the variation’s URL.
The control tracking script mentioned earlier does the session partitioning and will automatically ‘redirect’ the session / visitor to the allocated test (i.e. original or variation). Now, I believe they stick to their allocated variation (unless they clear their cookies).
I completely missed this trick the first time and my page went in an endless redirection loop because I was using the same URL for both the original and the variation. I was doing my own variation allocation and was emitting the appropriate javascript code (control or variation) from PHP. This is wrong, it’s much simpler than this. All you have to do is add control tracking script and it does all the magic.
3. Place the Control Tracking Script in the <head />:
Because it does a javascript redirection that is why it is important that you have the control tracking script inside <head></head> tags.
Results & Analysis
I kicked off the experiment and let it run for about 2 weeks. Website Optimizer has a very nice UI tool to see how your experiment is performing. Here’s a screenshot:
As it shows in the figure that the experiment has not thus far converged on any outcome with enough confidence. However, if you notice that there is an indication (although not with high confidence) that the variation has some chance to beat the original which is a good sign.
One option is to let it run for a longer duration to see whether it finds any high-confidence winner but if you look in the graph for the last couple of days (~5-6 days) it has pretty much stabilized. This raises the question that for how long will I have to leave it to run in order to get a high-confidence winner?
Another possible hypothesis that pops up is that perhaps the visitors and readers who were allocated the new homepage have adapted to it overtime. This might be a possible explanation for the steep downward trajectory of the variation at the start. But, then again, this is just an untested hypothesis.
Regardless, I interpret this in simple terms as:
The new homepage has not done any harm during the two weeks. It fulfills some strategic requirements so perhaps it’s safe to launch it.
Launch & Outcome
As per my interpretation (false or otherwise) of the experiment, I have now removed the query-string based switching code, removed all the javascript tracking snippets and launched the new homepage — which you can browse here.
The usage of Google Website Optimizer eliminated any room for guess work or speculative assessment on how the home page should or shouldn’t be. I think its a fairly easy, straight-forward and effective way to test a few concepts instead of debating over them.
In our case, I was able to directly compare the performance of homepage variations and was able make an informed decision based on the data.
Additionally, when go down this path you will also discover that during an experiment you might formulate a few new hypothesis and theories that you can possibly test in future experiments.
