Read on Twitter
A simple A/B testing setup for developers, a thread:
Lately, I’ve been experimenting and learning about AB testing!
In this post, I’ll walk through how simple it is to do AB testing with Google Analytics and a bit of custom code. I'll show you an example of a successful AB test and some things I learned.
Inspired by @levelsio
In this post, I’ll walk through how simple it is to do AB testing with Google Analytics and a bit of custom code. I'll show you an example of a successful AB test and some things I learned.
Inspired by @levelsio
I don’t use Google Optimize, Mixpanel, etc as I found them to be way too clunky and hard to figure out.
The original feature
The AB test here has to do with email collection.
Image 1: The original feature, or the "A" in the AB test:
Image 2: The new thing we want to try. Instead of just asking to join the newsletter, I wanted to test out a "content upgrade".
The AB test here has to do with email collection.
Image 1: The original feature, or the "A" in the AB test:
Image 2: The new thing we want to try. Instead of just asking to join the newsletter, I wanted to test out a "content upgrade".
Getting set up with Google Analytics:
The key here is to use Google Analytics' custom dimensions.
Go to Admin -> Custom Dimensions and create a dimension for every AB test you want to try:
The key here is to use Google Analytics' custom dimensions.
Go to Admin -> Custom Dimensions and create a dimension for every AB test you want to try:
Server-side code:
For every request made to our server, run a simple method putting the user into the random bucket.
In this case, there are only two buckets: 'control' (A) and 'upsell_widget' (B). It picks a random bucket and assign a cookie (as long as it's not already there)
For every request made to our server, run a simple method putting the user into the random bucket.
In this case, there are only two buckets: 'control' (A) and 'upsell_widget' (B). It picks a random bucket and assign a cookie (as long as it's not already there)
Client-side code:
Now, on the client-side, you need to send this data over to Google Analytics for every pageview or event.
Keep in mind, you must send the mapping as well as the values. Took me forever to figure this out.
For me, it looks like something like this this:
Now, on the client-side, you need to send this data over to Google Analytics for every pageview or event.
Keep in mind, you must send the mapping as well as the values. Took me forever to figure this out.
For me, it looks like something like this this:
The last thing is to change your code based on the test that each user is in.
This is as simple as looking at what cookie they have:
(pseudocode)
This is as simple as looking at what cookie they have:
(pseudocode)
Now, set an event in your calendar to review the results a few weeks later.
The results:
In Google Analytics, go to Acquisition -> All Traffic -> Channels.
Right above the grid, look for the dropdown of Secondary Dimension, and search for your AB test:
In Google Analytics, go to Acquisition -> All Traffic -> Channels.
Right above the grid, look for the dropdown of Secondary Dimension, and search for your AB test:
For this AB test, I'll use GA "Goals" because I'm tracking email collection conversion rate, but you might be testing something else such as time on site, pageviews, etc.
Since the AB test is a custom dimension inside GA, you can do pretty much anything!
Since the AB test is a custom dimension inside GA, you can do pretty much anything!
I added all of these up inside Google Sheets and here are the results:
Bucket B in the AB test wins!
Then I plugged this into @neilpatel's AB test calculator, it shows we are statistically significant.
Then I plugged this into @neilpatel's AB test calculator, it shows we are statistically significant.
And that’s it!
Now that it’s confirmed to be statistically significant, I’ll remove the old code and think of the next AB test to run!
Thanks for reading!
Here’s also 3-minute video that walks through this as well. https://www.loom.com/share/9ff05905a7e547eca9629247b139ac6c
Now that it’s confirmed to be statistically significant, I’ll remove the old code and think of the next AB test to run!
Thanks for reading!
Here’s also 3-minute video that walks through this as well. https://www.loom.com/share/9ff05905a7e547eca9629247b139ac6c
