Let the data guide you: How to conduct A/B testing
As a product manager, you need to make decisions often. What can assist you to make a decision without questioning yourself? A/B testing is one of essential skills of a product manager. It gives you the quantitive evidence to prove your assumption, which is a scientistic way to measure your product change. Let the data guide you.
What is A/B testing
According to Harvard Business Review, A/B testing is a way to compare two versions of something to figure out which performs better. It can be applied in many areas, like marketing, e-commerce, graphic design, and software engineering, especially on websites and apps. From a small feature to a large scale product design revamp, you can use A/B testing to help you make decisions.
How to conduct A/B testing
1. Make an Assumption and your goal
You start A/B testing by deciding what you want to test. For example, you assume that if you change the color of a CTA button on a web page, it will increase the CTR (click through rate).
2. Define metrics
After making an assumption, you need to know how to evaluate the performance. Take the above button color change as an example, obviously the metric you need is the CTR of the button. If you are doing a product revamp project, changing the whole UI of the page, you need to dig deeper into the metrics. Daily active users, page views, session duration, clicks, revenue, etc, are all important metrics you need to pay attention to.
3. Design experiment
Now, it’s time to set up your buckets. You need to set one control bucket and one or more treatment buckets. For example, you can set one bucket as control and one bucket is treatment, and pick 10% of users at random. And you have to define your experiment start time and end time. If your product is cyclical where it has higher engagement on weekdays and lower on weekends, such as stock related products. You have to set your experiment period to contain at least one complete cycle; if it may be affected by other external factors, like the public news, you have to extend your experiment period to avoid the bias.
4. Measure impact
Great! Finally the experiment completes. At this time, you use statistical analysis to compare the results of the treatment bucket with the control bucket, to determine which version performs better. The one with the better metrics, wins. However, while you analyze the result, remember to consider the statistical significance. Your tool may provide the information, including the level of confidence and p-value. If not, you can use the significance calculator to know the confidence level of the result.
Tools
- FireBase
- Google Optimize
- Optimizely
- Mailchimp (for email content A/B testing)
- SurveyMonkey A/B testing calculator
