1_Reach the “why” piece of the puzzle: A/B test results usually tell what the impact of a feature is. What this type of data doesn’t tell is the motivation behind it and why users behaved the way they did. Sometimes the why can be obvious but – most of the time – it isn’t so easy to understand. A/B testing may result negative, ambiguous or flat, but even with positive A/B tests there will be the need to iterate on a feature. Talking to users will help to easily understand where it isn’t working yet.
2_To create hypotheses to be tested: In order to provide meaningful insights, qualitative user research is actually the best tool. With the obtained insights it’s possible to generate hypotheses to test with the A/B method. Obviously A/B test can be the starting point, but qualitative research is going to be faster and more effective. Even during an advanced stage of the project, talking to users is the right tool to adjust the path.
3_A/B test can’t validate features: Not all the features can be A/B tested with meaningful results. Usually, features have so little usage that in order to get an enough consistent sample, the test has to be run forever. Moreover, features have interaction effects that include the ability to react in relation to another user’s action. Even a simple function as “like”, could impact on both people involved in this cross-user situation: A/B test hasn’t the chance to filter it correctly.
4_Unveil new opportunities: In order to expand the reach of a product or service and jump into new customer bases, there is the need to identify those new users’ needs and verify if the solution works for them. This cluster isn’t using the product/service yet, so there isn’t the chance to A/B testing. In this case qualitative research can identify these opportunities talking to prospective customer in order to test prototypes of the new solution.
“A/B testing can yield incremental improvements, but qualitative research is required to jump an entirely new level”
(Jens-Fabian Goetzmann, “Why PMs Need Qualitative Research”).
5_Faster answers: Even if talking to user involves all the existing communication problems, it’s way easier to get a qualitative feedback talking to them instead of actually building an A/B testing. It is much faster and cheaper to build a prototype rather than the whole experience. Moreover, when A/B testing, the hypothesis has to be as specific as possible, making the changes as small as possible. Bigger changes affect multiple aspects of the product/service at once, and there is no chance to be punctual with the quantitative research.
6_Stay humble: Working on a project for a long time gives the feeling of knowing what users need, sometimes having the sensation to know it better than the subjects. This isn’t true. Talking to users on a regular basis provides an indispensable checks and keeps everything on a ground level.