A/B Testing

Controlled experimentation is a powerful tool that technology companies use to optimize their products and improve the user experience. Netflix, for example, personalizes the covers of series or movies according to your preferences, perhaps showing you your favorite actor while other users watch the original cover. These types of strategies, known as AB testing, allow companies to make decisions based on real data. Let's see how to implement these techniques in our own projects using Firebase.

What is AB testing and how does it differ from other strategies?

AB testing works differently from the dark launch and feature flags we have seen before. This technique allows us to create experiments of different types to evaluate the impact of specific changes on the user experience. In Firebase, we can implement AB testing through:

• Remote Config: allows you to modify application parameters without updating the code.
• InApp Messenger: uses push notifications to communicate with the users.

The main goal is to divide users into groups and show them different versions of a feature to determine which one works best according to specific metrics.

How to set up an AB testing experiment in Firebase?

To create an AB testing experiment with Remote Config in Firebase, follow these steps:

1. Select Remote Config in the Firebase console
2. Create a new experiment with a descriptive name (e.g. "TSAB")
3. Add a relevant description such as "to optimize resources".
4. Select the application we want to apply the experiment to
5. Define the percentage of users that will participate (usually 50%)
6. Configure an activation event:
   • When the application is opened
   • When the application is removed
   • On the first screen of the application

Defining objectives and variants

It is essential to establish clear metrics to measure the success of the experiment:

1. Select the objectives according to the metric we want to measure (e.g. retention of 15 days or more).
2. Define the variants for our samples:
   • Group A: parameter set to true
   • Group B: parameter set to false

Once configured, the experiment will remain in "draft" state until we start it manually from the Firebase console.

How to implement AB testing in an Android application?

To implement our experiment in Android, we need to follow these steps:

1. Create a variable in the Companion Object with the same name as the parameter defined in Remote Config:

// Example of variable definition for the experimentcompanion object { const val IS_ALERT_EMOJI = "isAlertEmoji" // Other variables...}

2. Create a variable that receives the value of the experiment
3. Implement the function that receives the remote data:

// Function that receives the experiment dataprivate fun handleRemoteConfig() { val isAlertEmoji = remoteConfig.getBoolean(IS_ALERT_EMOJI) Log.d("RemoteConfig", "isAlertEmoji: $isAlertEmoji")    
 // Condition according to the received value if (isAlertEmoji) { title.text = "Title with emoji 😊" } else { title.text = "Normal title" }}

With this implementation, half of the users will see one title and the other half will see a different one. This will allow us to evaluate which of the two options generates better retention or better meets the established objectives.

Evaluation of results

To evaluate the success of our AB test, we must:

1. Analyze the two user segments
2. Verify if the difference in retention between the two groups is statistically significant (p < 0.05).
3. If there is a significant difference, we can conclude that the change had a real impact.
4. If there is no significant difference, we will need to collect more data or design a new experiment.

Controlled experimentation is a fundamental tool in modern digital product development. Implementing AB testing allows us to make decisions based on real data and not on assumptions, continuously improving our users' experience. Have you ever implemented this type of experiments in your applications? Share your experience and results in the comments.