Mobile Game A/B Testing: Strategies for Improving Player Engagement

In the competitive world of mobile gaming, creating an engaging user experience is crucial. A/B testing enables developers to make data-driven decisions, ensuring their game resonates with players. Let's delve into the strategies for effective A/B testing in mobile games, complete with concrete JSON examples of parameters to test.

What is A/B Testing in Mobile Games?

A/B testing, also known as split testing, involves comparing two versions of a game element to see which performs better. In mobile gaming, this could be anything from user interface (UI) elements to game mechanics to in-game rewards. This approach is particularly important in mobile gaming, where player engagement and retention are crucial: you can significantly improve your game's performance by testing and optimizing different elements.

Don't know how to get started or need more help? Chat with us on Discord. We have scaled games to millions of players and can help you too.

Key Parameters to Test

In-Game Economy: Tweaking the economy can significantly impact player engagement. For instance, altering the cost of in-game items can affect how players purchase behavior.

Test: Compare player purchase rates between two price points for coin packs.

Example JSON showing three different purchase options to present to the player.

{
    "inAppPurchasePackages": [
        {
            "price": 0.99, "gems": 100
        },
        {
            "price": 1.99, "gems": 200
        },
        {
            "price": 2.99, "gems": 300
        }
    ]
}

Config setup with a 10% test group that gives more coins. Learn more about setting up AB Testing in Joystick.

Default ConfigDynamic Content Definition with AB Test

{
    "inAppPurchasePackages": "#LK-options:array:options#",
    "_dynamic_values_": {
        "options": [
            {
                "price": 0.99, "gems": 100
            },
            {
                "price": 1.99, "gems": 200
            },
            {
                "price": 2.99, "gems": 300
            }
        ]
    }
}

[
    {
        "name": "TestPurchaseOptions", // "name" (Optional) - Provide a short name to identify this segment or test. Max 50 chars.
        "expression": "1 = 1", // This expression is for segmenting users. In this case, we use 1 = 1 to include all users.
        "groups": [
            {
                "variantName": "Variant A", // Short name to identify this variant. Max 50 chars.
                "mod": [[0,99]], // This defines a 10% test. Max mod value is 999.
                "values": {
                    // The LK-options key matches the key in the default config, and the content here will be used for the 10% test group.
                    "LK-options": [
                        {
                            "price": 0.99, "gems": 100
                        },
                        {
                            "price": 1.99, "gems": 220 // 10% more coins at this price point
                        },
                        {
                            "price": 2.99, "gems": 360 // 20% more coins at this price point
                        }
                    ]
                }
            }
        ]
    }
]

Level Difficulty: Adjusting the difficulty of levels can help in finding the sweet spot that keeps players challenged but not frustrated.

Test: Measure user engagement and completion rates between two difficulty settings.

Example JSON:

{
    "levels": [
        {
            "level": 1, "enemies": 5
        },
        {
            "level": 2, "enemies": 5
        },
        {
            "level": 3, "enemies": 5
        }
    ]
}

UI Changes: Even small changes in UI and UX can have a big impact on player experience. This includes size, color and especially the call to action text.
- Test: Observe which button color yields higher click-through rates.
- Example JSON:
```
{
    "startButton": {
        "color": "green",
        "text": "Let's Go"
    }
}
```

Setting Up A/B Tests

Define Your Hypothesis: Clearly state what you are testing and expect to find. For example, "Lowering the price of coin packs for players who previously purchased, have not purchased for 14 days will increase in-app purchases."
Create Test Variants: Use JSON remote configurations to set up different variants in your game. Each variant should have one or more changes that are coherent and aimed towards a singular thesis.
Segment Your Audience: Divide your players into groups for your tests, and also keep a 'control' group that has no change/treatment applied. Each test group should receive a different variant of the game. Adjust the group size based on the number of players you have and the significance of the change you are testing.
Collect and Analyze Data: Monitor metrics like engagement, retention, and conversion rates. Tools like Amplitude Unity Analytics can be used here.
Make Data-Driven Decisions: Use the results of the A/B test to decide which variant of the game element performs better and increase the number of players on that variant using remote configuration, and make sure the trend continues.

Best Practices

Depending on your experience, you can be running multiple tests at the same time. However, it is best to start with one test at a time to avoid confusion.
When running multiple tests, you have to create stable, non-overlapping groups that have single and multiple treatment changes. You can do this in Joystick.
Ensure your sample size is large enough to get statistically significant results. Use a significance calculator.
Run the test long enough to capture meaningful data but avoid seasonality (week, month, etc) bias.

Conclusion

A/B testing is a cornerstone of user experience optimization in mobile games. By methodically testing and analyzing different game elements, developers can enhance the player experience, leading to increased engagement and revenue. Remember, the key to successful A/B testing is a well-thought-out strategy and a focus on data-driven decision-making.

Get Going

Use Joystick today to integrate A/B testing into your game development process and see the difference it can make in player satisfaction and your game’s success!