Frank Moyer, CTO, Kobiton, says that in taking your mobile testing strategy to the next level a little chaos goes a long way.
How do you know that your mobile testing strategy is mature?
The conventional answer to that question centers around factors like the abilities to automate a high percentage of mobile tests, run tests across a large number of devices and detect software defects early in the development lifecycle.
Those are all indeed important components of an effective mobile testing strategy. But I’d like to suggest that there is another critical element in mobile testing that teams often overlook: Chaos.
What I’m referring to here is the ability to inject chaotic conditions into mobile testing environments and assess how devices and applications respond. When you can do this, you achieve an even deeper level of mobile testing maturity.
Allow me to explain by discussing why performing chaos testing as part of mobile testing is important and how to get started doing it.
In the context of mobile software testing, chaos is conditions that might cause performance or reliability problems for real-world users.
Examples of chaos that are relevant for mobile tests include:
- Weak cellular connections, which could reduce device bandwidth or increase network latency
- Network connectivity interference from Bluetooth devices.
- Low battery power on mobile devices, which could decrease hardware performance
- Low proximity to wireless access points, causing devices to vacillate between using a Wi-Fi connection or relying on the cellular network for connectivity
These are all circumstances that occur regularly in the real world, as virtually anyone who owns a smartphone, tablet or any other type of mobile device knows all too well.
Unfortunately, not all mobile testing strategies address the possibility of circumstances like those described above.
Typically, teams run mobile tests under conditions that are as close as possible to ideal. They test either using mobile device simulators, where it’s simply impossible to evaluate what happens when physical infrastructure runs suboptimally because there is no physical infrastructure or they test on well-maintained, fully charged real devices.
For most testing needs, it makes good sense to test under best-case conditions. A stable testing environment removes many of the ancillary environment variables that could leave engineers wondering whether a failed mobile test resulted from a problem with the application they are testing or from an environmental factor beyond their control, such as a weak network connection. Thus, in many cases, you don’t want chaos to be part of your mobile testing workflows.
The benefits of chaos testing for mobile software tests
That said, there are situations where being able to inject chaos into some of your mobile tests is highly valuable.
The main reason why is simple: Your end-users are likely to encounter problems like poor network connectivity or erratic device behavior due to low battery power and running mobile tests under these conditions allows you to evaluate how your app will behave in a less-than-perfect environment. In turn, these tests can help developers identify ways to improve application performance under unusual conditions.
As a real-world example of why you might want to inject chaos into mobile tests, imagine you’re a retailer and you provide your customers with a mobile app that they can use in your stores to help locate and research items on the shelves. If your app is prone to slow response rates when many clients are connecting from the same local area – as they would be if they’re shoppers concentrated in a store – you’ll want to know about that risk so that your developers can mitigate it (which they might do by, for instance, writing logic that evenly distributes requests between concentrated sets of users). Otherwise, your app won’t reliably deliver the experience you promise to customers, and you’ll probably lose revenue.
As a second example, consider a mobile network carrier who promises 5G connectivity to customers. Chaos testing can help a company like this to determine whether devices with low battery power will struggle to achieve connectivity at 5G levels, for example, and how interference from other devices could impact connectivity.
The list of examples could go on, but the point is simple enough: Any business that needs to know how defects in mobile devices or networks could impact the user experience should add chaos testing to its mobile testing strategy.
To make chaos testing a part of your mobile testing operations, you need a test environment that allows you to create chaos on demand.
You can’t do this effectively when testing on cloud-based mobile devices because you can’t simulate problems like Bluetooth interference in a device cloud. Instead, you’ll need to set up your own so-called isolation chamber, which is a physical space that hosts a set of mobile devices that are disconnected from outside infrastructure. There, you can fully control all aspects of the environment in order to test under whichever chaotic conditions matter to you.
Not every mobile test needs to include chaos. Sometimes, you just want to perform compatibility tests to ensure that your apps work as expected across different devices, browsers and operating systems. Chaos testing is not helpful for that purpose.
But if you want to take your mobile testing strategy to the next level and ensure that you deliver an ideal user experience even under less-than-ideal conditions, a little chaos testing goes a long way. It provides insight into how mobile hardware and software interact in the real world, not just the close-to-perfect world of a standard mobile testing environment.
