Looking back at the last year, here are some of the important things I have learned about mobile testing.
1. Accept stories on devices only, don’t trust simulators/emulators
2. Exploratory test on devices that have other apps running, test devices can be too clean, have 50 games running in the background like your users do
3. Know your most used devices, Google Analytics can tell you this, our top 10 Android devices are all Samsung so we run most of our tests against Samsung.
4. Automate the regression tests, but only what you would bugfix if broken.
5. Test with every pre release of a new iOS version, and not just one regression test, every new feature being worked on too. Yes, we got caught out bad by ios8.
6. For Android, release to 1%, 5%, 20%, check your monitoring for increased crashes, or calls to your backend.
7. Find the balance between regular small releases, and annoying users with updates. We try to make it 2-3 weeks with at least one new user feature to encourage the download. Avoid new permissions on Android at all costs as adoption rate of new release falls from 80% to 15%.
8. The most important test is that upgrading from one release to the next works. The database migrations can go wrong and your app is unusable. Repeat this on devices on the main supported os versions.
9. Be aware of Accessibility, know how VoiceOver works, and test new designs with VoiceOver to ensure functionality is not broken.
10. Use the device you users do, its very temping to get hold of the newest devices, we are all geeks after all, but for every day use, you want to be in an oldish phone with the newest OS. This is how to find the most crashes.