Month: December 2014

Moving from desktop to mobile app testing

As traffic move from desktop to mobile, more testers will be asked to test for mobile. Mobile may be the future for ecommerce but for testing it will feel like a step backwards.

Automation tools and frameworks are 5 years behind desktop, and reliability of simulators/emulators are a big issue. Every new release of Xcode gives a huge headache as every simulator type has a different bug in it. Limitations on VM’s and hardware by apple make scaling up automated testing very difficult. Android emulators are also slow and unreliable.

Continuous deployment is not a reality, App store approval delays and manually triggered downloads means the release cycle can be fortnightly at best. This means that the level of risks you can take – fail fast – are more limited, there is no rollback option.

Backwards Compatability is something that may not have been an issue you had to consider before. We can’t stop people from using an app version – so messaging becomes important. Also users may not be able to get latest app if on an device where they cannot get the last supported version of an OS. Depending on your market you may want to keep these existing apps running for as long as you can.

Big Bang releases are sometimes unavoidable, like for a redesign due to new OS design standards, or a change in the navigation.  For a tester, this is the worst nightmare, big bang release to 100% of users with no rollback option.

and there are just new things to consider,

interruptions and interactions with other apps, using external memory (android), permissions, GPS, battery drain, rotation, gestures, native shortcuts, accessibility, and so it goes on.

It is a lot of fun though – and you don’t have to worry about Internet Explorer anymore

My painful lessons learned in Mobile Testing

Looking back at the last year, here are some of the important things I have learned about mobile testing.

1. Accept stories on devices only, don’t trust simulators/emulators

2. Exploratory test on devices that have other apps running, test devices can be too clean, have 50 games running in the background like your users do

3. Know your most used devices, Google Analytics can tell you this, our top 10 Android devices are all Samsung so we run most of our tests against Samsung.

4. Automate the regression tests, but only what you would bugfix if broken.

5. Test with every pre release of a new iOS version, and not just one regression test, every new feature being worked on too. Yes, we got caught out bad by ios8.

6. For Android, release to 1%, 5%, 20%, check your monitoring for increased crashes, or calls to your backend.

7. Find the balance between regular small releases, and annoying users with updates.  We try to make it 2-3 weeks with at least one new user feature to encourage the download.  Avoid new permissions on Android at all costs as adoption rate of new release falls from 80% to 15%.

8.  The most important test is that upgrading from one release to the next works.  The database migrations can go wrong and your app is unusable.  Repeat this on devices on the main supported os versions.

9.  Be aware of Accessibility, know how VoiceOver works, and test new designs with VoiceOver to ensure functionality  is not broken.

10. Use the device you users do, its very temping to get hold of the newest devices, we are all geeks after all, but for every day use, you want to be in an oldish phone with the newest OS.   This is how to find the most crashes.