Speeding up our App UI tests – to be continued

I had a quiet week as a lot of the team were away at a conference – so I set myself the task of speeding up the execution time of the tests. I was inspired by this post Fix Your Unstable Automated UI Tests – some of the things we already did but I thought there was a few things I could try.

First task was to see how bad the problem was and the results were pretty embarrassing.

On the 25th of March – the total number of minutes for our tests was 684, 2 months later it was 1172.

Thats over 8 hours increase in our tests at runtime (they run in parallel but the hours we use is limited per month as part of the contract). What happened?

Well, we had added more devices to the runs, so that ios 7 and ios 8 were always covered.  This shouldn’t have made the runtime significantly longer as they were running in parallel but the wait time for the device to be ready, and the chances for a test to fail and need a retry also increased. We don’t want to reduce the number of devices – so what else can we do?

We had recently implemented deep linking so that smart banners could be shown on the responsive website which when clicked on opened the app at the same screen within the app.  A nice side effect is that we now had a way to bypass the “place an ad via api, wait for ad to be indexed, launch app, search for ad, click on ad, check content” cycle.  We could now skip a few steps and just place ad via api, open deep link, check content.

This should save us time in two ways.  Not having to wait for the app to be indexed – an ad is visible right away if you know the id, via search you have to wait for indexing which takes longer.  When performing the search, we had to use the keyboard to type the search term to find the ad which contained a long timestamp to make the ad title unique.   The keyboard loading and then the actual typing on the keyboard letter by letter took several seconds.

We already checked the current login status and only logged in it was the wrong user currently logged in.  However, sometimes this logic was forgotten and new users created and logged in for no real reason.  Reviewing some tests found ways we could reduce the number of login/logouts going on during a test run.

We had recently changed the tests to add an image as part of placing an ad – instead of doing this once, we did it every test that placed an ad. We fixed this as testing it once was enough and it was a time consuming action.

After implementing all these improvements – the runtime of the tests increased by 107 minutes.  What does this tell you? Mobile UI testing drives me crazy and its back to the drawing board.

One of the main suggestions was to have a dedicated backend – my only reluctance here is that we often catch bugs that are pushed to the backend which break the apps.  We do have api tests but sometimes the apps just catch something that the api tests don’t.  To see if the backend was responsible for a lot of failures – I will add to the test reporting an error type (BACKEND, CRASH, ASSERT_FAILURE) so I can track going forward where the biggest pain points are. For the rest of my time its going to be fixing flacky tests, finding smarter waits and hoping for some more bright ideas.

Moving from desktop to mobile app testing

As traffic move from desktop to mobile, more testers will be asked to test for mobile. Mobile may be the future for ecommerce but for testing it will feel like a step backwards.

Automation tools and frameworks are 5 years behind desktop, and reliability of simulators/emulators are a big issue. Every new release of Xcode gives a huge headache as every simulator type has a different bug in it. Limitations on VM’s and hardware by apple make scaling up automated testing very difficult. Android emulators are also slow and unreliable.

Continuous deployment is not a reality, App store approval delays and manually triggered downloads means the release cycle can be fortnightly at best. This means that the level of risks you can take – fail fast – are more limited, there is no rollback option.

Backwards Compatability is something that may not have been an issue you had to consider before. We can’t stop people from using an app version – so messaging becomes important. Also users may not be able to get latest app if on an device where they cannot get the last supported version of an OS. Depending on your market you may want to keep these existing apps running for as long as you can.

Big Bang releases are sometimes unavoidable, like for a redesign due to new OS design standards, or a change in the navigation.  For a tester, this is the worst nightmare, big bang release to 100% of users with no rollback option.

and there are just new things to consider,

interruptions and interactions with other apps, using external memory (android), permissions, GPS, battery drain, rotation, gestures, native shortcuts, accessibility, and so it goes on.

It is a lot of fun though – and you don’t have to worry about Internet Explorer anymore

My painful lessons learned in Mobile Testing

Looking back at the last year, here are some of the important things I have learned about mobile testing.

1. Accept stories on devices only, don’t trust simulators/emulators

2. Exploratory test on devices that have other apps running, test devices can be too clean, have 50 games running in the background like your users do

3. Know your most used devices, Google Analytics can tell you this, our top 10 Android devices are all Samsung so we run most of our tests against Samsung.

4. Automate the regression tests, but only what you would bugfix if broken.

5. Test with every pre release of a new iOS version, and not just one regression test, every new feature being worked on too. Yes, we got caught out bad by ios8.

6. For Android, release to 1%, 5%, 20%, check your monitoring for increased crashes, or calls to your backend.

7. Find the balance between regular small releases, and annoying users with updates.  We try to make it 2-3 weeks with at least one new user feature to encourage the download.  Avoid new permissions on Android at all costs as adoption rate of new release falls from 80% to 15%.

8.  The most important test is that upgrading from one release to the next works.  The database migrations can go wrong and your app is unusable.  Repeat this on devices on the main supported os versions.

9.  Be aware of Accessibility, know how VoiceOver works, and test new designs with VoiceOver to ensure functionality  is not broken.

10. Use the device you users do, its very temping to get hold of the newest devices, we are all geeks after all, but for every day use, you want to be in an oldish phone with the newest OS.   This is how to find the most crashes.