I had a quiet week as a lot of the team were away at a conference – so I set myself the task of speeding up the execution time of the tests. I was inspired by this post Fix Your Unstable Automated UI Tests – some of the things we already did but I thought there was a few things I could try.
First task was to see how bad the problem was and the results were pretty embarrassing.
On the 25th of March – the total number of minutes for our tests was 684, 2 months later it was 1172.
Thats over 8 hours increase in our tests at runtime (they run in parallel but the hours we use is limited per month as part of the contract). What happened?
Well, we had added more devices to the runs, so that ios 7 and ios 8 were always covered. This shouldn’t have made the runtime significantly longer as they were running in parallel but the wait time for the device to be ready, and the chances for a test to fail and need a retry also increased. We don’t want to reduce the number of devices – so what else can we do?
We had recently implemented deep linking so that smart banners could be shown on the responsive website which when clicked on opened the app at the same screen within the app. A nice side effect is that we now had a way to bypass the “place an ad via api, wait for ad to be indexed, launch app, search for ad, click on ad, check content” cycle. We could now skip a few steps and just place ad via api, open deep link, check content.
This should save us time in two ways. Not having to wait for the app to be indexed – an ad is visible right away if you know the id, via search you have to wait for indexing which takes longer. When performing the search, we had to use the keyboard to type the search term to find the ad which contained a long timestamp to make the ad title unique. The keyboard loading and then the actual typing on the keyboard letter by letter took several seconds.
We already checked the current login status and only logged in it was the wrong user currently logged in. However, sometimes this logic was forgotten and new users created and logged in for no real reason. Reviewing some tests found ways we could reduce the number of login/logouts going on during a test run.
We had recently changed the tests to add an image as part of placing an ad – instead of doing this once, we did it every test that placed an ad. We fixed this as testing it once was enough and it was a time consuming action.
After implementing all these improvements – the runtime of the tests increased by 107 minutes. What does this tell you? Mobile UI testing drives me crazy and its back to the drawing board.
Next things to try
One of the main suggestions was to have a dedicated backend – my only reluctance here is that we often catch bugs that are pushed to the backend which break the apps. We do have api tests but sometimes the apps just catch something that the api tests don’t. To see if the backend was responsible for a lot of failures – I will add to the test reporting an error type (BACKEND, CRASH, ASSERT_FAILURE) so I can track going forward where the biggest pain points are. For the rest of my time its going to be fixing flacky tests, finding smarter waits and hoping for some more bright ideas.