How to uncover invisible changes in automated tests

KazuCocoa
Source Diving
Published in
5 min readMar 9, 2018

It's important to measure and uncover hidden things so that we understand them accurately.

In software development, performance is important to an application, but it is hard to visualise how 'fast' our app is. That is why it is crucial to have measurable metrics. When we make things measurable, we can uncover hidden performance bottlenecks.

Meanwhile, an application crash can be easily noticed, since it's very easy to detect (we can see with our own eyes), and we have crash logs for debugging. It's another story to detect it though.

In this article, I'd like to show my experience to measure and uncover such hidden thing and automate them. In this story, I pick up Android platform and mobile native application.

Before starting the story, let me introduce myself briefly. I've been working in Cookpad as a Software Engineer in Test/Quality. I had been working for Japan team before, and I moved to the International team around a half year ago. After joining the team, I've been working to improve automation stuff and follow incidents/bug management flows.

One day we got many user feedback which their device uses battery heavily

There's an incident I would like to share from 3 or 4 years ago.
We released the new version of our app as usual. However, we got some nasty feedback from our contact form and the Google Play Store. The feedback was about our app heat up users' devices. The number of complaints increased, and it escalates into a major problem for us. But none of our team members can figure out what is the root cause of the issue.

After our investigation, we found it happened if users used the version for a while. Were you able to imagine and predict if one application use battery power heavily and heat up your device? It was functionally correct, but it wasn't normal.

It was easy to reproduce manually, but it required kind of long scenarios. We could replicate it with Appium stably, not Espresso then.

Automate to catch up with the problem

I wrote It's important to measure and uncover hidden things so that we understand them accurately. at the top line. In this story, the hidden thing is associated with the device resources.

Lately, Android Studio provides us with an excellent resource profiling tool. Once we open the panel on the Android Studio and connect it to a target device, then we can get the device resources. But we only open the panel and check it when we are motivated to observe the device resources.

profiler

The tool is good at troubleshooting, but not for prevention. Can you foresee your current implementation will lead such problem accurately? It's important to find how to prevent such thing or increase an opportunity to catch up with the problem and integrate them into your development flow. It's the best if the way is automated.

Choose Appium

Espresso is very famous for Android developers to implement test cases for UI related since Google provides the library for Android officially. I believe Appium is also one of the great tools as well.

We've been using both Appium and Espresso directly for UI related automation since they have pros and cons each other. It depends on what you'd like to achieve. If I get an opportunity to explain our automation strategy, I'd love to show it though.

In that time, I would have liked to measure device resources against the release apk from outside of the test apk since I would have loved to follow application lifecycle on user devices. The lifecycle was from starting the app process and killing the process. We also hoped to conduct kind of long test scenarios stable. It's significant to make visible the differences between the previous result and the current result as well.

Thus, I chose Appium to emulate user actions and measure it automatically on our development line.

Get device resources

We have been able to get a snapshot of the device resources via adb dumpsys. The command is convenient for getting the device data from the connected devices. We don't need any additional implementations since Android OS provides us with the commands as a part of them.

I developed a simple library, named droid-monitor to measure and collect them. The library can generate graphs and export JSON formatted data. The library is written in Ruby, so you can integrate them to Ruby script and libraries such as RSpec.

We had used Ruby to implement Appium test scenarios and integrated the library into it. The scenario was simple. We just conduct Appium tests by running droid-monitor.

device resource graphs

The graphs below show a single day CPU and memory usage captured by droid-monitor. We can see the used resources fluctuate over time. We can also monitor network traffic as well.

cpu usage
memory usage

Recently Appium also provides similar data. Performance Testing of Android Apps is an example of it in Java. It's also helpful for you if you would like to measure performance through Appium.

Scenario automation

From the above, we already could automate and integrate them with our development flow as a test suite. After the release apk was generated, the test suite was conducted, and it created some test reports as visual graphs. With the help of these reports and visual graphs, it made it significantly easier for us to spot an anomaly when we compare it to the previous version.

Then, we could uncover something happened if we could see something different from the previous version. In the ideal world, I would like everything to be automated, but the comparison has to be done manually because we still couldn't figure out a good way for anomaly detection in the graphs.

Conclusion

As I described at the top of this article,

It's important to measure and uncover hidden things so that we understand them accurately.

is important. Recent Android Studio provides a rich resource monitor for developers and it's excellent. But prevention is better than cure and to achieve it. It's important to automate them since we're not a wizard. If you have such automation in your development flow, you can uncover such changes.

On the above, I show you the story using Appium. I became one of Appium's Technical Committee last year though. Cookpad loves OSS and you can see some of our activities for the OSS world, and we have two Ruby full-time committers. The culture has affected my OSS activities as well. You can see my history for OSS, and my experiences for the OSS comes from the culture.

I'll attend the AppiumConf2018, feel free to talk to me if you were there!

That's all.

Do you have any impressive automation stories?

Enjoy testing and development

Sign up to discover human stories that deepen your understanding of the world.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

Published in Source Diving

Developer blog from the folks at Cookpad. @cookpad_dev on twitter

Written by KazuCocoa

A Senior Software Engineer, Device Automation at @headspin_io. Maintaining @AppiumDevs. ex-Cookpad as a Test/QA/Automation Engineer, ex-ACCESS.

No responses yet

What are your thoughts?