Curiousily

How to get Code Coverage during Manual Testing for Android App

02.03.2017 — Android, Java, Testing, Code Coverage — 2 min read

This one should be a quick one, hopefully! Recently, I’ve been trying to gather coverage data of an app during manual testing. Imagine exercising the app using MonkeyRunner, Ui Automator or some smart AI agent (more about that in later posts).

Many tools (and papers) describe that these unicorns exist and are easy to use. Yet, my experience suggests that this is not the case. The few I’ve tried are: BBoxTester (never got it to instrument a custom app), SwiftHand (same as BBoxTester) and various hacks using Emma. None of it works (but that might be just me).

How to do it?

Meet JaCoCo, the tool that really works (yes, there is a catch!). Normally, JaCoCo is used for code coverage when you are executing tests. However, a few tweaks will do the trick.

To “install” JaCoCo in your Android project open the build.gradle file within the app folder and add the following at the top level:

1def coverageSourceDirs = [
2        '../app/src/main/java'
3]
4
5jacoco{
6    toolVersion = "0.7.6.201602180812" // try a newer version if you can
7}

Next, let’s define a task (in the same file) which will generate HTML report for the code coverage achieved during the testing:

1task jacocoTestReport(type: JacocoReport) {
2    group = "Reporting"
3    description = "Generate Jacoco coverage reports after running tests."
4    reports {
5        xml.enabled = true
6        html.enabled = true
7    }
8    classDirectories = fileTree(
9            dir: './build/intermediates/classes/debug',
10            excludes: ['**/R*.class',
11                       '**/*$InjectAdapter.class',
12                       '**/*$ModuleAdapter.class',
13                       '**/*$ViewInjector*.class'
14            ])
15    sourceDirectories = files(coverageSourceDirs)
16    executionData = files("$buildDir/outputs/code-coverage/connected/coverage.exec")
17    doFirst {
18        new File("$buildDir/intermediates/classes/").eachFileRecurse { file ->
19            if (file.name.contains('$$')) {
20                file.renameTo(file.path.replace('$$', '$'))
21            }
22        }
23    }
24}

Great! JaCoCo is installed and should be working nicely! Add the following within android -> buildTypes (in the same file):

1debug {
2    testCoverageEnabled = true
3}

Next, add resources directory to app -> src -> main. Add jacoco-agent.properties file to that folder. The file should contain:

1destfile=/storage/sdcard/coverage.exec

The coverage data will be recorded at the device. So, we need a permission to write there. Add the following to your AndroidManifest.xml file:

1<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

On Android 6+ you should request this permission during runtime. Here is a sample:

1public static void verifyStoragePermissions(Activity activity) {
2        // Check if we have read or write permission
3        int writePermission = ActivityCompat.checkSelfPermission(activity,
4                Manifest.permission.WRITE_EXTERNAL_STORAGE);
5        int readPermission = ActivityCompat.checkSelfPermission(activity,
6                Manifest.permission.READ_EXTERNAL_STORAGE);
7
8        if (writePermission != PackageManager.PERMISSION_GRANTED ||
9            readPermission != PackageManager.PERMISSION_GRANTED) {
10            // We don't have permission so prompt the user
11            ActivityCompat.requestPermissions(
12                    activity,
13                    PERMISSIONS_STORAGE,
14                    REQUEST_EXTERNAL_STORAGE
15            );
16        }
17}

Make sure you call this method and obtain the permission before starting/stopping any tests. Next, let’s define a helper class which will generate the report file:

1import android.os.Environment;
2import android.util.Log;
3
4import java.io.File;
5import java.lang.reflect.Method;
6
7public class JacocoReportGenerator {
8    static void generateCoverageReport() {
9        String TAG = "jacoco";
10        // use reflection to call emma dump coverage method, to avoid
11        // always statically compiling against emma jar
12        Log.d("StorageSt", Environment.getExternalStorageState());
13        String coverageFilePath = Environment.getExternalStorageDirectory() + File.separator + "coverage.exec";
14        File coverageFile = new File(coverageFilePath);
15        try {
16            coverageFile.createNewFile();
17            Class<?> emmaRTClass = Class.forName("com.vladium.emma.rt.RT");
18            Method dumpCoverageMethod = emmaRTClass.getMethod("dumpCoverageData",
19                    coverageFile.getClass(), boolean.class, boolean.class);
20
21            dumpCoverageMethod.invoke(null, coverageFile, false, false);
22            Log.e(TAG, "generateCoverageReport: ok");
23        } catch (Exception  e) {
24            throw new RuntimeException("Is emma jar on classpath?", e)
25        }
26    }
27}

Now, it is up to decide where the call to generateCoverageReport() should happen. To test it out, put it in some onPause() method of an Activity.

Run the app and do your testing. When you are done execute the following adb command:

1adb pull /sdcard/coverage.exec app/build/outputs/code-coverage/connected/coverage.exec

Make sure you execute it in the root folder of your project and all folders are already created. Finally, generate the report using the task we created:

1./gradlew jacocoTestReport

Open the index.html file in app/build/reports/jacoco/jacocoTestReport/html/ folder. Now go grab a cookie and enjoy the victory!

Is this the best solution?

No! But it is a start. There is no sure way to receive an event when the app is closing/finishing and save the report then. However, some magic tools like ProbeDroid might offer ways to alleviate that pain. Please, write in the comments below if other, easier, solutions exist!

UPDATE: A (much) better approach is described in my next blog post. It appears to be much faster, as well!

Want to be a Machine Learning expert?

Join the weekly newsletter on Data Science, Deep Learning and Machine Learning in your inbox, curated by me! Chosen by 10,000+ Machine Learning practitioners. (There might be some exclusive content, too!)

You'll never get spam from me

Hacker's Guide to Neural Networks in JavaScript

Build Machine Learning models (especially Deep Neural Networks) that you can easily integrate with existing or new web apps. Think of your ReactJs, Vue, or Angular app enhanced with the power of Machine Learning models.

Get SH*T Done with PyTorch

Learn how to solve real-world problems with Deep Learning models (NLP, Computer Vision, and Time Series). Go from prototyping to deployment with PyTorch and Python!

Hacker's Guide to Machine Learning with Python

This book brings the fundamentals of Machine Learning to you, using tools and techniques used to solve real-world problems in Computer Vision, Natural Language Processing, and Time Series analysis. The skills taught in this book will lay the foundation for you to advance your journey to Machine Learning Mastery!

Hands-On Machine Learning from Scratch

This book will guide you on your journey to deeper Machine Learning understanding by developing algorithms in Python from scratch! Learn why and when Machine learning is the right tool for the job and how to improve low performing models!