This post describes how to setup test code coverage report for an iOS project on Jenkins. As an example, here I’ll use the ConnectSDK-Lite project. This post assumes your Jenkins setup is based on my post: Jenkins in OSX guest in VirtualBox for iOS jobs – full setup guide, if something doesn’t work (e.g.,
xctool), please check that post first.
Configuration (not required anymore)
I initially setup an
xcconfig and a new configuration for code coverage based on this very helpful post: http://stackoverflow.com/questions/19136767/generate-gcda-files-with-xcode5-ios7-simulator-and-xctest/19386835#19386835. Then I found this tweet somewhere:
@richardbuckle There’s no more need for the `__gcov_flush()` workaround. Try eliminating the GTM coverage stuff.
And it worked with Xcode 6.1! No need to commit extra files. Let’s get to reports straight away.
NB: the commands below are tested with a library target, and may or may not work for an application target.
I found two ways of publishing test code coverage reports on Jenkins:
lcov: it generates really nice HTML pages, so you should use the HTML publisher plugin on Jenkins. A disadvantage is that you don’t get a diff between jobs.
gcovr: it converts
gcdafiles from Clang-LLVM into
coberturaXML format, so you can use the Cobertura plugin on Jenkins and get diffs between jobs.
I’ll describe both ways.
Gathering required information
The first build step to publish the reports is to generate coverage information. Here is a sample
Execute shell build step:
1 2 3 4 5 6 7 8
Important bits here are:
-scheme ConnectSDK -sdk iphonesimulator– specify your scheme that has tests; the
iphonesimulatorbecause these test can run on the iOS simulator only;
-IDECustomDerivedDataLocation="out/build_ccov"– specifies where all the build products should be placed; it simplifies our life in that we don’t have to search for the proper path in
GCC_GENERATE_TEST_COVERAGE_FILES=YES GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES– these flags generate coverage files;
lcov on Jenkins host:
Second, install the HTML Publisher plugin.
Then, use this
Execute shell build step as an example:
1 2 3 4 5 6
First, this step collects coverage info to a file, then removes info about iOS frameworks (
/Applications/*) and 3-rd party frameworks (stored in
Frameworks/ directory), because we’re interested in our code only, and generates an HTML report at the end.
In Post-Build Actions, add “Publish HTML reports” step. Add an entry with these values:
HTML directory to archive:
Test Code Coverage Report (lcov). You’ll get a link in the project’s left side bar to a nice HTML report.
I’ve found some hints for this part in this blog post: https://softnoise.wordpress.com/2010/11/14/ios-running-unit-tests-with-code-coverage-in-hudson/
(Since it’s a python script, you can also install it with
pip3 install gcovr. However, in my case of python 3.4 and gcovr 3.4, I needed to patch line 1742 in the
/usr/local/bin/gcovr script: in
for i in xrange(len(lines)): replace
Second, install the Cobertura Plugin.
And here is a sample shell script to use as a build step:
1 2 3 4 5
Only one command here, and an interesting part is the excluding regex:
.*#(?:ConnectSDKTests|Frameworks)#.*, here we’re ignoring tests themselves (
ConnectSDKTests) and 3-rd party frameworks (
Frameworks) directories. I found experimentally that you need to replaces slashes
/ in paths with hashes
Publish Cobertura Coverage Report post-build action with
Cobertura xml report pattern set to
Source Encoding to
UTF-8. You’ll get a link in the left side bar and also a code coverage trend graph and fully functional report on coverage of packages, files, classes, lines, and conditionals.
Oh, important stuff: this publisher step should be in the post-build actions step (you could use it in build steps using Any Build Step Plugin), otherwise you get a null pointer exception.
To get an idea what the result looks like, here are a couple of screenshots:
Project page with code coverage trend and links to reports:
Cobertura coverage report:
Lcov code coverage report index:
Seeing the real test code coverage report should motivate developers to improve the system, write more tests, and increase the coverage.
Hope it helps. I wish I had found such a detailed instruction before!