KISS 🇺🇦

Stop the war!

Stop the war in Ukraine! Fuck putin!

More information is at: https://war.ukraine.ua/.

There is a fund to support the Ukrainian Army: https://savelife.in.ua/en/donate/, and there is a special bank account that accepts funds in multiple currencies: https://bank.gov.ua/en/about/support-the-armed-forces. I donated to them. Please donate if you can!

Killer putin

Killer putin. Source: politico.eu.

Arrested putin

"It hasn't happened yet, but it will happen sooner or later. Beautiful photo, isn't it?" Source: twitter.

iOS Test Code Coverage report on Jenkins

| comments

This post describes how to setup test code coverage report for an iOS project on Jenkins. As an example, here I’ll use the ConnectSDK-Lite project. This post assumes your Jenkins setup is based on my post: Jenkins in OSX guest in VirtualBox for iOS jobs – full setup guide, if something doesn’t work (e.g., xctool), please check that post first.

Configuration (not required anymore)

I initially setup an xcconfig and a new configuration for code coverage based on this very helpful post: http://stackoverflow.com/questions/19136767/generate-gcda-files-with-xcode5-ios7-simulator-and-xctest/19386835#19386835. Then I found this tweet somewhere:

@richardbuckle There’s no more need for the `__gcov_flush()` workaround. Try eliminating the GTM coverage stuff.

And it worked with Xcode 6.1! No need to commit extra files. Let’s get to reports straight away.

Reports

NB: the commands below are tested with a library target, and may or may not work for an application target.

I found two ways of publishing test code coverage reports on Jenkins:

  • lcov: it generates really nice HTML pages, so you should use the HTML publisher plugin on Jenkins. A disadvantage is that you don’t get a diff between jobs.
  • gcovr: it converts gcda files from Clang-LLVM into cobertura XML format, so you can use the Cobertura plugin on Jenkins and get diffs between jobs.

I’ll describe both ways.

Gathering required information

The first build step to publish the reports is to generate coverage information. Here is a sample Execute shell build step:

1
2
3
4
5
6
7
8
# cleanup the output directory first
rm -rf "out/build_ccov"

# run tests with code coverage
xctool -scheme ConnectSDK -sdk iphonesimulator -configuration Debug \
    -reporter plain -IDECustomDerivedDataLocation="out/build_ccov" \
    GCC_GENERATE_TEST_COVERAGE_FILES=YES GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES \
    clean test

Important bits here are:

  • -scheme ConnectSDK -sdk iphonesimulator – specify your scheme that has tests; the sdk must be iphonesimulator because these test can run on the iOS simulator only;
  • -IDECustomDerivedDataLocation="out/build_ccov" – specifies where all the build products should be placed; it simplifies our life in that we don’t have to search for the proper path in ~/Library/Developer/Xcode/DerivedData/;
  • GCC_GENERATE_TEST_COVERAGE_FILES=YES GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES – these flags generate coverage files;

lcov

First, install lcov on Jenkins host:

1
brew install lcov

Second, install the HTML Publisher plugin.

Then, use this Execute shell build step as an example:

1
2
3
4
5
6
# generate lcov report
lcov --capture --directory "out/build_ccov/ConnectSDK/Build/Intermediates/ConnectSDK.build/"\
"Debug-iphonesimulator/ConnectSDK.build/Objects-normal/i386/" --output-file "out/coverage.info"
lcov --remove "out/coverage.info" '/Applications/*' --remove "out/coverage.info" 'Frameworks/*' \
    --output "out/coverage-csdk.info"
genhtml "out/coverage-csdk.info" --output-directory "out/coverage_report"

First, this step collects coverage info to a file, then removes info about iOS frameworks (/Applications/*) and 3-rd party frameworks (stored in Frameworks/ directory), because we’re interested in our code only, and generates an HTML report at the end.

In Post-Build Actions, add “Publish HTML reports” step. Add an entry with these values: HTML directory to archive: out/coverage_report/, Index page: index.html, Report title: Test Code Coverage Report (lcov). You’ll get a link in the project’s left side bar to a nice HTML report.

gcovr

I’ve found some hints for this part in this blog post: https://softnoise.wordpress.com/2010/11/14/ios-running-unit-tests-with-code-coverage-in-hudson/

First, install gcovr:

1
brew install gcovr

(Since it’s a python script, you can also install it with pip3: pip3 install gcovr. However, in my case of python 3.4 and gcovr 3.4, I needed to patch line 1742 in the /usr/local/bin/gcovr script: in for i in xrange(len(lines)): replace xrange with range (https://github.com/gcovr/gcovr/pull/89))

Second, install the Cobertura Plugin.

And here is a sample shell script to use as a build step:

1
2
3
4
5
# generate gcovr+cobertura report
gcovr --object-directory="out/build_ccov/ConnectSDK/Build/Intermediates/ConnectSDK.build/"\
"Debug-iphonesimulator/ConnectSDK.build/Objects-normal/i386/" \
    --root=. --xml-pretty --gcov-exclude='.*#(?:ConnectSDKTests|Frameworks)#.*' \
    --print-summary --output="out/coverage.xml"

Only one command here, and an interesting part is the excluding regex: .*#(?:ConnectSDKTests|Frameworks)#.*, here we’re ignoring tests themselves (ConnectSDKTests) and 3-rd party frameworks (Frameworks) directories. I found experimentally that you need to replaces slashes / in paths with hashes #.

Add a Publish Cobertura Coverage Report post-build action with Cobertura xml report pattern set to out/coverage.xml, set Source Encoding to UTF-8. You’ll get a link in the left side bar and also a code coverage trend graph and fully functional report on coverage of packages, files, classes, lines, and conditionals.

Oh, important stuff: this publisher step should be in the post-build actions step (you could use it in build steps using Any Build Step Plugin), otherwise you get a null pointer exception.

Screenshots

To get an idea what the result looks like, here are a couple of screenshots:

  • Project page with code coverage trend and links to reports:

  • Cobertura coverage report:

  • Lcov code coverage report index:

Seeing the real test code coverage report should motivate developers to improve the system, write more tests, and increase the coverage.

Hope it helps. I wish I had found such a detailed instruction before!

Comments