Collecting coverage info for a patch in a human-readable form

Декабрь 12th, 2012 | Posted by elenst in MariaDB | Pensieve | Testing - (Комментарии отключены)

Usually we collect coverage information for a certain source file or directory. gcov/lcov and genhtml are good enough for that. But what I’m actually interested in is coverage for a patch, or, in other words, for bzr diff between the base tree and the new code. A patch might affect dozens of files, a few lines in each, and it’s pain to go through all HTML reports and compare them to the diff to find out which added/modified lines are hit and which aren’t. I’m pretty sure I’m going to be inventing a wheel here, but sometimes it’s easier and faster than find one that fits.

So, what do I need to do?
For now I will hope that a developer rebases before pushing a patch — that is, the code change is one or several revisions over the base tree, rather than a mix of patch-related revisions and base tree revisions. In the latter case, I’m in trouble.

I’ll need gcov and lcov to be installed and be on the PATH.

First, clean up the source tree and build with

Or, if it’s already built, run
lcov --directory <basedir> --zerocounters
to remove all leftovers from previous tests. I want clean data.

Then, run the tests. If it’s RQG combinations, make sure not to use --gcov option, as it cleans up after each run and collects the data separately, while I want the total coverage. If it’s MTR, also no need to use --gcov option, I’ll process what I need by myself.

I think that the tests also need to shut down the server properly (rather than kill it in our usual savage fashion). Not that I’m completely sure it’s important, but lets do it just to be safe.

When the tests are finished, run
lcov --quiet --directory <basedir> --capture --output-file <basedir>/

It will create a nice text file Well, maybe it’s not that nice to read, but it’s a Perl coder dream. And we have a description of its format. So it’s all good.

I also need the actual patch file. It can be either taken from commit mails, or produced by
bzr diff -r<last base revision>

So, I have a patch file and an lcov info file.
From the patch file, I need ‘+’ lines with their numbers and names of the source files; in the lcov file, I need to find coverage info for these code lines, using the source name and the line number. Probably the branch info too, while we are still there. It requires a bit of scripting, but it’s not nuclear physics, is it?

perl ~/mariadb-toolbox/scripts/ --help

The script produces an lcov summary of gcov data stored in the basedir
(or uses already existing lcov info file), and extracts the coverage data related
to the code patch or diff

Usage: perl <options>

--basedir=<path>: source code work tree
needed if there is no lcov info file or patch/diff file yet;
it is also used to remove the prefix from absolute paths in

--diff-file=<path>: a patch or bzr diff file;
if there is no such file yet, it will be generated by bzr diff

--prev-revno=<bzr revision>: a revision to compare with the work tree;
-2 by default (meaning last but one)

--lcov-info=<path>: a coverage summary file produced by lcov;
if there is no such file yet, it will be generated

--branch-info: include branch coverage info into the report
(FALSE by default)

--debug: script debug output

--help: print this help and exit

Make sure basedir is correct and is not a symlink! In other words, it should be the same as in, otherwise the result will be very confusing.

Example of the command line:
perl ~/mariadb-toolbox/scripts/ --basedir=/data/repo/bzr/5.5 --diff-file=/home/elenst/bzr/5.5/3733.dif --lcov-info=/home/elenst/bzr/5.5/ --branch-info 1>3733.coverage 2>missings

If we test not the tip of the tree, line numbers in the patch diff might be off, and it will create a totally wrong coverage report.
We need to adjust it. Here is one way to do it, it’s ugly, but it worked for me today:

1. run
bzr diff -c ${revno} > ${revno}.dif

It will produce the diff between the target revision and the previous one.

2. branch the tree locally, e.g.
bzr branch 5.5 5.5-temporary

3. go to the temporary branch and apply the created patch in the reverse mode:
patch -R -p0 < ${revno}.dif Hopefully it will apply all right (with lines properly shifted). If it fails on test/result files, it can be ignored, we're only interested in the code. If it worked, commit the temporary tree (don't push!) 4. bzr commit Thus we'll have in 5.5 the tip of the tree (including the patch), and in 5.5-temporary the tip of the tree minus the patch. Now we just need to run the bzr diff.
Go back to the main tree.

5. bzr diff —old ../5.5-temporary > ${revno}.dif.adjusted

Now in ${revno}.dif.adjusted we should have the very same patch, but with the right line numbers. Use it instead of the original one in the script command line.