Collecting coverage info for a patch in a human-readable form

Декабрь 12th, 2012 | Posted by elenst in MariaDB | Pensieve | Testing - (Комментарии отключены)

Usually we collect coverage information for a certain source file or directory. gcov/lcov and genhtml are good enough for that. But what I’m actually interested in is coverage for a patch, or, in other words, for bzr diff between the base tree and the new code. A patch might affect dozens of files, a few lines in each, and it’s pain to go through all HTML reports and compare them to the diff to find out which added/modified lines are hit and which aren’t. I’m pretty sure I’m going to be inventing a wheel here, but sometimes it’s easier and faster than find one that fits.

So, what do I need to do?
For now I will hope that a developer rebases before pushing a patch — that is, the code change is one or several revisions over the base tree, rather than a mix of patch-related revisions and base tree revisions. In the latter case, I’m in trouble.

I’ll need gcov and lcov to be installed and be on the PATH.

First, clean up the source tree and build with

Or, if it’s already built, run
lcov --directory <basedir> --zerocounters
to remove all leftovers from previous tests. I want clean data.

Then, run the tests. If it’s RQG combinations, make sure not to use --gcov option, as it cleans up after each run and collects the data separately, while I want the total coverage. If it’s MTR, also no need to use --gcov option, I’ll process what I need by myself.

I think that the tests also need to shut down the server properly (rather than kill it in our usual savage fashion). Not that I’m completely sure it’s important, but lets do it just to be safe.

When the tests are finished, run
lcov --quiet --directory <basedir> --capture --output-file <basedir>/

It will create a nice text file Well, maybe it’s not that nice to read, but it’s a Perl coder dream. And we have a description of its format. So it’s all good.

I also need the actual patch file. It can be either taken from commit mails, or produced by
bzr diff -r<last base revision>

So, I have a patch file and an lcov info file.
From the patch file, I need ‘+’ lines with their numbers and names of the source files; in the lcov file, I need to find coverage info for these code lines, using the source name and the line number. Probably the branch info too, while we are still there. It requires a bit of scripting, but it’s not nuclear physics, is it?

perl ~/mariadb-toolbox/scripts/ --help

The script produces an lcov summary of gcov data stored in the basedir
(or uses already existing lcov info file), and extracts the coverage data related
to the code patch or diff

Usage: perl <options>

--basedir=<path>: source code work tree
needed if there is no lcov info file or patch/diff file yet;
it is also used to remove the prefix from absolute paths in

--diff-file=<path>: a patch or bzr diff file;
if there is no such file yet, it will be generated by bzr diff

--prev-revno=<bzr revision>: a revision to compare with the work tree;
-2 by default (meaning last but one)

--lcov-info=<path>: a coverage summary file produced by lcov;
if there is no such file yet, it will be generated

--branch-info: include branch coverage info into the report
(FALSE by default)

--debug: script debug output

--help: print this help and exit

Make sure basedir is correct and is not a symlink! In other words, it should be the same as in, otherwise the result will be very confusing.

Example of the command line:
perl ~/mariadb-toolbox/scripts/ --basedir=/data/repo/bzr/5.5 --diff-file=/home/elenst/bzr/5.5/3733.dif --lcov-info=/home/elenst/bzr/5.5/ --branch-info 1>3733.coverage 2>missings

If we test not the tip of the tree, line numbers in the patch diff might be off, and it will create a totally wrong coverage report.
We need to adjust it. Here is one way to do it, it’s ugly, but it worked for me today:

1. run
bzr diff -c ${revno} > ${revno}.dif

It will produce the diff between the target revision and the previous one.

2. branch the tree locally, e.g.
bzr branch 5.5 5.5-temporary

3. go to the temporary branch and apply the created patch in the reverse mode:
patch -R -p0 < ${revno}.dif

Hopefully it will apply all right (with lines properly shifted). If it fails on test/result files, it can be ignored, we're only interested in the code.
If it worked, commit the temporary tree (don't push!)

4. bzr commit

Thus we'll have in 5.5 the tip of the tree (including the patch), and in 5.5-temporary the tip of the tree minus the patch. Now we just need to run the bzr diff.
Go back to the main tree.

5. bzr diff —old ../5.5-temporary > ${revno}.dif.adjusted

Now in ${revno}.dif.adjusted we should have the very same patch, but with the right line numbers. Use it instead of the original one in the script command line.

RQG: grammar keywords

Декабрь 11th, 2012 | Posted by elenst in Pensieve | Testing - (Комментарии отключены)

Since I’m totally lost in the github version of the RQG documentation, and can only find things there by pure chance, I will start adding here links to useful parts and pages I ran into; and if I don’t find something and have to figure it out by myself, I will put it here too. Maybe later, if it has enough contents, we will move it to AskMonty KnowledgeBase, but so far let it be here.

There used to be a nice list of keywords (e.g. _table, _field etc.) that can be used in a grammar file, but I can’t find it anymore. So, lets see what accepts and returns.

letter || _letter :
$_ = $prng->letter();

digit || _digit :
$_ = $prng->digit();

_table :
$tables = $executors->[0]->metaTables($last_database);
$last_table = $prng->arrayElement($tables);
$_ = '`'.$last_table.'`';

_field :
$fields = $executors->[0]->metaColumns($last_table, $last_database);
$_ = '`'.$prng->arrayElement($fields).'`';

_hex :
$_ = $prng->hex();

_cwd :
$_ = "'".$cwd."'";

_tmpnam || tmpnam || _tmpfile :
# Create a new temporary file name and record it for unlinking at the next statement
$generator->[GENERATOR_TMPNAM] = tmpdir()."gentest".abs($$).".tmp" if not defined $generator->[GENERATOR_TMPNAM];
$_ = "'".$generator->[GENERATOR_TMPNAM]."'";
$_ =~ s{\\}{\\\\}sgio if osWindows();   # Backslash-escape backslashes on Windows

_tmptable :
$_ = "tmptable".abs($$);

_unix_timestamp :
$_ = time();

_pid :
$_ = abs($$);

_thread_id :
$_ = $generator->threadId();

_thread_count :

_database || _db || _schema :
my $databases = $executors->[0]->metaSchemas();
$last_database = $prng->arrayElement($databases);
$_ = '`'.$last_database.'`';

_field_list :
my $fields = $executors->[0]->metaColumns($last_table, $last_database);
$_ = '`'.join('`,`', @$fields).'`';

_field_count :
my $fields = $executors->[0]->metaColumns($last_table, $last_database);
$_ = $#$fields + 1;

_field_next :
# Pick the next field that has not been picked recently and increment the $field_pos counter
my $fields = $executors->[0]->metaColumns($last_table, $last_database);
$_ = '`'.$fields->[$field_pos++ % $#$fields].'`';

_field_no_pk :
my $fields = $executors->[0]->metaColumnsTypeNot('primary',$last_table, $last_database);
$_ = '`'.$prng->arrayElement($fields).'`';

_field_indexed || _field_key :
my $fields_indexed = $executors->[0]->metaColumnsType('indexed',$last_table, $last_database);
$_ = '`'.$prng->arrayElement($fields_indexed).'`';

_field_unindexed || _field_nokey :
my $fields_unindexed = $executors->[0]->metaColumnsTypeNot('indexed',$last_table, $last_database);
$_ = '`'.$prng->arrayElement($fields_unindexed).'`';

_collation :
my $collations = $executors->[0]->metaCollations();
$_ = '_'.$prng->arrayElement($collations);

_collation_name :
my $collations = $executors->[0]->metaCollations();
$_ = $prng->arrayElement($collations);

_charset :
my $charsets = $executors->[0]->metaCharactersets();
$_ = '_'.$prng->arrayElement($charsets);

_charset_name :
my $charsets = $executors->[0]->metaCharactersets();
$_ = $prng->arrayElement($charsets);

_data :
$_ = $prng->file($cwd."/data");

# Additional logic, another time...

} elsif (
($field_type == FIELD_TYPE_NUMERIC) ||
($field_type == FIELD_TYPE_BLOB)
) {
$_ = $prng->fieldType($_);
} elsif ($field_type) {
$_ = $prng->fieldType($_);
if (
(substr($orig_item, -1) eq '`') ||
(substr($orig_item, 0, 2) eq "b'") ||
(substr($orig_item, 0, 2) eq '0x')
) {
# Do not quote, quotes are already present
} elsif (index($_, "'") > -1) {
$_ = '"'.$_.'"';
} else {
$_ = "'".$_."'";
} elsif (substr($_, 0, 1) eq '_') {
$item_nodash = substr($_, 1);
if ($prng->isFieldType($item_nodash)) {
$_ = "'".$prng->fieldType($item_nodash)."'";
if (index($_, "'") > -1) {
$_ = '"'.$_.'"';
} else {
$_ = "'".$_."'";

# If the grammar initially contained a ` , restore it. This allows
# The generation of constructs such as `table _digit` => `table 5`

if (
(substr($orig_item, -1) eq '`') &&
(index($_, '`') == -1)
) {
$_ = $_.'`';

$invariants{$orig_item} = $_ if $modifier eq 'invariant';

Test installation of MariaDB-Galera RPM packages from buildbot using yum

Ноябрь 30th, 2012 | Posted by elenst in Pensieve | Testing - (Комментарии отключены)

Buildbot only builds MariaDB-Galera-X.Y.Z-system-arch-server packages, and the rest is supposed to be taken from the standard repositories. In real life the package will live in the same repository, so everything should work smoothly. But I need to test it before it got there…

- start a nice clean VM, preferably with the standard MariaDB repository already configured, but if not, then add
/etc/yum.repos.d/MariaDB.repo with the contents like

# MariaDB 5.5 repository list - created 2012-11-29 22:41 UTC
name = MariaDB
baseurl =

or take the right link in the repository tool;

- download MariaDB-Galera-X.Y.Z-system-arch-server.rpm from the buildbot archive;

- run under root or sudo
yum install ./MariaDB-Galera-X.Y.Z-system-arch-server.rpm --nogpgcheck

If it works, also run

sudo yum install MariaDB-client

The server should start as is, but won’t be a real cluster node (as of 5.5.28 at least), because it is not pre-configured. Add Galera-related options to /etc/my.cnf file, e.g.


/etc/init.d/mysql start

and pray to your tester’s supersomething.

Hardcoded RQG constants

Июнь 16th, 2012 | Posted by self in Pensieve | Testing - (Комментарии отключены)

There are some important RQG constants that sometimes need to be changed to make particular tests more reasonable. Maybe some day I’ll make them configurable, but so far I just modify them in the code. Since it doesn’t happen every day, I tend to forget where they are located, so here they are:

For performance comparison test (new values can be different depending on the nature of the test):


# Configurable constants:
-use constant MIN_DURATION   => 0.02;  # (seconds) Queries with shorter execution times are not processed.
+use constant MIN_DURATION   => 0.2;  # (seconds) Queries with shorter execution times are not processed.
use constant MAX_DURATION   => 300;   # (seconds) Queries with longer execution times are not processed.
-use constant MIN_RATIO      => 1.5;   # Minimum speed-up or slow-down required in order to report a query
+use constant MIN_RATIO      => 5;   # Minimum speed-up or slow-down required in order to report a query
use constant MAX_ROWS       => 200;   # Skip query if initial execution resulted in more than so many rows.
use constant MIN_SAMPLES    => 0;     # Minimum number of execution time samples to do for each query per server.
use constant MAX_SAMPLES    => 0;     # Max number of execution time samples per query per server. Must be no less than MIN_SAMPLES.

For result comparison tests, where queries too often fail due to reaching max rows threshold:

-use constant MAX_ROWS_THRESHOLD => 50000;
+use constant MAX_ROWS_THRESHOLD => 500000;

For running experimental combinations, to avoid aborting a set of tests prematurely due to an error which RQG considers an environment failure, make STATUS_ENVIRONMENT_FAILURE not fatal:

-use constant STATUS_ENVIRONMENT_FAILURE => 110; # A failure in the environment or the grammar file
+use constant STATUS_ENVIRONMENT_FAILURE => 90; # A failure in the environment or the grammar file

For creating less tables than it’s usually done by default (but not bothering to create your own zz file just yet):

=== modified file 'lib/GenTest/App/'
-use constant GDS_DEFAULT_NAMES => ['A', 'B', 'C', 'D', 'E', 'AA', 'BB', 'CC', 'DD'];
+use constant GDS_DEFAULT_NAMES => ['A', 'B', 'C', 'D'];

Get rid of the Windows problem solution dialog

Июнь 4th, 2012 | Posted by self in Pensieve | Testing - (Комментарии отключены)

foo.exe has stopped working Windows can check online for a solution to the problem Check online for a solution and close the program Close the program Debug the program

This screen (a problem notification dialog, debug screen, or whatever it is called), is all cool and nice when you are actually debugging something, but is real pain when you are running a batch job, auto tests, or anything else where you’d be happy with just a windows dump and error logs from your application. But no, the screen pops up, everything stops and waits for your reaction.

Here is the solution if your driver is in Perl, as mine often are:

require Win32::API;
my $errfunc = Win32::API->new('kernel32', 'SetErrorMode', 'I', 'I');
my $initial_mode = $errfunc->Call(2);
$errfunc->Call($initial_mode | 2);

It’s not my invention, I found it long time ago on ActiveState lists. Works like a charm.