Generate clover report from 2 test runs' recording data

[will zhang]

Hi,

I'm trying to generate combine report from 2 different testbeds on same packages: unit test and system test. What I found is the combined report is exactly the same as my unit test result, same coverage number, same hit number on lines.

Here is my setup to get the combine report:

1. run unit test using Maven, get clover.db and clover.db* record data.
2. Run system test with the instrumented binary. Got clover.db and clover.db* record data.
3. Copy over clover.db* record data into the same directory of step2.
4. Run command line command like this:
   sudo java com.atlassian.clover.reporters.html.HTMLReporter -i $STEP2_DB_DIR/clover.db -o /output_dir

What is wrong in my setup?

Thanks,

Will

[Marek Parfianowicz]

Please have a look at this article:

https://confluence.atlassian.com/display/CLOVERKB/Ignoring+coverage+recording+files

[will zhang]

Hi Marek,

When I merge 2 application's database into one, I hit this problem again.

Steps I did:

1. Compile application A and test it, get clover.db and clover.db* record files
2. Compile application B and test it, get clover.db and clover.db* record files
3. Merge A and B's clover.db and put it into the report dir
4. Copy A's clover.db* record files into report dir
5. Copy B's clover.db* record files into report dir
6. Run html generate command to generate report:
   1. java com.atlassian.clover.reporters.html.HtmlReporter -i clover.db -o clover_html

When I generate separate for each of the applications, I was able to see 65% percentage coverage for one application and 55% for another application. However, the combined report shows 15% only. 

Also, the covered element is even smaller than one application's report.

What could be the issue?

Thanks,

Will

[Marek Parfianowicz]

In case you merge two or more clover.db databases, please ensure that all of them contain distinct sets of classes. In other words, if the same class is repeated in two (or more) databases, then Clover will pick coverage for this class from only one database.

Such situation can happen when you perform compilation of the same source code twice, for instance:

unit testing:

mvn clean clover:setup test -> will produce one clover.db

integration testing:

mvn clean clover:setup verify -> will produce another clover.db, but the same application code was compiled

You wrote that:

Run system test with the instrumented binary. Got clover.db and clover.db* record data

"got clover.db" suggests that you compiled the code. Am I correct?

[Marek Parfianowicz]

You wrote that:

Merge A and B's clover.db and put it into the report dir

Copy A's clover.db* record files into report dir

Copy B's clover.db* record files into report dir

How did you merge A and B clover.db files? Did you use CloverMerge command for this purpose (or alternatively the clover:merge Maven goal or the <clover-merge> Ant task)?

I'm asking, because in case you used CloverMerge, then there is no need to copy recording files - coverage data is already merged in cloverMerge.db.

You wrote that:

java com.atlassian.clover.reporters.html.HtmlReporter -i clover.db -o clover_html

Is the "clover.db" the merged database? I'm asking because it's typically named "cloverMerge.db".

[Marek Parfianowicz]

In case your build setup requires compiling the same application code twice, then setup similar to the following could work:

1. run unit test:
      compile code and run unit tests with Clover
      clover.db and recording files will be created
2. run system test:
      copy clover.db created in step 1 to this build
      compile code and run system tests with Clover
      clover.db will be updated and recording files will be created
3. generate Clover report
      take updated clover.db from step 2,
      take recording files from step 1 and step 2 (put in the same directory as clover.db)
      run HtmlReporter
      note that there's no need to call CloverMerge task

The above is a general hint. I could tell you more if I could analyse details of your build setup. Feel free to create a ticket at support.atlassian.com. 

[will zhang]

Hi Marek,

Please check my inline reply. Thanks for your prompt reply!

"got clover.db" suggests that you compiled the code. Am I correct?

Will Zhang: Yes, that's correct. We use maven clover plugin in this case to compile the code. And we copy over the generate clover.db into test machine to run tests and collect coverage data.

1. How did you merge A and B clover.db files? Did you use CloverMerge command for this purpose (or alternatively the clover:merge Maven goal or the <clover-merge> Ant task)?

   I'm asking, because in case you used CloverMerge, then there is no need to copy recording files - coverage data is already merged in cloverMerge.db.

   You wrote that:

   java com.atlassian.clover.reporters.html.HtmlReporter -i clover.db -o clover_html

   Is the "clover.db" the merged database? I'm asking because it's typically named "cloverMerge.db".

   Will Zhang: We merged the database using CloverMerge command. And I just tried not copying over record file after merge, it still shows less coverage number, as 15.6%

1. Marek Parfianowicz [Atlassian]

   • about 2 hours ago

   In case your build setup requires compiling the same application code twice, then setup similar to the following could work:

   1. run unit test:
         compile code and run unit tests with Clover
         clover.db and recording files will be created
   2. run system test:
         copy clover.db created in step 1 to this build
         compile code and run system tests with Clover
         clover.db will be updated and recording files will be created
   3. generate Clover report
         take updated clover.db from step 2, 
         take recording files from step 1 and step 2 (put in the same directory as clover.db)
         run HtmlReporter
         note that there's no need to call CloverMerge task

   
   The above is a general hint. I could tell you more if I could analyse details of your build setup. Feel free to create a ticket at support.atlassian.com. 

   Will Zhang: I think I'm almost exactly the same setup as you mentioned above and able to get a merged report of my unit test report and system test.
   
   However, the problem now is that I'm having 2 different applications that has separate source code and I'm merging their report into one. And the merged report shows really low coverage number.

I just created a ticket here: https://support.atlassian.com/servicedesk/customer/portal/13/CLV-6373

Please note that my company already made the purchase, I'm waiting for the license. The SEN number on the ticket for now is my trail license.

Thanks,

Will

[will zhang]

By turning the debug info, I found following log while generating report, could this be the reason? If so, how can we fix it?

It might because the builds I used in those 2 testbed has different directory?

Read data for file "/tmp/clover/ccp/clover.dbs124v8_ityee58c.1": FileBasedCoverageRecordingTranscript[header=Header[dbVersion=1475751270364, writeTimeStamp=1475761955725, format=0], coverageSum=80111, hitCounts.length=34485]
Global recording file:
file=RecordingTranscripts.FileRef[datafile=/tmp/clover/ccp/clover.dbs124v8_ityee58c, testRecording=false, typedTestId=-1, runId=0, hash=1694832164, timestamp=1475761666044]
dbVersion=1475751270364 (Thu Oct 06 03:54:30 PDT 2016)
writeTimeStamp=1475761955725 (Thu Oct 06 06:52:35 PDT 2016)
Ignoring coverage recording FileBasedCoverageRecordingTranscript[header=Header[dbVersion=1475751270364, writeTimeStamp=1475761955725, format=0], coverageSum=80111, hitCounts.length=34485] because no FileInfo supports its coverage range