Clover Per Test Coverage for Automated Functional Test

Aryan March 11, 2015

My product code and Automation code are in totally different projects. We use TestNG for automation. 

Currently we instrument product code using clover and then run our automation to generate coverage report. This gives the overall number, but i would like to know if it is possible to generate per test coverage report in my case? 

If yes, how do i do it ? 

Thanks !

3 answers

1 accepted

0 votes
Answer accepted
Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 13, 2015

No, Build is create on different machine and deployed on different machine. Also Test code and Application code are built separately. Application is deployed on one system and tests are executed from different system.

In such case the scenario will look slightly different:

1) Instrument your application code and test code. I assume that each has only one database produced. Use "interval" or "threaded" flush policy.

  • two Clover databases are created, let's name them "clover-test.db" and "clover-app.db"

 

2) Copy application and test classes/JARs to your test machines. Copy "clover-test.db" and "clover-app.db" databases as well. You can put them in any location, just ensure that JVM processes will have a write access to directories.

 

3) Start your tests with the following JVM options:

-Dclover.distributed.coverage=numClients=1 -Dclover.server=true -Dclover.initstring=/path/to/clover-test.db
  • numClients=1 will ensure that test execution will pause until your application is started, see next point
  • clover.initstring points to a database on your test machine

 

4) Start your application with the following JVM options:

-Dclover.distributed.coverage=host=your.test.server -Dclover.initstring=/path/to/clover-app.db
  • host is a name or IP of your test server where tests are launched
  • clover.initstring points to a database on your test machine
  • as soon as application is started, instrumented code will connect to the "Clover Server" and test execution will proceed

 

5) As soon as tests are finished, you shall see coverage recording files for both databases on both test machines. Copy coverage recording files (clover-test.db* and clover-app.db*) back to your build machine - they will be used for report generation. Put these coverage recording files in the same directory(-ies) where databases are located.

 

6) Merge two Clover databases. You can use <clover-merge> (Ant), clover2:aggregate or clover2:merge (Maven) or CloverMerge (command line tool) for this purpose. Example for Ant:

&lt;clover-merge initString="clover-merged.db"&gt;
    &lt;cloverDb initString="clover-app.db"/&gt;
    &lt;cloverDb initString="clover-test.db"/&gt;
 &lt;/clover-merge&gt;

 

7) Use a merged database to generate HTML report. Example for Ant:

&lt;clover-report initstring="clover-merged.db"&gt;
   &lt;current outfile="clover-html"&gt;
      &lt;format type="html"/&gt;
   &lt;/current&gt;
&lt;/clover-report&gt;

 

Feel free to ask in case you get stuck in any point.

 

Cheers
Marek

Aryan March 15, 2015

Hi Marek, Followed above mentioned steps, but when i generate report it does not show me any test coverage. Test Code tab shows 0% coverage and I get below message on Test Results tab. "No test results No test results could be found. Please ensure that you have instrumented your unit tests correctly." My application and Automation projects are different and are instrumented separately to get two clover.db's I also noticed below error while generating the report: [clover-report] Ignoring unknown context filter "hashCode" [clover-report] Ignoring unknown context filter "equals" [clover-report] Ignoring unknown context filter "toString" Thanks

Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 15, 2015

Do you see both global coverage recording files (they're named like 'clover.dbhash_code' and 'clover.dbhash_code.1') and per-test coverage recording files (they're named like 'clover.dblonger_hash_code.s') after test execution? If not, then please check your application logs, the most probably there'll be a message saying that Clover was unable to find the database. If such message is present - please check if -Dclover.inistring property has been passed. In case you have coverage recording files - did you copy them back to your build server? If yes, then what messages do you see in <clover-merge> or <clover-report>? There shall be information about number of recording files processed. "Ignoring unknown context filter" - did you use the same set of context filters for both application code and test code? If yes, then does your <clover-report> use context filters which have been defined in <clover-setup>?

Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 15, 2015

4) Start your application with the following JVM options: -Dclover.distributed.coverage=host=your.test.server Did you use name of the machine where functional tests are running?

Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 15, 2015

Do you see any Clover's log messages related with a distributed coverage feature in both application and test logs? I'm asking about messages like: "Distributed coverage is enabled with: <settings>" "Starting distributed coverage service." "Starting distributed coverage client: <settings>" "Clover waiting for 1 remote clients to attach to this remote testing session. " "Could not initialise Distributed Coverage collection in Clover: <exception>" ...

Aryan March 15, 2015

- Coverage recording files are getting created on Application Machine as well as on Test Machine - On Test machine I am using below options to start the tests -Dclover.initstring=path/to/cloverdb/clover-test.db -Dclover.distributed.coverage=host=<hostname of my test machine>;numClients=1 -Dclover.server=true And I get following message : Distributed coverage is enabled with: name=clover.tcp.server;host=<my test machinename>;port=1198;timeout=5000;numClients=1;retryPeriod=1000 Starting distributed coverage service. Clover waiting for 1 remote clients to attach to this remote testing session. - On Application I am using options -Dclover.initstring=path\to\clover.db -Dclover.distributed.coverage=host=<Host name of TestMachine> - On Merge I get following output : [clover-merge] Merging database 1 of 2: clover-test.db [clover-merge] Merging database 2 of 2: clover.db [clover-merge] Writing merged database registry [clover-merge] Merge complete - After running the report on merged db I do get following message [clover-report] Done. Processed 459 packages in 87405ms (190ms per package).

Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 16, 2015

- Coverage recording files are getting created on Application Machine as well as on Test Machine Did you copy them back to a build machine? Could you please run <clover-merge> task with debug logging (ant -d) and find messages similar to these: "Processed NN recording files in MMms ..." "Processed NN per-test recording files in MMms" and compare number of coverage recording files from these log messages with a number of files present on a disk? Could you also check if there are any messages like these: "Ignoring coverage recording <file name> because no FileInfo supports its coverage range"

Aryan March 16, 2015

To merge / generate report, i have a separate folder in which i copy the all the files from test and application and then run the command to get report or merge the db's. I can see below lines when run in debug mode [clover-merge] Ignoring coverage recording FileBasedCoverageRecordingTranscript[header=Header[dbVersion=1426360953414, writeTimeStamp=1426367764985, format=0], coverageSum=17573, hitCounts.length=124683] because no FileInfo supports its coverage range [clover-merge] Processed 0 recording files in 11ms (- per recording) [clover-merge] Processed 0 per-test recording files in 0ms (- per recording)

Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 16, 2015
Aryan March 17, 2015

Now i don't see any "Ignoring coverage recording .." message. Also merge log shows below lines [clover-merge] Processed 3 recording files in 19ms (6ms per recording) [clover-merge] Processed 0 per-test recording files in 0ms (- per recording) But now when i generate the report, my automation test project is listed as part of Application Code, And Test Results shows "No Test Results" Why there is 0 per-test recording ? even after files are being processed.

Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 17, 2015

The most probably Clover did not classify your tests as test classes. Clover recognizes JUnit, TestNG, Spring and Instinct annotations as follows: https://confluence.atlassian.com/display/CLOVER/Clover+test+detection Please check whether naming convention of your classes and annotations used match any of these patterns. If not - you can "teach" Clover how your tests are named - see the <testsources> tag in <clover-setup> (https://confluence.atlassian.com/display/CLOVER/clover-setup). Note: in case you use <testsources> tag in <clover-setup>, you will have to use the same <testsources> tag in <clover-report> too.

Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 17, 2015

> Why there is 0 per-test recording ? even after files are being processed. See my comment above. You can also run build with debug logging (ant -d) and see output produced by <clover-setup>. In case it finds any test method/class, Clover will print messages like the following: [clover] Clover all over. Instrumented 2 files (1 package). [clover] 23 test methods detected.

Aryan March 17, 2015

When i instrument my AutomationProject, i do see following log message, [INFO] Clover all over. Instrumented 219 files (68 packages). [INFO] 884 test methods detected. do I still need to use <testsources> ?

Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 17, 2015

I don't know how many tests do you have. But number != 0 suggests that Clover recognized your test sources correctly. At least in the <clover-setup> task. Could you make a test and define <testsources> tag in <clover-report> to see what happens?

Aryan March 19, 2015

Hi Marek, Tried adding testsoruces tag but still no success. It shows the tests project as application code. Once again to brief the steps i am following are : 1. Instrument ApplicationCode, this generates clover.db 2. Instrument AutomationCode, this generates clover-tests.db 3. Copy clover.db and clover-tests.db to a shared n/w path. 4. Copy instrumented application jars to application machine and Start application with -Dclover.distributed.coverage=host=your.test.server -Dclover.initstring=/shared-network/path/to/clover.db 5. Copy the instrumented Automation jars to test machine and Run Automation Code from Test Machine with -Dclover.distributed.coverage=numClients=1 -Dclover.server=true -Dclover.initstring=/shared-network/path/to/clover-test.db NOTE : Both the clover.db and clover-test.db are in shared path and accessible to both test and application machine. 6. Once the test run is complete, i do see clover.db* and clover-tests.db* files in shared folder. 7. Then i copy all the files from shared folder to a new folder on build machine. 8. And run ant to merge both the db's. I am using following in my build.xml <project name="clover" default="clover.html" basedir="."> <taskdef resource="cloverlib.xml" classpath="${basedir}/clover-4.0.2.jar"/> <target name="clover.html" description="Generates clover report"> <clover-merge initString="clover-merged.db"> <cloverDb initString="clover-test.db"/> <cloverDb initString="clover.db"/> </clover-merge> </target> </project> 9. Then on the same folder i run clover report using below build.xml <project name="clover" default="clover.html" basedir="."> <taskdef resource="cloverlib.xml" classpath="${basedir}/clover-4.0.2.jar"/> <target name="clover.html" description="Generates clover report"> <!-- Generate HTML report --> <clover-report initstring="C:\Clover\target\clover-merged.db"> <current outfile="clover-html"> <format type="html"/> </current> </clover-report> </target> </project> It does generate report but shows no test results, instead my automation project is also listed in application.

Aryan March 19, 2015

Hi Marek, its working, seems like test classes should end with *Test*.java only then it adds it as test code instead of Application Code.

Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 19, 2015

Clover's test detector recognizes a class as the test one when any of the following conditions are met: * class has @org.testng.annotations.Test or @org.testng.annotations.ExpectedExceptions annotations * class has a javadoc @testng.test tag * class has 'test' in its name (case insensitive) * super class is TestCase or junit.framework.TestCase or has 'test' in its name This test detector can be overridden, as I mentioned before.

0 votes
sanjay_bhatta April 23, 2020

Need more help in modifying pom

0 votes
Marek Parfianowicz
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 12, 2015

Hi Aryan,

The exact answer depends on how your development environment is configured. Optimal Clover configuration can depend on a number of factors, such as:

  • build separation - do you compile application code and test code separately?
  • JVM separation - do you run test code in one JVM and an application runs in another JVM?
  • machine separation - do you build and test using the same physical machine? or maybe application runs on a dedicated test server?

 

 

I would say that in a "standard" configuration:

  • application under test is deployed to some application server (Tomcat / JBoss / ...)
  • compilation and tests are performed on the same physical machine
  • tests are launched from a build (e.g. "mvn test")

In such case you would have to do the following:

1) Instrument your application code and test code. Use absolute paths to a database in "initstring". Use "interval" or "threaded" flush policy.

  • thanks to this you will have no problems with running application/test code from a different working directory
  • thanks to this you will have two separate Clover databases - one for application and one for tests
  • thanks to this coverage data will be recorded on disk on the fly, without waiting for JVM termination

2) Start your application with the following JVM option:

-Dclover.distributed.coverage=ON
  • thanks to this the Distributed Per-Test Coverage feature will be enabled
  • application will start as "Clover Client", you may see connection attempts to the "Clover Server" - see below

3) Start your tests with the following JVM options:

-Dclover.distributed.coverage=ON -Dclover.server=true

4) As soon as tests are finished:

  • you shall see coverage recording files along with both databases

5) Merge two Clover databases. You can use <clover-merge> (Ant), clover2:aggregate or clover2:merge (Maven) or CloverMerge (command line tool) for this purpose. 

6) Use a merged database to generate HTML report.

  • if everything went well, you shall see not only global coverage but also per-test coverage in the report

 

You can find more hints here:

 

warning ad 3&4) You may encounter some issues related with starting "Clover Server" and "Clover Client". In case of time-outs or starting them in a wrong order you may not find per-test coverage in a report. In such case, please have a look at hints in the "Working with Distributed Applications" manual. Quite often it's necessary to start tests with "-Dclover.distributed.coverage=numClients=1" option.

 

Feel free to ask if you have any questions.

 

Cheers
Marek

kvallish April 9, 2018

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events