My product code and Automation code are in totally different projects. We use TestNG for automation.
Currently we instrument product code using clover and then run our automation to generate coverage report. This gives the overall number, but i would like to know if it is possible to generate per test coverage report in my case?
If yes, how do i do it ?
Thanks !
No, Build is create on different machine and deployed on different machine. Also Test code and Application code are built separately. Application is deployed on one system and tests are executed from different system.
In such case the scenario will look slightly different:
1) Instrument your application code and test code. I assume that each has only one database produced. Use "interval" or "threaded" flush policy.
2) Copy application and test classes/JARs to your test machines. Copy "clover-test.db" and "clover-app.db" databases as well. You can put them in any location, just ensure that JVM processes will have a write access to directories.
3) Start your tests with the following JVM options:
-Dclover.distributed.coverage=numClients=1 -Dclover.server=true -Dclover.initstring=/path/to/clover-test.db
4) Start your application with the following JVM options:
-Dclover.distributed.coverage=host=your.test.server -Dclover.initstring=/path/to/clover-app.db
5) As soon as tests are finished, you shall see coverage recording files for both databases on both test machines. Copy coverage recording files (clover-test.db* and clover-app.db*) back to your build machine - they will be used for report generation. Put these coverage recording files in the same directory(-ies) where databases are located.
6) Merge two Clover databases. You can use <clover-merge> (Ant), clover2:aggregate or clover2:merge (Maven) or CloverMerge (command line tool) for this purpose. Example for Ant:
<clover-merge initString="clover-merged.db"> <cloverDb initString="clover-app.db"/> <cloverDb initString="clover-test.db"/> </clover-merge>
7) Use a merged database to generate HTML report. Example for Ant:
<clover-report initstring="clover-merged.db"> <current outfile="clover-html"> <format type="html"/> </current> </clover-report>
Feel free to ask in case you get stuck in any point.
Cheers
Marek
Hi Marek, Followed above mentioned steps, but when i generate report it does not show me any test coverage. Test Code tab shows 0% coverage and I get below message on Test Results tab. "No test results No test results could be found. Please ensure that you have instrumented your unit tests correctly." My application and Automation projects are different and are instrumented separately to get two clover.db's I also noticed below error while generating the report: [clover-report] Ignoring unknown context filter "hashCode" [clover-report] Ignoring unknown context filter "equals" [clover-report] Ignoring unknown context filter "toString" Thanks
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Do you see both global coverage recording files (they're named like 'clover.dbhash_code' and 'clover.dbhash_code.1') and per-test coverage recording files (they're named like 'clover.dblonger_hash_code.s') after test execution? If not, then please check your application logs, the most probably there'll be a message saying that Clover was unable to find the database. If such message is present - please check if -Dclover.inistring property has been passed. In case you have coverage recording files - did you copy them back to your build server? If yes, then what messages do you see in <clover-merge> or <clover-report>? There shall be information about number of recording files processed. "Ignoring unknown context filter" - did you use the same set of context filters for both application code and test code? If yes, then does your <clover-report> use context filters which have been defined in <clover-setup>?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
4) Start your application with the following JVM options: -Dclover.distributed.coverage=host=your.test.server Did you use name of the machine where functional tests are running?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Do you see any Clover's log messages related with a distributed coverage feature in both application and test logs? I'm asking about messages like: "Distributed coverage is enabled with: <settings>" "Starting distributed coverage service." "Starting distributed coverage client: <settings>" "Clover waiting for 1 remote clients to attach to this remote testing session. " "Could not initialise Distributed Coverage collection in Clover: <exception>" ...
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
- Coverage recording files are getting created on Application Machine as well as on Test Machine - On Test machine I am using below options to start the tests -Dclover.initstring=path/to/cloverdb/clover-test.db -Dclover.distributed.coverage=host=<hostname of my test machine>;numClients=1 -Dclover.server=true And I get following message : Distributed coverage is enabled with: name=clover.tcp.server;host=<my test machinename>;port=1198;timeout=5000;numClients=1;retryPeriod=1000 Starting distributed coverage service. Clover waiting for 1 remote clients to attach to this remote testing session. - On Application I am using options -Dclover.initstring=path\to\clover.db -Dclover.distributed.coverage=host=<Host name of TestMachine> - On Merge I get following output : [clover-merge] Merging database 1 of 2: clover-test.db [clover-merge] Merging database 2 of 2: clover.db [clover-merge] Writing merged database registry [clover-merge] Merge complete - After running the report on merged db I do get following message [clover-report] Done. Processed 459 packages in 87405ms (190ms per package).
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
- Coverage recording files are getting created on Application Machine as well as on Test Machine Did you copy them back to a build machine? Could you please run <clover-merge> task with debug logging (ant -d) and find messages similar to these: "Processed NN recording files in MMms ..." "Processed NN per-test recording files in MMms" and compare number of coverage recording files from these log messages with a number of files present on a disk? Could you also check if there are any messages like these: "Ignoring coverage recording <file name> because no FileInfo supports its coverage range"
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
To merge / generate report, i have a separate folder in which i copy the all the files from test and application and then run the command to get report or merge the db's. I can see below lines when run in debug mode [clover-merge] Ignoring coverage recording FileBasedCoverageRecordingTranscript[header=Header[dbVersion=1426360953414, writeTimeStamp=1426367764985, format=0], coverageSum=17573, hitCounts.length=124683] because no FileInfo supports its coverage range [clover-merge] Processed 0 recording files in 11ms (- per recording) [clover-merge] Processed 0 per-test recording files in 0ms (- per recording)
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Now i don't see any "Ignoring coverage recording .." message. Also merge log shows below lines [clover-merge] Processed 3 recording files in 19ms (6ms per recording) [clover-merge] Processed 0 per-test recording files in 0ms (- per recording) But now when i generate the report, my automation test project is listed as part of Application Code, And Test Results shows "No Test Results" Why there is 0 per-test recording ? even after files are being processed.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
The most probably Clover did not classify your tests as test classes. Clover recognizes JUnit, TestNG, Spring and Instinct annotations as follows: https://confluence.atlassian.com/display/CLOVER/Clover+test+detection Please check whether naming convention of your classes and annotations used match any of these patterns. If not - you can "teach" Clover how your tests are named - see the <testsources> tag in <clover-setup> (https://confluence.atlassian.com/display/CLOVER/clover-setup). Note: in case you use <testsources> tag in <clover-setup>, you will have to use the same <testsources> tag in <clover-report> too.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
> Why there is 0 per-test recording ? even after files are being processed. See my comment above. You can also run build with debug logging (ant -d) and see output produced by <clover-setup>. In case it finds any test method/class, Clover will print messages like the following: [clover] Clover all over. Instrumented 2 files (1 package). [clover] 23 test methods detected.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
When i instrument my AutomationProject, i do see following log message, [INFO] Clover all over. Instrumented 219 files (68 packages). [INFO] 884 test methods detected. do I still need to use <testsources> ?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I don't know how many tests do you have. But number != 0 suggests that Clover recognized your test sources correctly. At least in the <clover-setup> task. Could you make a test and define <testsources> tag in <clover-report> to see what happens?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Marek, Tried adding testsoruces tag but still no success. It shows the tests project as application code. Once again to brief the steps i am following are : 1. Instrument ApplicationCode, this generates clover.db 2. Instrument AutomationCode, this generates clover-tests.db 3. Copy clover.db and clover-tests.db to a shared n/w path. 4. Copy instrumented application jars to application machine and Start application with -Dclover.distributed.coverage=host=your.test.server -Dclover.initstring=/shared-network/path/to/clover.db 5. Copy the instrumented Automation jars to test machine and Run Automation Code from Test Machine with -Dclover.distributed.coverage=numClients=1 -Dclover.server=true -Dclover.initstring=/shared-network/path/to/clover-test.db NOTE : Both the clover.db and clover-test.db are in shared path and accessible to both test and application machine. 6. Once the test run is complete, i do see clover.db* and clover-tests.db* files in shared folder. 7. Then i copy all the files from shared folder to a new folder on build machine. 8. And run ant to merge both the db's. I am using following in my build.xml <project name="clover" default="clover.html" basedir="."> <taskdef resource="cloverlib.xml" classpath="${basedir}/clover-4.0.2.jar"/> <target name="clover.html" description="Generates clover report"> <clover-merge initString="clover-merged.db"> <cloverDb initString="clover-test.db"/> <cloverDb initString="clover.db"/> </clover-merge> </target> </project> 9. Then on the same folder i run clover report using below build.xml <project name="clover" default="clover.html" basedir="."> <taskdef resource="cloverlib.xml" classpath="${basedir}/clover-4.0.2.jar"/> <target name="clover.html" description="Generates clover report"> <!-- Generate HTML report --> <clover-report initstring="C:\Clover\target\clover-merged.db"> <current outfile="clover-html"> <format type="html"/> </current> </clover-report> </target> </project> It does generate report but shows no test results, instead my automation project is also listed in application.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Clover's test detector recognizes a class as the test one when any of the following conditions are met: * class has @org.testng.annotations.Test or @org.testng.annotations.ExpectedExceptions annotations * class has a javadoc @testng.test tag * class has 'test' in its name (case insensitive) * super class is TestCase or junit.framework.TestCase or has 'test' in its name This test detector can be overridden, as I mentioned before.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Aryan,
The exact answer depends on how your development environment is configured. Optimal Clover configuration can depend on a number of factors, such as:
I would say that in a "standard" configuration:
In such case you would have to do the following:
1) Instrument your application code and test code. Use absolute paths to a database in "initstring". Use "interval" or "threaded" flush policy.
2) Start your application with the following JVM option:
-Dclover.distributed.coverage=ON
3) Start your tests with the following JVM options:
-Dclover.distributed.coverage=ON -Dclover.server=true
4) As soon as tests are finished:
5) Merge two Clover databases. You can use <clover-merge> (Ant), clover2:aggregate or clover2:merge (Maven) or CloverMerge (command line tool) for this purpose.
6) Use a merged database to generate HTML report.
You can find more hints here:
ad 3&4) You may encounter some issues related with starting "Clover Server" and "Clover Client". In case of time-outs or starting them in a wrong order you may not find per-test coverage in a report. In such case, please have a look at hints in the "Working with Distributed Applications" manual. Quite often it's necessary to start tests with "-Dclover.distributed.coverage=numClients=1" option.
Feel free to ask if you have any questions.
Cheers
Marek
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I have a slightly different problem (described at https://community.atlassian.com/t5/Clover-questions/How-do-I-get-functional-test-to-product-code-coverage-mapping/qaq-p/742178). Can someone help?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.