Reporting of ignored tests in Bamboo using NUnit

[vegarwe]

We generate NUnit reports in our test runs that are picked up by Bamboo. These nunit.xml files are generated by a custom script, since we don't have a test runner that will generate a report recognized by Bamboo. This is working just fine.

But, we do have tests that are not run every time for some reason or another. Maybe the test is not finished yet or there is a know bug that we can live with for now. In which case we want the test to be listed as skipped, or ignored or not-run. All of which are possible with the NUnit file format.

My question is can Bamboo pick up on different statuses than success or failure? Does it recognize that tests can be skipped and can we get it to report the skipped tests. What would the nunit.xml file look like with a skipped test in it?

[PiotrA · Accepted solution]

Hello vegarwe,

I'm not sure. I remember that in Bamboo 3.2 we've added some routines for handling 'skipped/inconclusive' test results, but I think we're currently not displaying in them in UI (if we encounter skipped test we just ignore it, that is). However, I just browsed the code for NUnit test result collecting, and it seems to me it has only stub implementation for the NUnit skipped tests:

https://bitbucket.org/atlassian/bamboo-dotnet-plugin/src/52c84b97f73f/src/main/java/com/atlassian/bamboo/plugin/dotnet/tests/nunit/NUnitXmlTestResultsParser.java?at=default#cl-218

So... I think the answer is "no", Bamboo can't pickup 'inconclusive' status for NUnit file format yet. Though I think enabling support for the third test status is easy(TM) task for a person having basic java programming skills (and access to the Bamboo sources... not sure if they are distributed with every license, or only the most expensive ones...). Would you like me to raise a relevant feature request on Bamboo issue tracker (http://jira.atlassian.com/browse/BAM) ?

[vegarwe]

Are there support for inconclusive status in any other reporting format (say JUnit for instance)? If not, then yes I appreciate it if you could raise a feature request.

[PiotrA]

It seems to me the JUnit is supported. At least the <skipped> element should be handled by Bamboo as 'not failure, not success'. Though I might be wrong - I didn't check that manually right now, I just looked at Bamboo source code.

[vegarwe]

I can't seem to get skipped to work with JUnit either. Here is a stripped down sample of the file I tried:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="1" failures="1" skips="2" tests="5" time="0.048" timestamp="2007-11-02T23:13:50">
  <testcase classname="net.cars.engine.ValveTest" name="checkSpeed" time="0.0080" />
  <testcase classname="net.cars.engine.ValveTest" name="moveDown" time="0.0010">
    <skipped />
  </testcase>
  <testcase classname="net.cars.engine.ValveTest" name="fjasefisken" time="0.0030">
    <skipped message="i am skipped." type="junit.framework.Skipped">junit.framework.Skipped: i am skipped.
    C:\exp\test.py: 8: Skipped: 1
</skipped>
  </testcase>
  <testcase classname="net.cars.engine.ValveTest" name="checkStatus" time="0.0030">
    <failure message="Plunger status invalid." type="junit.framework.AssertionFailedError">junit.framework.AssertionFailedError: Plunger status invalid
        at net.cars.engine.ValveTest.checkStatus(ValveTest.java:42)
</failure>
  </testcase>
  <testcase classname="net.cars.engine.ValveTest" name="moveUp" time="0.022">
    <error message="test timed out" type="java.lang.Exception">java.lang.Exception: test timed out
</error>
  </testcase>
  <system-out><![CDATA[Slowly moving up]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

What I see is the passing test, the failure and the error. The two skipped are just ignored. We are using an old version of Bamboo though, so maybe we need to upgrade. (Are running 3.4.3 now)

Anyway, Piotr: I would appriciate if you could raise that issue with NUnit.

[PiotrA]

https://jira.atlassian.com/browse/BAM-12340 raised.

cheers

[vegarwe]

Thanks for all the great help