How can I get Test Result during executing WebDriver tests in Jenkins?

Question

I'm using the Play! WebDrive Module and the jobs run okay but when tests fail the Jenkins job shows 'All tests passed'. I'm trying to wire in test results but I keep running into troubles.

I tried using the SeleniumHQ plugin for Jenkins but that results in the error shown below. I use Firefox so I had the test results location set to {app}/test-result/FirefoxDriver/*.html

Publishing Selenium report...
ERROR: No Test Report Found
Build step 'Publish Selenium Report' changed build result to FAILURE
Finished: FAILURE

So I guess the output isn't really Selenium. I tried pulling in the Play! Plugin and that didn't give any good options. I did try out the Play! auto-test reports post-build step but that only resulted in showing the application log.

Finally, I tried using the Publish testing tools result report post-build step. This one is particularly frustrating because docs are so awful and Jenkins doesn't provide any debugging / trouble-shooting info that I can find.

Custom Tool Pattern: {app}/test-result/FirefoxDriver/*.html
Custom stylesheet: scripts/webdrive2junit.xsl

The XSL that I set up is below and then the error that I consistently get is below that. Is anyone getting test results from webdrive:test into the Jenkins job?

<?xml version="1.0" encoding="UTF-8"?>
<!--
 - Hopefully we won't use this for long.  I put it together to get some information
 - out of the webdrive:test results while I look for a better solution.
 -->
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="child::body">
    <xsl:variable name="test" select="normalize-space(descendant::div/h1)"/>
    <testsuite>
      <xsl:attribute name="name">
        <xsl:value-of select="$test"/>
      </xsl:attribute>
      <testcase>
        <xsl:attribute name="classname">
          <xsl:value-of select="$test"/>
        </xsl:attribute>
        <xsl:attribute name="name">
          <xsl:value-of select="descendant::div/div/table/thead/tr/th"/>
        </xsl:attribute>
        <xsl:if test="descendant::tr[@class='  status_failed']">
          <error />
        </xsl:if>
      </testcase>
    </testsuite>   
  </xsl:template>
</xsl:stylesheet>

Here's the error that I get in the job log.

[xUnit] [INFO] - [Custom Tool] - 1 test report file(s) were found with the pattern 'src/test-result/FirefoxDriver/*.html' relative to '/opt/ci/hudson/workspace/enterprise_another-play-test' for the testing framework 'Custom Tool'.
[xUnit] [ERROR] - Conversion error Error to convert - A file not found
ERROR: Publisher org.jenkinsci.plugins.xunit.XUnitPublisher aborted due to exception
hudson.util.IOException2: remote file operation failed: /opt/ci/hudson/workspace/enterprise_another-play-test at hudson.remoting.Channel@34b246:build-trunk-2
...
Caused by: com.thalesgroup.hudson.plugins.xunit.exception.XUnitException: Conversion error Error to convert - A file not found
    at com.thalesgroup.hudson.plugins.xunit.service.XUnitConversionService.convert(XUnitConversionService.java:89)
    at com.thalesgroup.hudson.plugins.xunit.service.XUnitTransformer.invoke(XUnitTransformer.java:135)
    ... 11 more
Caused by: com.thalesgroup.dtkit.util.converter.ConversionException: Error to convert - A file not found
    at com.thalesgroup.dtkit.util.converter.ConversionService.convert(ConversionService.java:369)
    at com.thalesgroup.dtkit.util.converter.ConversionService.convert(ConversionService.java:177)
    at com.thalesgroup.dtkit.util.converter.ConversionService.convert(ConversionService.java:114)
    at com.thalesgroup.dtkit.metrics.model.InputMetricXSL.convert(InputMetricXSL.java:196)
    at com.thalesgroup.dtkit.metrics.model.InputMetricXSL.convert(InputMetricXSL.java:202)
    at com.thalesgroup.hudson.plugins.xunit.service.XUnitConversionService.convertCustomInputMetric(XUnitConversionService.java:104)
    at com.thalesgroup.hudson.plugins.xunit.service.XUnitConversionService.convert(XUnitConversionService.java:78)
    ... 12 more
Caused by: org.xml.sax.SAXParseException: The element type "meta" must be terminated by the matching end-tag "</meta>".
    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)
    at com.thalesgroup.dtkit.util.converter.ConversionService.convert(ConversionService.java:324)
    ... 18 more
Finished: FAILURE

Answer 1

My current work-around is below. Any improvements would be appreciated. Since the Play plugin doesn't work well for me (because of the webdrive module is appears) my build steps are bash scripts.

# work-around for the XUnit Plugin trouble; feel free to Tidy up
find src/test-result/FirefoxDriver -name "*.html" -print0 | xargs -r0 -n 1 bash -c '
  testname=$(basename ${0%.html.*.html});
  xsltproc --html -o src/test-result/FirefoxDriver/TEST-$testname.xml scripts/webdrive2junit.xsl $0
'

I looked at the code for the XUnit Plugin to Jenkins (which I believe is behind the Publish testing tools result report post-build step) and I realize that it is not doing what I thought.