Generating HTML reports from TRX using XSLT

Visual studio Team System 2008 can give you all kinds of reports for different kinds of tests you ran. This is possible only when you have a TFS - Team Foundation Server installed and configured. What if you do not have a TFS set up? In that case, you have to have VSTS in order to be able to open the test result files. VSTS uses a propritory format for results and it has extension .trx. If you look inside any .trx, you will find it to be an XML file. If you have VSTS on your computer, you can easily open this .trx file and do all sorts of filtering to analyze the results. But not all on your team would have VSTS installed, most of them would be your bosses who would be keep to see these results.

Currently, there is no easy, in-built solution to this problem and it is often left you testers (like you and me) to figure our how these reports should look so they can be published and used by everybody on the project - from devs to managers.

There are several ways to solve this -
1. Using XSLT, which I am going to explain in this article
2. Using ASP.NET,
3. Using C#

I will talk about last 2 methods in other articles.

To keep this article close to the subject, I am skipping explaination of what is XSLT and how to use it? You can find detailed help and instructions at http://www.w3schools.com/xsl (I used this site to learn it myself, and found extremely helpful). I am also skipping what is XML, you can always study it at http://www.w3schools.com/xml

Lets take a look inside a .trx file generated after a sample test run of 2 units tests and 2 manual tests.

<?xml version="1.0" encoding="UTF-8"?>
<TestRun>
<TestRunConfiguration name="Local Test Run" id="17cd0818-bbb7-4954-82ed-84261dd1282e">
<Description>This is a default test run configuration for a local test run.</Description>
<Deployment runDeploymentRoot="C:\TestBox 2009-05-15 14_32_12" />
<TestTypeSpecific />
</TestRunConfiguration>
<ResultSummary outcome="Failed">
<Counters total="4" executed="4" passed="3" error="0" failed="1" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
</ResultSummary>
<Times creation="2009-05-15T14:32:12.6406250-07:00" queuing="2009-05-15T14:32:17.3281250-07:00" start="2009-05-15T14:32:17.8281250-07:00" finish="2009-05-15T14:32:28.6562500-07:00" />
<TestDefinitions>
<UnitTest name="TestMethod1" storage="c:\testproject2\testproject2\bin\debug\testproject2.dll" id="486e3d31-ff0f-b600-efc1-68daab05c957">
<Css projectStructure="" iteration="" />
<Description>Unit test 1 : verify that...</Description>
<Owners>
<Owner name="" />
</Owners>
<Execution id="cbd9e486-4201-46b5-a3d2-6d4b0287b849" />
<TestMethod codeBase="c:\TestProject2\TestProject2\bin\Debug\TestProject2.dll" adapterTypeName="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" className="TestProject2.UnitTest2, TestProject2" name="TestMethod1" />
</UnitTest>
<UnitTest name="TestMethod1" storage="c:\testproject2\testproject2\bin\debug\testproject2.dll" id="b96f2172-51f7-3767-573a-01ebaf3120be">
<Css projectStructure="" iteration="" />
<Owners>
<Owner name="" />
</Owners>
<Execution id="548c261d-a723-4201-97ec-147c18eed999" />
<TestMethod codeBase="c:\TestProject2\TestProject2\bin\Debug\TestProject2.dll" adapterTypeName="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" className="TestProject2.UnitTest1, TestProject2" name="TestMethod1" />
</UnitTest>
<ManualTest name="manualtest2" storage="c:\testproject2\testproject2\manualtest2.mht" id="a5e8cb5d-ddf4-4cf0-9529-26ec4ed90311">
<Css projectStructure="" iteration="" />
<Description>Manual test 2 : Verify that...</Description>
<Owners>
<Owner name="" />
</Owners>
<Execution id="5ae7f29b-a09b-4878-a024-5a9db0cb7a6b" />
</ManualTest>
<ManualTest name="ManualTest1" storage="c:\testproject2\testproject2\manualtest1.mht" id="0bd2b5ae-2e79-44ff-bb77-225b4096011e">
<Css projectStructure="" iteration="" />
<Description>Manual test 1 : verify that...</Description>
<Owners>
<Owner name="" />
</Owners>
<Execution id="c4f31131-55d5-49cc-af2e-996dcec14235" />
</ManualTest>
</TestDefinitions>
<TestLists>
<TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
<TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
</TestLists>
<TestEntries>
<TestEntry testId="0bd2b5ae-2e79-44ff-bb77-225b4096011e" executionId="c4f31131-55d5-49cc-af2e-996dcec14235" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
<TestEntry testId="486e3d31-ff0f-b600-efc1-68daab05c957" executionId="cbd9e486-4201-46b5-a3d2-6d4b0287b849" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
<TestEntry testId="a5e8cb5d-ddf4-4cf0-9529-26ec4ed90311" executionId="5ae7f29b-a09b-4878-a024-5a9db0cb7a6b" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
<TestEntry testId="b96f2172-51f7-3767-573a-01ebaf3120be" executionId="548c261d-a723-4201-97ec-147c18eed999" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
</TestEntries>
<Results>
<ManualTestResult executionId="c4f31131-55d5-49cc-af2e-996dcec14235" testId="0bd2b5ae-2e79-44ff-bb77-225b4096011e" testName="ManualTest1" computerName="TestBox" duration="00:00:06.1909681" startTime="2009-05-15T14:32:17.9531250-07:00" endTime="2009-05-15T14:32:24.1406250-07:00" testType="7a24f8ff-85da-d664-ef44-83ff4743dae6" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" testFile="c:\TestProject2\TestResults\admin_TestBox 2009-05-15 14_32_12\Out\manualtest1.mht">
</ManualTestResult>
<UnitTestResult executionId="cbd9e486-4201-46b5-a3d2-6d4b0287b849" testId="486e3d31-ff0f-b600-efc1-68daab05c957" testName="TestMethod1" computerName="TestBox" duration="00:00:00.0220444" startTime="2009-05-15T14:32:24.2343750-07:00" endTime="2009-05-15T14:32:24.6093750-07:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d">
<Output>
</Output>
</UnitTestResult>
<ManualTestResult executionId="5ae7f29b-a09b-4878-a024-5a9db0cb7a6b" testId="a5e8cb5d-ddf4-4cf0-9529-26ec4ed90311" testName="manualtest2" computerName="TestBox" duration="00:00:03.8898731" startTime="2009-05-15T14:32:24.6250000-07:00" endTime="2009-05-15T14:32:28.5000000-07:00" testType="7a24f8ff-85da-d664-ef44-83ff4743dae6" outcome="Failed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" testFile="c:\TestProject2\TestResults\admin_TestBox 2009-05-15 14_32_12\Out\manualtest2.mht">
</ManualTestResult>
<UnitTestResult executionId="548c261d-a723-4201-97ec-147c18eed999" testId="b96f2172-51f7-3767-573a-01ebaf3120be" testName="TestMethod1" computerName="TestBox" duration="00:00:00.0006224" startTime="2009-05-15T14:32:28.5156250-07:00" endTime="2009-05-15T14:32:28.5312500-07:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d">
<Output>
</Output>
</UnitTestResult>
</Results>
</TestRun>

I know, it is a lot of data, just after execution of 4 test cases. Just imagine going through this file after execution of hundreds if not thousands of test cases.

But we do not need all the data given in trx file to generate our simple report which can at least show us what happened, how many tests pass, how many failed, which passed, which failed and a summary of the overall run. We will need to work with 2 main nodes of the file.
1. ResultSummary (found near the top of the file) and
2. Results (found near the end of the file)
Other nodes can be used based on your reporting requirements.

Here is how the XSLT is layed out

<xsl:stylesheet version="1.0" xmlns:xsl="<a href="http://www.w3.org/1999/XSL/Transform">http://www.w3.org/1999/XSL/Transform</a>">
<xsl:template match="/">
<html>
<link href="/Style.css" rel="stylesheet" type="text/css" />
<body>
<h2 id="top" class="title">VSTS Test results</h2>
<hr />
<h3>Summary:</h3>
<table>
<tr valign="top" class="TableHeader">
<td width="50px">
<b>Tests</b>
</td>
<td width="70px">
<b>Executed</b>
</td>
<td width="70px">
<b>Failed</b>
</td>
<td width="70px">
<b>Time(s)</b>
</td>
</tr>
<tr valign="top" class="Success">
<td>
<xsl:value-of select="<a>//Counters/@total"/</a>>
</td>
<td>
<xsl:value-of select="<a>//Counters/@executed"/</a>>
</td>
<td>
<xsl:value-of select="<a>//Counters/@failed"/</a>>
</td>
<td>
<xsl:value-of select="<a>//Times/@finish"/</a>>
</td>
</tr>
</table>
<hr />
<table>
<tr valign="top" class="TableHeader">
<td>Test Name</td>
<td>Description</td>
<td>Result</td>
</tr>
<xsl:for-each select="//ManualTest">
<tr>
<td width="70px">
<xsl:value-of select="@name"/>
</td>
<td>
<xsl:value-of select="Description"/>
</td>
<td>
<xsl:value-of select='//ManualTestResult[@testId=current()/@id]/@outcome'/>
</td>
</tr>
</xsl:for-each>
</table>
</body>
</html>
</xsl:template>
</xsl:stylesheet>

Lets understand what I am doing here,
I am creating an HTML page which will have a Header as "VSTS Test Results", then a sub-heading as "Summary". Then I am laying out the summary table which will have 4 columns - Tests, Executed, Failed and Time(s). Now, the real deal, getting values from XML and transforming them into using this XSL. As I know that all my summary values are going to come from node "Counters", I am using XPATH expression to read the corresponding values of attributes and filling them out in the summary table. Example, <xsl:value-of select="//Counters/@total"/> statement is going to jump to the "Counters node" by //Counters, then it is going to look for an attirbute called "total" and xsl:value-of command is going to read the corresponding value of that attribute. In the same way, I read all the required values. My summary table is done.

Next, I create a test result details table with columns - Test Name, Description and Result. Now, the problem here is, name and description of the test case is under "TestDefinitions" node and result is under "Results" node. Much like, two tables in a database. If I jump to "ManualTest" I will get name and description for each test case easily and I can populate the whole table using a loop. But, how am I going to fetch the result??

I start easy, I put a loop around fetching statements of name and description of each test case. When it comes to fill the third column of result, I should find out the corresponding test case in the "Results" node and then fetch the "outcome" attribute. That is exactly what - <xsl:value-of select='//ManualTestResult[@testId=current()/@id]/@outcome'/> is doing. My loop is using node //ManualTest to get the name and description, so to get the corresponding result from //ManualTest Result, I am comparing/matching the "testid" attributes (much like a unique key between these 2 tables). When the testid from //ManualTest matches testid of //ManualTestResult, I am fetching the "outcome" attribute, which is nothing but the result of that test case.

The beauty of this is, I do not have to write 2 cascading loops to go through 2 nodes and their child node and attributes to find the correct result value of that test case, XSL takes care of the inner loop for me (this is more like jumping through nodes and attributes than looping through, so it is also much faster).

To give some final touches, I use a CSS stylesheet to make my results look good!

How to I use it?
Once I have my trx generated after a test pass, I open it in notepad and insert below line just below the XML declare. I am calling above shown XSL transform file "XSLT.xslt"

<?xml-stylesheet type="text/xsl" href="/XSLT.xslt"?>

This statement tell the browser that, when you are opening the xml, look for that particular XSL transform file and apply it before rendering the output. Also, I remove any attributes from the first node of the trx file, it should look just as <TestRun>. If you keep the attributes from <TestRun>, your browser does not treat is as XML format.

Here is how the above TRX will look usning above method of XSL transformation -