TRX merger for Visual Studio Team System 2008

People working on test projects that involve test management and execution from Visual Studio Team System 2008 and who do not have a TFS server for reporting will find this a handy utility.

I have already shown how one can generate a simple html report from a .trx - Visual Studio result file in Generating HTML reports from TRX using XSLT. But what if you want your html report to show test results from different testers in your team or you just want to overwrite the existing results with the latest? Without a TFS, this remains a challenge.

Let me explain the situation in detail -

1. I have 5 testers on my team, each one test a specific area and generate a .trx at the end of the test pass. I get 5 .trx daily and I don't want to show 5 different html reports. At the end of the day, I want to generate one single report for all the tests that were run on that day. This is first part of the problem. And clearly, I need to somehow bring in all .trxs into one so I can apply my XSLT or other custom tool for that matter to produce single html report.

2. Some areas under test are weak, or may be the tests are weak, because of which, one (or more) of the testers need to execute tests multiple times. Think of an automation suite, you execute 100 and 80 pass, then you execute the failing 20 and some 12 pass and so on, till you get to a point where you have a small set of "true" failures. Everytime you run some tests, VSTS is goint to produce a separate trx file. But none of those have the full and consoldiated information you need. Here, you need some magic which can consolidate results over multiple test passes to generate a single result trx file, which goes to the html generator or gets merged with other trxs, that depends!

This was one of the problems which landed on my lap for solution. Above two situations are my design requirements -
Build a utility which can merge two trx files and produce a single output file.

1. If the files are different, append them and add up numbers in the summary

2. If the files are subsets, overwrite the newer file on the older file with a condition that only passing test results are overwritten. In this case, you need to be careful for the summary as just adding numbers does not help!

    1. if test1 passed, do not overwrite it (with any outcome in second/latest tests pass.) Mostly, you will not run this test case over again on that day/build as it is already passing.
    2. If test1 failed, and you have re-ran it and is now passed, overwrite the result.
    3. If test1 passed, and you still re-ran (for some reason) and it failed, do not overwrite the result, as you know it passes and nothing is wrong!!

I called it "TRXMerger" project, and here is how it is built. I am using Console Application to build this utility. It will take 3 parameter, first 2 are trx files to be merged and 3rd is the output file, I want to save the merged results to.

You does not need anything more than,

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.IO;

The Main() goes as, As it is not a big program, I have stuffed pretty much everything in main(). When it comes to merging, essentially, you will have to merge two main node branches, "TestDefinitions" and "Results". The 2 helper functions - IfTestExists and IfResultExists guide the append or overwrite decision.

static int Main(string[] args)
        {
            string message = "\nUsage: TRXMerge <first trx file> <second trx file> <output XML file>\n\nInfo: Changes from second trx file will be overwritten on first if there are matching test cases";
            if (args.Length < 3)
            {
                Console.WriteLine(message);
                return 1;
            }
            else
            {
                if (!File.Exists(args[0])) { Console.WriteLine(message); return 1; }
                System.Xml.XmlDocument oDocFirst = new XmlDocument();
                oDocFirst.Load(MakeCompatXML(args[0]));
                if (!File.Exists(args[1])) { Console.WriteLine(message); return 1; }
                System.Xml.XmlDocument oDocSecond = new XmlDocument();
                oDocSecond.Load(MakeCompatXML(args[1]));
                ////locate sections in first and append data from second...
                XmlNode oNodeWhereInsert = oDocFirst.SelectSingleNode("//TestDefinitions");
                int i = 0;
                while (oDocSecond.SelectSingleNode("//TestDefinitions").ChildNodes.Count != i)
                {
                    ////insert test only if it is not already present
                    if (!IfTestExists(oDocFirst, oDocSecond.SelectSingleNode("//TestDefinitions").ChildNodes[i].Attributes["name"].Value))
                    {
                        oNodeWhereInsert.AppendChild(oDocFirst.ImportNode(oDocSecond.SelectSingleNode("//TestDefinitions").ChildNodes[i], true));
                    }
                    i++;
                }
                ////insert new results and update existing if present
                XmlNode oNodeWhereInsertResult = oDocFirst.SelectSingleNode("//Results");
                i = 0;
                while (oDocSecond.SelectSingleNode("//Results").ChildNodes.Count != i)
                {
                    XmlNode oldNode;
                    if (IfResultExists(oDocFirst, oDocSecond.SelectSingleNode("//Results").ChildNodes[i].Attributes["testName"].Value, out oldNode))
                    {
                        oNodeWhereInsertResult.RemoveChild(oldNode);
                        oNodeWhereInsertResult.AppendChild(oDocFirst.ImportNode(oDocSecond.SelectSingleNode("//Results").ChildNodes[i], true));
                    }
                    else
                    {
                        oNodeWhereInsertResult.AppendChild(oDocFirst.ImportNode(oDocSecond.SelectSingleNode("//Results").ChildNodes[i], true));
                    }
                    i++;
                }
                if(File.Exists(args[2]))
                {
                    File.Delete(args[2]);
                }
                oDocFirst.Save(args[2]);
                SetSummary(args[2]);
                return 0;
            }
            
        }

Always remeber that the first file you pass to the program is the one being searched.

public static bool IfTestExists(XmlDocument doc, string testName)
        {
            int i = 0;
            while (doc.SelectSingleNode("//TestDefinitions").ChildNodes.Count != i)
            {
                if (doc.SelectSingleNode("//TestDefinitions").ChildNodes[i].Attributes["name"].Value == testName)
                {
                    return true;
                }
                i++;
            }
            return false;
        }
        public static bool IfResultExists(XmlDocument doc, string testName, out XmlNode oldNode)
        {
            int i = 0;
            while (doc.SelectSingleNode("//Results").ChildNodes.Count != i)
            {
                if (doc.SelectSingleNode("//Results").ChildNodes[i].Attributes["testName"].Value == testName)
                {
                    oldNode = doc.SelectSingleNode("//Results").ChildNodes[i];
                    return true;
                }
                i++;
            }
            oldNode = null;
            return false;
        }

Till this points, things are pretty routine and easy. But building the "ResultSummary" node is a tricky job, as when the tests are distinct, you want to add up the numbers, but when it is a subset overwrite; you have keep track.

To simplify, I am performing that as a separate activity "after" merging. If you notice in the Main(), I am closing the file after I am done with merging and then pass the "merged" file through another processing unit (SetSummary) to set the "ResultSummary" values correctly. This is much easier than, keeping track of passed and failed tests as I merge.

public static void SetSummary(string fileName)
        {
            System.Xml.XmlDocument oDoc = new XmlDocument();
            oDoc.Load(fileName);
            XmlNode master = oDoc.SelectSingleNode("//ResultSummary/Counters");
            summary oSummary;
            oSummary.passed = 0;
            //count the number of test cases for total
            oSummary.total = oDoc.SelectSingleNode("//TestDefinitions").ChildNodes.Count;
            //count the number of test cases executed from count of test results
            oSummary.executed = oDoc.SelectSingleNode("//Results").ChildNodes.Count;
            //count the number of passed test cases from results
            int i = 0;
            while (oDoc.SelectSingleNode("//Results").ChildNodes.Count != i)
            {
                if (oDoc.SelectSingleNode("//Results").ChildNodes[i].Attributes["outcome"].Value == "Passed")
                {
                    oSummary.passed++;
                }
                i++;
            }
            ////update summary with new numbers
            master.Attributes["total"].Value = oSummary.total.ToString();
            master.Attributes["executed"].Value = oSummary.executed.ToString();
            master.Attributes["passed"].Value = oSummary.passed.ToString();
            ////locate and update times
            XmlNode oTimes = oDoc.SelectSingleNode("//Times");
            //add a new attribute elapsed time, original trx would not have that
            XmlAttribute elapsed = oDoc.CreateAttribute("elapsedtime");
            elapsed.Value = CalculateSummaryTime(oDoc).ToString();
            oTimes.Attributes.Append(elapsed);
            oDoc.Save(fileName);
            
        }

The "ResultSummary" node has a lot of information, but I am only interested in correct nubmers for "total number of tests", "total number of tests ran", "number of tests passed" and "total time of execution" in my output/merged file. You can always calculate other attributes with the same logic shown above.

Counting childnodes of "TestDefinitions" node gives me "total number of tests". Counting childnodes of "Results" gives me "total number of tests ran". With little more parsing of Results gives me "number of tests passed". I am setting these values for the corresponding attributes of ResultSummary node of my merged file.

To calculate "total time of execution", I need some extra processing power and some workaround which is built inside "CalculateSummaryTime()" helper function. If you look inside a normal trx file, you will find that ResultSummary node has attributes - starttime and finishtime which will give you the total execution time for reporting. But when you merge two trx files, generated at 2 different times (if not time-zones!), basically you have 2 "ranges" of times. You can not add up these numbers in any case, whether you are appending or overwritting. This is a totally different game!

So, to get the "total execution time" for my reporting, I am going to parse the merged file and add up execution time for each test case. That makes the total time, isn't it? But the problem is, I don't have any attribute for ResultSummary which can/should hold this calculated time. I am going to add a new attribute to my output file, called as "elapsedtime" in addition to othe attributes.(and as my html reporting tool is also home grown, I can make a change to it to handle this special case!).

public static double CalculateSummaryTime(XmlDocument doc)
        {
            DateTime sTime;
            DateTime eTime;
            double sDiff = 0;
            int i = 0;
            while (doc.SelectSingleNode("//Results").ChildNodes.Count != i)
            {
                sTime = Convert.ToDateTime(doc.SelectSingleNode("//Results").ChildNodes[i].Attributes["startTime"].Value);
                eTime = Convert.ToDateTime(doc.SelectSingleNode("//Results").ChildNodes[i].Attributes["endTime"].Value);
                //// calculate timespan for each test case and add them to calculate total time
                sDiff += (eTime - sTime).TotalSeconds;
                i++;
            }
            return sDiff;
        }

Done!