At Trinicom, the place I work, we are creating a new customer experience product. We lean heavily on good engineering practices and use Scrum as our development process.
For the application we defined cornerstones, or fundamentals: usability being one of them.
Because our application needs to be zero footprint web based, we use JavaScript a lot, mostly JQuery actually. We structure our JavaScript code using module and jquery widget patterns. We are thinking about implementing an mvc framework for Javascript, or use something like knockout.js, because we depend on real-time data.
Our application needs real-time info and because we develop for the future we are leveraging Websockets to make the real-time updates a reality.
As I mentioned we lean on good engineering practices, TDD is important for our C# code, so because we develop more and more JavaScript code, we are using TDD for our client-side code also.

QUnit
We use QUnit for our JavaScript unit tests. We use Continuous Integration and run our unit tests at every check-in, on TFS 2010. We wanted that for our JavaScript unit tests also.
So we started searching for information, there was very little. It seems that most ASP.NET devs do not integrate JavaScript unit tests into their builds, or do not unit test their JavaScript at all.

Web Client Developer Guidance
The Web Client Developer Guidance from Patterns & Practices on Codeplex provides some information on how to integrate QUnit unit tests into a TFS 2008 build. The problem was, that the guidance is incomplete and unclear. The P&P group created the QUnitExtensions.js that provides the familiar Asserts C# developers know and love, isTrue, isFalse, areEqual, IsNull, isNotNull, isUndefined, isNullOrUndefined, isNotNullNorUndefined.
The guidance exist of a ASP.NET MVC  app that takes JSON and serializes that to some xml format. The xml format is than persisted on disk. The JSON is provided by the QUnit testrunner that is extended and uses the url to a Controller of ASP.NET MVC app that takes JSON. The extended testrunner serializes the testresults into a hidden formfield and submits the contents of the field to the MVC app’s controller.

The format that was persisted to disk is unfamiliar and does not seem to map to any TFS format. The guidance says all you have to do is write the xml file to disk and tell the buildserver where it is by using an msbuild script and the result is written to the buildlog etc. That did not do the trick for us.

Workflow Activity
We created a Workflow activity that we use in our build workflow.

[RequiredArgument]
public InArgument ResultFile { get; set; }

[RequiredArgument]
public InArgument BuildDetail { get; set; }

protected override void Execute(CodeActivityContext context)
{
	IBuildDetail buildDetail = context.GetValue(this.BuildDetail);
	string file = context.GetValue(this.ResultFile);

	TestRun testRun = DeserializeTestRun(file);
	string outcome = GetOutcome(testRun.TestsFailed);

	IActivityTracking currentTracking = context.GetExtension().GetActivityTracking(context);
	IBuildInformationNode childNode = currentTracking.Node.Children.CreateNode();
	childNode.Type = currentTracking.Node.Type;

	IBuildStep  buildStep = childNode.Children.AddBuildStep("JS Unittest Build Step", string.Format("The Javascript Unittests for build: {0} have run", testRun.Name));
	if(testRun.TestsFailed > 0)
	{
	   IBuildError buildError = childNode.Children.AddBuildError(string.Format("The Javascript UnittestRun {0} ", outcome), DateTime.Now);
		buildError.Save();
	}

	WriteNewBuildInformationNode(currentTracking, string.Format("The Javascript UnittestRun for build: {0} {1}",  testRun.Name,outcome));
	WriteNewBuildInformationNode(currentTracking, string.Format("{0} test(s) run", testRun.TestsTotal));
	WriteNewBuildInformationNode(currentTracking, string.Format("{0} test(s) succesfull", testRun.TestsPassed));
	WriteNewBuildInformationNode(currentTracking, string.Format("{0} test(s) failed", testRun.TestsFailed));
	WriteTestCaseInformation(testRun.TestCases, currentTracking, buildDetail);

	SetBuildStatus(testRun.TestsFailed > 0, buildStep, buildDetail);

	if (testRun.TestsFailed == 0)
	{
		WriteToBuildSummary(buildDetail, string.Format("The Javascript UnittestRun for build: {0} {1}", testRun.Name, outcome));
	}

	buildStep.FinishTime = DateTime.Now;
	buildStep.Save();
	buildDetail.Information.Save();
}	

Listing 1

The activity takes a file and deserializes the xml into objects, it than uses the object graph to report into the buildlog if the unit test where succesfull, or not.

TestRun Domain objects

post_jsut_01_TestRun_classdiagram

Img 1: The TestRun domain

Image 1 shows the domain objects of the TestRun. A TestRun has one or more TestCases and a TestCase can have  an Output that is of the ErrorInfo type.
A TestCase has an OutCome and a StatusPassed, if an Exception was thrown, the details of that Exception are found in the Output property.
The TestRun has three properties that hold the result for the run, how many Tests where run (TestTotal), how many Test failed (TestFailed) and how many tests passed (TestsPassed).
The Activity ofcourse uses the three properties of the TestRun to decide wether the Run was succesfull or not. When the TestRun is not successful, the TestCases that failed are written to the buildlog and the build will fail.

ITestResultSerializer interface
The Web Client Developer Guidance contains an MVC application that uses an interface named: ITestResultSerializer. I wrote our own implementation that is simpler than the implementations provided by Patterns & Practices.
They use XmlTextWriter to write out the xml, I just serialize the objectgraph to xml.

ITestResultSerializer Tfs implementation

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;
using System.Xml;
using System.IO;
using System.Xml.Serialization;
using QUnitTestResult.Contract;
using QUnitTestResult.Models.Tfs;

namespace QUnitTestResult.Models
{
    public class TfsTestResultSerializer : ITestResultSerializer
    {

        public void Serialize(TestRun testRun, string resultsFilePath, string fileName, string buildName)
        {
            string filename = Path.Combine(resultsFilePath, fileName);
            if (!Directory.Exists(resultsFilePath))
            {
                Directory.CreateDirectory(resultsFilePath);
            }

            using (var textWriter = new XmlTextWriter(filename, Encoding.UTF8))
            {
                var xmlSerializer = new XmlSerializer(typeof(TestRun));
                xmlSerializer.Serialize(textWriter, testRun);
            }
        }

    }
}

Listing 2: TfsTestResultSerializer

The TfsTestResultSerializer writes the xml to a location on disk. The build workflow knows this location and picks the xml up and the Activity publishes the results to the buildlog of our TFS 2010 TeamBuild.

Workflow – BuildTemplate

To make it possible to run JavaScript unittests in an TFS 2010 Teambuild, I made some changes to the workflow of the BuildTemplate.

Arguments
In the workflow I defined a few Arguments, that can be set in the builddefinition. The arguments are:

  • RunJavascriptUnittests (datatype boolean - default value = false)
  • JavascriptUnittestBrowserFiles (datatype List<string> – default value = new String() { “c:\program files\apple\safari.exe” })
  • TestresultAcceptorWebsite (datatype string – default value = “localhost:8081”)
  • JavascriptTestRunnerFile (datatype string – default value = “scripts/unittests/testrunner.htm”)

 

post_jsut_buildtemplate_args

Img 2: Buildtemplate arguments

 

In the default BuildTemplate there is an Activity that is called: “Run Tests”. You can find it if you follow the default BuildTemplate and click your way through the following path in the workflow:

Process > Sequence > Run On Agent > Compile, Test, and Associate Changesets and Work Items > Sequence > Compile, Test, and Associate Changesets and Work Items > Try Compile and Test > Compile and Test > For Each Configuration in BuildSettings.PlatformConfigurations > Compile and Test for Configuration > If Not DisableTests > Run Tests

Below the “If Then” Activity: “If Not TestSpec Is Nothing” I add my own “If Then Activity” and call it: “If RunJavascriptUnittests is True”.

post_jsut__02_RunTests_SequenceImg 3: Run Tests Sequence Build Workflow

In the If I check a boolean Argument I added to the builddefinition (as stated in the Arguments section above), that is used to specify if we want to run JaveScript Unittests in the build.

post_jsut__03_If_jsutests_need_torunImg 4: If RunJavaScriptUnitTests is set to true

If this argument is set to true, the “Then” branche in the If then Activity is folllowed. In the “Then" part I add a Sequence called: “Run Javascript Unittests”.

 post_jsut__04_ForeachbrowserexeinImg 5: Run JS UnitTests

The sequence holds a “For Each” Activity: “ForEach Browser exe in JavascriptUnittestBrowserFiles”.

post_jsut__05_ForeachtestrunnerinImg 6: ForEach browser exe in Argument

In the “ForEach” Activity all strings in the “JavascriptUnittestBrowserFiles” argument are iterated over, these strings hold the path and name of the executable of a browser on the buildserver (for example “C:\users\username\AppData\Local\Google\Chrome\Application\chrome.exe”).
Inside the Body another “ForEach” activity “Foreach testRunner in JavascriptTestRunnerFiles” exists.

Sequence

post_jsut_foreach_testrunner_in_jstestrunnerFiles

Img 7: For each testrunner in argument

This “ForEach” Activity will iterate over all strings in the Argument: “JavascriptTestRunnerFile”, this is so that we can have more than one html file containing Javascript unittests. Inside the body I put another Sequence activity (img 8).

post_jsut_sequence_actual_workImg 8: Sequence that does the work 

WriteBuildMessage
The “WriteBuildMessage” only writes to the buildlog for debugging purposes, also it comes in handy when someting goes wrong in a build sometimes.

Start testrunner js unittests (InvokeProcess)
Next the “Start testrunner js unittests” InvokeProces Activity will be run. The properties are shown in img  9.

post_jsut__10_StartTestRunnerJsUT

Img 9: InvokeProcess calls browser exe with html file as parameter

The “FileName” property is filled with the “browserFile” variable from img 6 and holds the exe of the browser.

The ‘”Arguments” property  is the string that concatenates all variables to a string that holds the important pieces of the puzzle.

post_jsut__11_StartTestRunnerJsUTArguments
Img 10: Arguments property The string looks like listing 4:

"""file:///" + BuildDirectory + "\Binaries\_PublishedWebsites\" + JavascriptTestRunnerFile + "?submitResults=" + TestresultAcceptorWebsite + "/qunit/acceptor&resultfilePath=" + TestResultsDirectory + "&build=" + BuildDetail.BuildNumber + "&platform=" + platformConfiguration.Platform + "&flavor=" + platformConfiguration.Configuration + "&teamproject=" + BuildDetail.BuildDefinition.TeamProject + """"

Listing 4

Listing 4 shows how the Arguments are built up: It tells the browser to open the “JavascriptTestRunnerFile” which holds the html filename in our project as a file on the filesystem, we use the “"BuildDirectory” variable and the kwoledge where the websites are published in this directory to point to the right file.
Next we add querystring variables and values so the testrunner javascript file knows where to send the results of the tests and which build we run, what solutionconfiguration we are building etc .

 

browserFile.Substring(0, browserFile.LastIndexOf("\"))

Listing 5 Working Directory

Listing 5 shows the value for the Working directory property.

FindMatchingFiles
The next step is to check if the results are written to disk as expected, with the “FindMatchingFiles” activity using the MatchPattern shown in Listing 6:

TestResultsDirectory + String.Format("\QUnitResult_{0}.xml", BuildDetail.BuildNumber)

Listing 6 MatchPattern

 

When the file is found the variable JSUnitTestMatchingFileResult is filled with the number of results.

If Js UnitTestResultFile is found
The “If Then” activity “If Js UnitTestResultFile is found” is run.

post_jsut__14_IfFileFoundImg 11: “If Then” activity

If the results Count() is equal to 1 the Workflow Activity mentioned in the beginning of the post is called (“JsUnitTestResultReader”), if not a message is written to the buildlog.

post_jsut__15_ReadJSResultsActivityImg 12: Properties JsUnitTestResultReader 

We pass the BuildDetail to the Activity, so the context is known.
We also need the name and path of the resultfile, to read the results and write them to the buildlog.

post_jsut__16_ReadJSResultsActivityResultFileImg 13: ResultFile property

TestResultsDirectory + String.Format("\QUnitResult_{0}.xml", BuildDetail.BuildNumber)

Listing 7 ResultFile

Now the circle is full, and the results can be written to the buildlog.

post_jsut_resultbuildlog_posImg 14: Passed unittests in log details

Of course when a test fails the build fails partially (like normal unittests) and the result shows in the buildlog summary, the results like shown in img 14 are only visible in the details (diagnostics) part of the buildlog.

Henry Cordes
My thoughts exactly…


Comments (11) -

Gurpreet Sohal United Kingdom | Reply

6/15/2011 10:59:44 PM #

This is fantastic, if only there was a set of complete source code for this, that would help immensly. Would you be able to arrange this?

6/16/2011 12:55:10 PM #

What part of the source do you need?
I can arrange to give you the code, but there are a few moving parts.
I can send you the Build Workflow Activity, the javascript to run the testrunner and send the result to a service and the mvc application that acts as the service that accepts the result and writes it to file.

Gurpreet Sohal United Kingdom | Reply

6/16/2011 7:49:10 PM #

Hey Henry,

That would be great. If I could just get the components so that I can throw this at my TFS solution and see it work!

Kind regards,

Gurpreet Sohal United Kingdom | Reply

6/16/2011 7:52:24 PM #

The other thing I was wondering, with qunit, how do you actually test the DOM?

I am using knockout and have found it easy to unit test the viewmodel. Would I need to have some sort of coded ui/selenium solution and then push my test scripts into that?

My aim is to integrate javascript testing at a viewmodel and DOM level, so that I can assert the attributes of certain html elements.

If you could give me some advice, along with the source as outlined in the previous comment I have made, that would be much appreciated.

Thank you again

6/17/2011 9:30:15 AM #

With QUnit you create a 'mock' of the html needed to test your javascript.
I will send you an example of how we use it.
We are very interested in using Knockout, so maybe we can help each other when we have questions Wink
I will send the code later today.

Gurpreet Sohal United Kingdom | Reply

6/17/2011 10:56:14 AM #

That's great.

I look forward to your email and possible collboration Smile

Gurpreet

Gurpreet Sohal United Kingdom | Reply

6/18/2011 9:48:18 AM #

did you send the email yet/?

6/20/2011 9:26:02 AM #

I sent it an hour after the comment, I will send again right away.

6/20/2011 10:19:26 AM #

The email is failed.
It seems like I cannot send email to the email address you gave me....

Gurpreet Sohal United Kingdom | Reply

6/20/2011 11:32:18 AM #

hey try sending it to gurpreet at gurpreetsohal.com

thanks =)

Luke Page United Kingdom | Reply

3/7/2012 8:42:51 AM #

Many many thanks for such an informative post ;- its very strange Microsoft do not make this easier.

I too would be very interested in the source so we can try to use it as the basis for establishing unit test checking in the builds. Would it be possible to stick it on a github repository or something like that?

Otherwise an email would be amazing.

Add comment




  Country flag
biuquote
  • Comment
  • Preview
Loading