CruiseControl.NET, MSTest, NUnit: A Hybrid Solution

As many frustrated build admins know by now, Microsoft has done a poor job of distributing the proper build tools with the .NET Framework. MSBuild is in there now, but MS has failed to see the importance of including MSTest; either that, or the idea was lynched by their marketing department. This leaves us with a problem when it comes to run the unit tests in an integration server that hasn’t been lucky enough to come with another Visual Studio license… While a complete makeover with NUnit is an option we contemplated, I wasn’t quite ready to lose the nice IDE support of MSTest that I’d gotten used to over these past few months. And, as previously discussed, there are a few ways to get MSTest running with CruiseControl.NET, so why get rid of it altogether? That being said, we were not allowed to install Visual Studio on the continuous integration server, nor was I willing to chase down the DLLs I was going to need to manually copy MSTest over.

I decided I wanted the best of both worlds, so I ended up creating hybrid unit tests, which act as MSTest in Visual Studio and NUnit in CruiseControl.NET. The solution, quite at hand, uses conditional compilation symbols, a custom MSBuild script and class imports with aliasing. I’m a fan of the solution, especially because the backing policies are all fail-fast. In other words, the build fails if you make any mistakes along the way. The alternative of CruiseControl.NET missing some of the tests because they weren’t hybridized, with false reports that everything is fine, would be disastrous. Don’t worry, that won’t happen.

The solution also enables one to change the active configuration in Visual Studio to INTEGRATION and compile with NUnit enabled. This provides a quick test that everything is fine.

Section 1: Solution setup

Step 1: Create a solution level build configuration named Integration. You may do this using the Configuration Manager, available in the Build menu. Configure this configuration to build the Debug configurations of the projects in your solution.

Step 2: Update all the existing test projects according to the steps outlined in the section on Test project setup.

Step 3: Create a MSBuild build script that builds the solution and runs NUnit; CruiseControl.NET will use this script. Feel free to use a corresponding NAnt script if NAnt is your preference. The crucial point here is to build the Integration configuration of the solution. Download the complete file. Some snippets below.

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
...
<ItemGroup>
<!-- Collections -->
<ProjectToBuild Include="My Own Spaceship.sln"/>
</ItemGroup>
<Target Name="Clean">
...
</Target>
<Target Name="Build" DependsOnTargets="Clean">
<!-- Build Tasks -->
<MSBuild
Projects="@(ProjectToBuild)"
Targets="Build"
Properties="Configuration=$(Configuration)"/>
</Target>
<Target Name="RunTests" DependsOnTargets="Build">
<CreateItem Include="Tests\**\bin\$(Configuration)\*.Test.dll">
<Output TaskParameter="Include" ItemName="TestAssemblies" />
</CreateItem>
<NUnit Assemblies="@(TestAssemblies)" ToolPath="$(NUNIT_HOME)\bin"/>
<!-- RunTests Tasks -->
</Target>
...
</Project>

Section 2: Test project setup

Run these steps for each unit test project, as you create it.

Step 1: Add NUnit as a reference to the project. NUnit may be in the GAC or as a file somewhere in the solution folder.

Step 2: Create a project level build configuration named Integration. You may do so in the Configuration Manager.

Step 3: For the Integration configuration of the solution, switch to the Integration configuration of the project.

Step 4: Add a conditional compilation symbol named INTEGRATION for the project’s Integration configuration. You may do so by using the Build tab of the project properties pane. By doing so, we instruct the compiler to define the symbol when compiling the project in INTEGRATION mode.

Step 5: Replace all the namespace imports of the form using Microsoft.VisualStudio.TestTools.UnitTesting; with:
#if INTEGRATION
using ExpectedException = NUnit.Framework.ExpectedExceptionAttribute;
using Assert = NUnit.Framework.Assert;
using TestMethod = NUnit.Framework.TestAttribute;
using TestClass = NUnit.Framework.TestFixtureAttribute;
using TestInitialize = NUnit.Framework.SetUpAttribute;
using TestCleanup = NUnit.Framework.TearDownAttribute;
using TestContext = System.Object;
#else
using Microsoft.VisualStudio.TestTools.UnitTesting;
#endif

Step 6: Tone down to the common featureset of MSTest and NUnit. This step includes replacing Assert.Inconclusive with Assert.Fail. As an alternative to this somewhat limiting solution, you may always implement a custom Assert class that fills the gaps, changing the using Assert = ... accordingly. Note that you may still have to drop features such as the use of a test context and so on. I’m not fully aware of all these features, and besides, I really only use the “magnificent seven”—Fail, IsTrue, IsFalse, IsNull, IsNotNull, AreEqual, AreSame.

Section 3: CruiseControl.NET setup

Step 1: Install NUnit 2.4.7 or later on the continuous integration server.

Step 2: Create environment variable NUNIT_HOME, pointing to the installation of NUnit. You will need to restart IIS (or your computer, you choose) to make the environment variable visible to CruiseControl.NET and the processes that it executes. If instead you prefer to hardcode the path to NUnit in the build script from section 1, skip this step.

Step 3: Install the MSBuild community tasks, to run the NUnit task in MSBuild.

Step 4: Ensure that the Microsoft.VisualStudio.QualityTools.UnitTestFramework assembly is NOT in the global assembly cache of the CruiseControl.NET machine. This step is paramount, as it enables the fail-fast policy of failing a build when a test has not been made hybrid.

Troubleshooting

You may get errors involving a sgen.exe tool when building a configuration other than Debug in the integration server. The error has to do with a reference of the MSTest task that is not satisfied. Although we don’t use the task, some form of reference checking still occurs that can blow the whole build process to pieces. This is the error we got on our server:

Task failed because “sgen.exe” was not found, or the correct Microsoft Windows SDK is not installed. The task is looking for “sgen.exe” in the “bin” subdirectory beneath the location specified in the InstallationFolder value of the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.0A. You may be able to solve the problem by doing one of the following: 1) Install the Microsoft Windows SDK for Windows Server 2008 and .NET Framework 3.5. 2) Install Visual Studio 2008. 3) Manually set the above registry key to the correct location. 4) Pass the correct location into the “ToolPath” parameter of the task.

These are the steps I ran in order to fix this issue:

Step 1: Install Microsoft Windows 2008 .NET Framework 3.5 SDK or later on the continuous integration server. (Not sure if this step is required. I decided it was enough of a hack that I didn’t care to clean up for the world of it.)

Step 2: Copy the tool SGEN.EXE from your development machine to the bin directory of the SDK installation. In this case, the path is C:\Program Files\Microsoft SDKs\Windows\v6.1\bin. The version of your SDK may be different, in which case you’d change the path.

Step 3: Create the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.0A. Add a string value namedInstallationFolder under this key, having the value C:\Program Files\Microsoft SDKs\Windows\v6.1\bin.

You may be able to prevent the error with just steps two and three, or with step three alone.

Bonus

If you’re consistent with your tests, you will be frustrated by copy/pasting conditional preprocessing directives into each test class that you create. To alleviate this issue, we’ll create a Visual Studio template and use it in Visual Studio to bootstrap any new test class with the correct imports.

Step 1: Download this ZIP file and save it as HybridUnitTest.zip.
Edit the contents of the files within, if you want them further customized.

Step 2: Copy the ZIP file to C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\ItemTemplates\CSharp\1033.

Step 3: Close any instances of Visual Studio.

Step 4: Open a Visual Studio command prompt. Run devenv /InstallVSTemplates and give it some time.

Step 5: Open Visual Studio; when creating a unit test with New -> Unit Test, one of the options is Hybrid Unit Test.

Questions?

One thought on “CruiseControl.NET, MSTest, NUnit: A Hybrid Solution

  1. Here is an simple class you can add that will allow for Assert.Inconclusive to function. It is wrapped in a conditional compile so that developers don’t need nunit installed on their boxes for compilation to occur. Just add these two lines to your UnitTest class’s using section:

    using Assert = NUnitAdapter.Assert;
    using AssertFailedException = NUnit.Framework.AssertionException;

    and create a file for this guy (I put it in a separate assembly so I can reference it from multiple projects easily):

    #if INTEGRATION
    namespace NUnitAdapter
    {
    public class Assert : NUnit.Framework.Assert
    {
    public static void Inconclusive(string message)
    {
    NUnit.Framework.Assert.Fail(message);
    }

    public static void Inconclusive()
    {
    NUnit.Framework.Assert.Fail();
    }
    }
    }
    #endif

Leave a Reply