MSTest: Measuring Test Quality With NCover

If you’re using the default unit testing tools that come with Visual Studio, then you’re using MSTest. MSTest integrates well with Visual Studio, but if you don’t have VS Team Edition and you want code coverage, you’re out of luck unless you use NCover. Thankfully, using NCover to cover your MSTest tests is fairly straightforward.

The Essentials

If you’re a guru and you already know the intricacies and idiosyncrasies of MSTest and MSBuild and you already have a working <Exec /> task that runs your MSTest tests, then adding NCover on top of the process is easy. Using that same <Exec /> task, just change it to

<Exec Command="$(NCoverPath)\NCover.Console.exe //pm VSTestHost.exe //x $(CoverageFilePath) $(MSTestExecCommand)" />

and you should be able to analyze the coverage in NCover Explorer, or create reports using NCover Reporting.

Covering Tests From The Command Line

The first step in running NCover and MSTest from the command line is to figure out where mstest.exe is installed. This is typically under the “Common7\IDE" folder of your Visual Studio install, but to make sure you have the right path, you need to open a Visual Studio Command Prompt (Start Menu, Program Files, Microsoft Visual Studio, Visual Studio Tools, Visual Studio Command Prompt) and then type "where mstest” on the command line and hit enter. You should then see the full path to the mstest.exe executable, as shown in the image below.

coverage_mstest_ncover_1.png

Next, to cover your unit tests, you’ll use a command line like the following:

ncover.console.exe //pm vstesthost.exe //x mstest_coverage.xml 
    "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"
    /testcontainer:C:\Dev\MyProject\TestAssembly1.dll
    /testcontainer:C:\Dev\MyProject\TestAssembly2.dll

This tells NCover to cover mstest.exe running tests from two assemblies – TestAssembly1.dll and TestAssembly2.dll. It also tells NCover to only collect data for the vstesthost.exe process that MSTest uses to run its tests, so that data that isn’t part of the tests is not returned.. It outputs the coverage file to mstest_coverage.xml.

If you’re running NCover on a 64-bit machine, then the command will take the fact that we’re running MSTest from the Program Files (x86) folder:

"C:\Program files (x86)\NCover\NCover.Console.exe" //pm vstesthost.exe //x mstest_coverage.xml 
    "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"
    /testcontainer:C:\Dev\MyProject\TestAssembly1.dll
    /testcontainer:C:\Dev\MyProject\TestAssembly2.dll

Hey, that was easy! So how can you run tests through NCover Explorer?

Covering Tests From NCover Explorer

Now that you have a command line that works, it’s easy to set up a project in NCover Explorer so you can run your tests and examine code coverage right away. Open NCover Explorer and create a new project by pressing Control-N. You should then see the Project Configuration window. Populate the fields with the information from the command you created in the previous step, as shown in the image below. Note that the Working Folder field should be the folder where your test assemblies are stored.

coverage_mstest_ncover_2.png

Next, since MSTest tests are actually run in the context of a helper process called “VSTestHost.exe”, you need to tell NCover to cover “VSTestHost.exe” instead of “MSTest.exe”. To do that, go to the “Advanced Options” screen, and in the “Profiled Process Name” field enter “VSTestHost.exe” (case-insensitive, but you must include the extension) as shown in the screenshot below.

coverage_mstest_ncover_4.png

On a 64-bit Machine

If you’re running NCover on a 64-bit machine, then you need to go to the NCover Path section and change the Path to NCover.Console field to the location of the 32-bit version of NCover, as was done in the screenshot below.

coverage_mstest_ncover_3.png

Finally, you should close the Project Configuration window and press the the green Run Coverage button to run NCover and start looking at coverage data!

Covering Tests From A Build Script

Well, now that you've set up a project in NCover Explorer, getting coverage from an automated build script is just a click away. Just press the Script Helper button in NCover Explorer’s toolbar and select either the MSBuild NCover Task or the MSBuild Exec Task to have NCover Explorer generate the build script for you.

<?xml version="1.0" encoding="utf-8"?>
<!-- Generated by NCover Explorer at http://www.ncover.com/ -->
<Project DefaultTargets="Coverage" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" >

  <!-- TODO: Uncomment and add the full path to NCover.MSBuildTasks.dll if not in your path -->
  <!-- TODO: Alternatively, remove this line if you have copied the .dll to your MSBuild bin folder -->
  <!-- <UsingTask TaskName="NCover.MSBuildTasks.NCover" AssemblyFile="NCover.MSBuildTasks.dll"/> -->
  <Target Name="Coverage">
    <NCover ToolPath="C:\program files (x86)\ncover"
            TestRunnerExe="C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"
            TestRunnerArgs="/testcontainer:&quot;D:\dev\ncover\release\ncover.test.unit.dll&quot;
                            /testcontainer:&quot;D:\dev\ncover\release\ncover.test.functional.dll&quot;"
            WorkingDirectory="D:\dev\ncover\release\"
            CoverageFile="mstest_coverage.xml"
            SymbolSearchLocations="Registry, SymbolServer, BuildPath, ExecutingDir"
            CoverChildProcess="vstesthost.exe"
            ProjectName="New Project"    />
  </Target>
</Project>

Just plug the coverage target or the NCover task into your build script wherever makes the most sense, and you're ready to integrate NCover into your build server.

Questions? Comments?

Contact us.