skip to content

How does running MSTest 4 tests on Microsoft.Testing.Platform differ from running them on VSTest?

level: seniorimportance: must knowfreq 40%

answer

  1. library hosted versus executable
  2. testhost and adapters versus in-process
  3. data collectors become extensions
  4. options come from installed packages
  5. exit code 8 when zero tests

basics

~20 s

VSTest loads a test library into its own testhost through vstest.console and discovered adapters. Microsoft.Testing.Platform compiles the framework and its extensions into the test project, which runs as an executable and accepts only the options its referenced extensions provide.

solid answer

~50 s

Under **VSTest** the test project is a library: `Microsoft.NET.Test.Sdk` wires it in, `vstest.console` (or `dotnet test` in VSTest mode) starts a `testhost` process, finds the MSTest adapter, and adds data collectors and loggers configured through `.runsettings` and switches such as `--logger trx`. Under **Microsoft.Testing.Platform (MTP)** the test project is an executable that hosts MSTest in-process; the build generates its entry point and registers whatever `Microsoft.Testing.Extensions.*` packages it references. You can run it as the executable itself, through `dotnet run`, through `dotnet test` in its MTP mode, or from an IDE. Options come from those extensions — `--report-trx`, `--coverage`, `--hangdump` — and an unknown option fails the run with exit code 5. A run that executes no tests fails with exit code 8 by default. In MSTest 4 the adapter talks to MTP natively, and AppDomain isolation is off by default under MTP.

code

bash · 8 lines
bash
# VSTest era: library + runner switches
dotnet test --logger trx --collect "Code Coverage"

# MTP: the test project is a program; run it directly
./tests/Billing.Tests/bin/Release/net10.0/Billing.Tests \
  --report-trx \
  --coverage --coverage-output-format cobertura
echo "exit code: $?"   # 0 pass, 2 failed tests, 5 bad option, 8 zero tests

go deeper

for a junior

Recall the headline: on VSTest the test project is a library run by vstest.console; on Microsoft.Testing.Platform it is an executable you can run directly.

for a middle

Explain how each platform finds the framework and adds features: adapters and data collectors versus compiled-in extension packages with their own command-line options.

for a senior

Anticipate the migration traps: rejected VSTest switches, exit code 8 on empty runs, coverage excluding the test assembly, AppDomains off by default, and never mixing both platforms in one solution.

for a principal

Judge when to move a portfolio to MTP: faster, self-contained test apps against the cost of retooling pipelines, IDE habits and any vendor tooling that only understands VSTest.

## Two ways to run the same tests MSTest 4 tests — the attributes, the `Assert` calls — are identical whichever **test platform** runs them. The platform is the machinery around them: what process hosts the tests, how the framework is found, how results and side features like coverage are produced. - **VSTest** is the long-standing platform behind `vstest.console`, `dotnet test`'s original mode and the Visual Studio Test Explorer. - **Microsoft.Testing.Platform (MTP)** is the newer, lighter platform that ships from the same repository as MSTest; MSTest 4.4.1 pairs with MTP 2.4.1. ## How VSTest runs a test project 1. The test project is a **class library**. A reference to `Microsoft.NET.Test.Sdk` marks it as a VSTest test project. 2. A runner — `vstest.console`, or `dotnet test` in VSTest mode — starts a separate **testhost** process. 3. The testhost loads the library and finds the **test adapter** (`MSTest.TestAdapter`) that knows how to discover and run its tests. 4. Cross-cutting features are **data collectors** (code coverage, blame) and **loggers** (TRX, console), configured through a `.runsettings` file and runner switches such as `--logger trx` or `--collect "Code Coverage"`. ## How MTP runs a test project 1. The test project is an **executable**. With MSTest.Sdk this is the default; with packages it is `EnableMSTestRunner` plus `OutputType` `Exe`. 2. The build generates the entry point and registers the test framework and every referenced extension, so the program needs nothing else to run. 3. It can be started as the executable itself, through `dotnet run`, through `dotnet test` in its MTP mode, or from an IDE. 4. Features are **extensions**: NuGet packages such as `Microsoft.Testing.Extensions.TrxReport` or `Microsoft.Testing.Extensions.CodeCoverage` that add command-line options when referenced. ## Side by side | Concern | VSTest | Microsoft.Testing.Platform | |---|---|---| | Test project output | library | executable | | Host process | separate `testhost` started by a runner | the test application itself | | Framework lookup | adapter discovery | compiled in at build time | | Coverage, dumps | data collectors | extension packages and options | | TRX | `--logger trx` | `--report-trx` from the TRX extension | | Configuration | `.runsettings` | `testconfig.json`, CLI options; MSTest still reads `.runsettings` via `--settings` | | Unknown option | often ignored | run fails, exit code 5 | | No tests executed | configurable | fails, exit code 8 | ## Behaviours that surprise teams - **Strict options.** Because an option exists only if its extension is referenced, a typo or a VSTest-only switch is rejected rather than silently ignored. MTP even suggests replacements, such as `--report-trx` for `--logger trx` and `--coverage` for `--collect "Code Coverage"`. - **Zero tests is a failure.** A filter that matches nothing, or a project with no tests, exits with code 8 (`ZeroTests`). `--minimum-expected-tests` raises the bar further. - **Coverage excludes the test assembly.** The MTP coverage extension defaults `IncludeTestAssembly` to `false`, whereas VSTest's collector included it, so coverage percentages can shift after a move. - **No AppDomain isolation by default.** In MSTest 4 under MTP, AppDomains are disabled unless `DisableAppDomain` is set to `false`, because the isolation cost up to 30% of run time and rarely helped. A suite that leaned on AppDomain unload to end stray foreground threads can now hang. - **One framework per assembly.** MTP has no concept of loading several adapters into one host; each test application carries exactly its own framework. Since MSTest 4.4 the MSTest adapter integrates with MTP natively, without the `Microsoft.Testing.Extensions.VSTestBridge` layer earlier versions used. The same MSTest project can still be built for VSTest — `UseVSTest` in MSTest.Sdk, or the packages without the runner — but a solution should pick one platform for all its test projects. ## A short migration checklist 1. Make every test project an MTP application — MSTest.Sdk, or `EnableMSTestRunner` with `OutputType` `Exe`. 2. Reference the extensions the pipeline needs (TRX, coverage, dumps) through the profile or `Enable…` properties. 3. Replace VSTest switches with extension options and move `.runsettings` content to `testconfig.json`. 4. Switch `dotnet test` to its MTP mode for the whole repository. 5. Run each test executable once by hand with the CI's options and check the exit code and the files it writes. ## Why move MTP removes the separate runner and testhost hop, makes a test project runnable and debuggable like any program, and turns every side feature into an explicit, versioned package. The cost is migration work wherever scripts, pipelines or tools assumed VSTest's switches and `.runsettings` data collectors.

  • Why does an MTP test run fail when a filter matches no tests?
    MTP treats a run that executes no tests as a failure and exits with code 8 (`ZeroTests`), on the view that a test step which ran nothing usually hides a broken filter or a mis-built project. `--minimum-expected-tests` lets a run demand more than one test. VSTest-era pipelines that tolerated empty runs need to fix the filter rather than the exit code.
  • What does exit code 5 from an MTP test application usually mean after a migration?
    Exit code 5 is `InvalidCommandLine`: the application was given an option no referenced extension provides. After a migration that is typically a leftover VSTest switch such as `--logger trx`, or an extension option like `--hangdump` whose package the project does not reference. MTP's error text suggests the replacement option and the package that provides it.
  • Why might coverage numbers change after moving from VSTest to MTP?
    The `Microsoft.Testing.Extensions.CodeCoverage` extension defaults `IncludeTestAssembly` to `false`, while VSTest's collector counted the test assembly too. Excluding test code usually lowers the covered-lines total and changes the percentage, so compare like with like or set the option explicitly in the coverage settings before treating a drop as a regression.

VSTest is a print shop: you hand over a document and the shop's own machines and add-ons decide how it is printed. Microsoft.Testing.Platform gives each project its own printer with the add-ons bolted on at build time, so it runs anywhere, but only with the add-ons you installed — ask for a missing one and it refuses the job.

  • Microsoft.Testing.Platform still launches a separate testhost through vstest.console.
  • An MTP test application ignores options it does not recognise.
  • MTP reports success when a filter matches zero tests.
  • Code coverage works on MTP without referencing a coverage extension.
  • Moving to MTP requires rewriting MSTest attributes and assertions.
  • Under MTP, MSTest 4 still isolates every test assembly in its own AppDomain.

Assumes MSTest 4.4.1 on Microsoft.Testing.Platform 2.4.1: the adapter is native to MTP since 4.4, and AppDomain isolation is off by default under MTP since MSTest 4.