Skip to content

[MTP Integration Analysis] MSBuild summary marks skipped-only and zero-test runs as failed #8433

@Evangelink

Description

@Evangelink

Summary

Microsoft.Testing.Platform.MSBuild decides whether the per-TFM summary says Passed or Failed by checking TotalPassed == 0. That means skipped-only runs and zero-test runs are always formatted as failed summaries even when there are no failed tests.

Evidence

  • src/Platform/Microsoft.Testing.Platform.MSBuild/Tasks/InvokeTestingPlatformTask.cs:511-516
    • string summary = string.Format(
    • runSummaryInfoRequest.TotalFailed > 0 || runSummaryInfoRequest.TotalPassed == 0
    • ? Resources.MSBuildResources.Failed
    • : Resources.MSBuildResources.Passed
  • src/Platform/Microsoft.Testing.Platform.MSBuild/Resources/MSBuildResources.resx:147-149
    • Summary is rendered as {0}! - Failed: {1}, Passed: {2}, Skipped: {3}, Total: {4}, Duration: {5}

Why this is a real issue

A run with Failed = 0, Passed = 0, and Skipped > 0 is not a failed run, but this logic still prints Failed!. The same happens for zero-test runs when the platform does not treat them as an error. That produces contradictory MSBuild output: the task can succeed while the summary line claims the run failed.

Suggested resolution

Base the summary status on actual failure state instead of TotalPassed > 0, e.g. treat TotalFailed > 0 as failed and handle skipped-only / zero-test runs explicitly. Add an acceptance test that exercises a skipped-only run (and optionally a zero-test run) through InvokeTestingPlatform.

Related issues

  • None found

Metadata

Metadata

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions