Skip to content

[MTP Reporters Analysis] AzureDevOps live publishing drops dequeued results on publish failure #8418

@Evangelink

Description

@Evangelink

Summary

AzureDevOpsTestResultsPublisher permanently loses a batch of live test results when PublishTestResultsAsync fails. The code dequeues results before publishing them and only logs on failure; it never requeues or preserves the failed batch for the forced final flush.

Evidence

  • src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsPublisher.cs:336-355
    List<AzureDevOpsTestCaseResult> batch = [];
    while (batch.Count < _options.BatchSize && _pendingResults.TryDequeue(out AzureDevOpsTestCaseResult? result))
    {
        batch.Add(result);
    }
    ...
    await _client.PublishTestResultsAsync(_publishConfiguration, CurrentRunId.Value, batch, cancellationToken).ConfigureAwait(false);
    _lastFlushTime = _clock.UtcNow;
  • On failure it only logs in src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsPublisher.cs:357-359
    catch (Exception ex) when (ex is not OperationCanceledException)
    {
        _logger.LogWarning($"{AzureDevOpsResources.AzureDevOpsLivePublishingPublishResultsFailed} {ex.Message}");
    }

Why this is a real issue

If Azure DevOps rejects one request or the network fails after the batch has been dequeued, those results are gone from _pendingResults. The publisher keeps running and the final forced flush has nothing left to resend, so the live run can finish with missing test results even though the extension only logged a warning.

Suggested resolution

Keep batches durable until the publish call succeeds:

  • requeue the batch on failure, or
  • only remove items from the pending buffer after a successful publish.

A max-retry / poison-batch strategy would still be better than silently dropping the data.

Related issues

Metadata

Metadata

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions