ASBS "Dead Letter Queue" custom check can never pass with Entra ID / managed identity authentication

[markgould]

Describe the bug

When ServiceControl runs on the Azure Service Bus transport authenticated with a managed identity, the transport itself works correctly, but the "Dead Letter Queue" custom check (category: Transport) fails on every run. The check constructs its client directly from the raw connection string:

// src/ServiceControl.Transports.ASBS/DeadLetterQueueCheck.cs (line 34 in 6.18.2; still present on master)
var managementClient = new ServiceBusAdministrationClient(connectionString);

That constructor only supports SAS credentials, so the check bypasses the ASBS transport's own authentication machinery (ConnectionStringParser / AuthenticationMethod / TokenCredentialAuthentication) that makes managed identity work everywhere else. Both documented managed-identity connection string forms fail:

• ConnectionString = bare fully-qualified namespace (<ns>.servicebus.windows.net): the transport connects fine, but the check throws a FormatException from the ServiceBusAdministrationClient constructor.
• ConnectionString = Endpoint=sb://<ns>.servicebus.windows.net/;Authentication=Managed Identity;ClientId=<clientId> (the documented managed-identity form, which the transport accepts): the transport connects fine, but the check throws System.ArgumentException from the same constructor — the connection-string-only constructor has no token-credential path, and on a namespace with disableLocalAuth=true there is no SAS to fall back to.

On an AAD-only namespace (disableLocalAuth=true, which organizational policy increasingly mandates) the check is therefore permanently red in ServicePulse. We found no configuration switch to suppress it; it re-fails hourly, which trains operators to ignore the one check that would flag real message loss in the <endpoint>.staging DLQ.

Observed error

With the documented managed-identity connection string form, every check run (hourly, per repeatAfter) logs:

Error|ServiceControl.CustomChecks.InternalCustomChecksHostedService|`ServiceControl.Transports.ASBS.DeadLetterQueueCheck` implementation failed to run|System.ArgumentException: The connection string used for an Service Bus client must specify the Service Bus namespace host and either a Shared Access Key (both the name and value) OR a Shared Access Signature to be valid. (Parameter 'connectionString')
   at Azure.Messaging.ServiceBus.Administration.ServiceBusAdministrationClient..ctor(String connectionString, ServiceBusAdministrationClientOptions options)
   at Azure.Messaging.ServiceBus.Administration.ServiceBusAdministrationClient..ctor(String connectionString)
   at ServiceControl.Transports.ASBS.DeadLetterQueueCheck.PerformCheck(CancellationToken cancellationToken) in /_/src/ServiceControl.Transports.ASBS/DeadLetterQueueCheck.cs:line 34
   at ServiceControl.CustomChecks.InternalCustomCheckManager.Run(CancellationToken cancellationToken) in /_/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomCheckManager.cs:line 43

ServicePulse shows the check as failed with "ServiceControl.Transports.ASBS.DeadLetterQueueCheck implementation failed to run."

Versions and environment

• ServiceControl 6.18.2 (error + audit + monitoring instances), ServicePulse 2.9.1
• Official Docker images (particular/servicecontrol*) on Azure Container Apps, --setup-and-run
• Transport NetStandardAzureServiceBus; namespace: Standard tier, disableLocalAuth=true
• User-assigned managed identity with Azure Service Bus Data Owner on the namespace; ClientId pinned in the connection string

Everything else works with this setup: ingestion (error + audit + monitoring), queue creation at setup, retries. Only this check fails.

To reproduce

1. Create an ASB namespace with local auth disabled (disableLocalAuth=true) and grant a managed identity Azure Service Bus Data Owner.
2. Run the ServiceControl error instance container with TRANSPORTTYPE=NetStandardAzureServiceBus and CONNECTIONSTRING=Endpoint=sb://<ns>.servicebus.windows.net/;Authentication=Managed Identity;ClientId=<clientId>.
3. Observe the instance run normally, then the "Dead Letter Queue" custom check fail on its first run (hourly thereafter). With the bare-FQNS connection string form, the instance logs a FormatException from the check instead.

Expected behavior

The check should honor the transport's configured authentication method — e.g. construct ServiceBusAdministrationClient(fullyQualifiedNamespace, tokenCredential) when the connection settings resolve to token-credential auth, exactly as the transport connection does. The parsing/credential machinery already exists in ServiceControl.Transports.ASBS; the check just doesn't use it.

A possible CI note: the ASBS transport-test leg currently provisions a SAS connection string, so this path has no coverage. Since ci.yml already runs azure/login before the transport tests, a managed-identity variant could plausibly reuse the runner's credential via DefaultAzureCredential against the provisioned namespace.

Related

• Azure Service Bus Managed Identity is not supported #2755 fixed Authentication=Managed Identity for the transport connection at startup (closed, 4.21.8) but did not cover this custom check.