I want to configure OpenTelemetry for Azure Service Bus. The documentation says it is built in. Let’s see how it behaves.

Using OpenTelemetry

Last year I did a fairly deep dive into using OpenTelemetry in .NET applications. OpenTelemetry is an open-source observability framework for collecting data about how software runs (metrics, logs, traces). It provides a standard (vendor-agnostic) way to instrument different programming languages so that this data can be easily analyzed by monitoring platforms.

To use OpenTelemetry with a .NET application, you include the following NuGet packages:

dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Extensions.Hosting

You configure OpenTelemetry by registering the services in host builder:

var builder = Host.CreateApplicationBuilder(args);

// Global settings
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resourceBuilder =>
    {
        resourceBuilder.AddService(builder.Environment.ApplicationName);
    });

// Logging
builder.Logging.AddOpenTelemetry(logging =>
{
    logging.IncludeFormattedMessage = true;
    logging.IncludeScopes = true;

    logging.AddOtlpExporter(configure =>
    {
        configure.Endpoint = new Uri("http://localhost:5341/ingest/otlp/v1/logs");
        configure.Protocol = OtlpExportProtocol.HttpProtobuf;
        configure.Headers = "X-Seq-ApiKey=XXXXXXXXXXXXXXX";
    });
});

// Tracing
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        // We want to view all traces
        tracing.SetSampler(new AlwaysOnSampler());

        tracing.AddOtlpExporter(exporter =>
        {
            exporter.Endpoint = new Uri("http://localhost:5341/ingest/otlp/v1/traces");
            exporter.Protocol = OtlpExportProtocol.HttpProtobuf;
            exporter.Headers = "X-Seq-ApiKey=XXXXXXXXXXXXXXX";
        });
    });

Please refer to the Seq documentation for the steps to generate an API Key: https://docs.datalust.co/docs/api-keys

Enabling Azure Service Bus Tracing

In the version of the SDK I am using (v7.20.1) including tracing data is still an experimental feature. You can enable it using a few options. I chose adding an environment variable:

AZURE_EXPERIMENTAL_ENABLE_ACTIVITY_SOURCE = true

Refer to this document for more details: https://devblogs.microsoft.com/azure-sdk/introducing-experimental-opentelemetry-support-in-the-azure-sdk-for-net/#get-started

First Impressions

The first thing I noticed was the logging level of the Azure Service Bus SDK is very noisy. Publishing a single message resulted in 2 log entries, and receiving a message created 5 log entries. These were not helpful for my use case, so I raised the logging level to “Warning” in the appsettings.json file:

{
    "Logging": {
        "LogLevel": {
            "Azure.Messaging.ServiceBus": "Warning"
        }
    }
}

When processing a message, I see a few activities auto-generated by the Azure Service Bus SDK:

  • ServiceBusSender.Send - sending the message to Azure Service Bus
  • Message - receiving the message
  • ServiceBusProcessor.ProcessMessage - processing the message
  • ServiceBusReceiver.Complete - flagging the message as complete
  • ServiceBusReceiver.Abandon - flagging the message as abandoned (when an exception occurs processing the message)

The other behavior I noted was during the shutdown of the application, the SDK would generate errors for task cancellation exceptions. I prefer error level log entries to only occur when something requires my attention. In this case, these errors don’t so I want to suppress them. I did that via a custom trace processor I found here:

https://github.com/Azure/azure-sdk-for-net/issues/49032#issuecomment-2743784548

public class TaskCanceledExceptionFilterProcessor : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        if (activity.Status == ActivityStatusCode.Error &&
            activity.StatusDescription.Contains("TaskCanceledException") &&
            activity.DisplayName == "ServiceBusReceiver.Receive")) // [Edit: only filter these activities for this exception]
        {
            // Suppress the activity                
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
        }
        base.OnEnd(activity);            
    }
}

tracing
    .AddSource("Azure.Messaging.ServiceBus.*")
    .AddProcessor(new TaskCanceledExceptionFilterProcessor());

For each message published by the application, there are 2 traces, and for each successfully consumed message there are 2 traces. as long as there is a parent activity when the message is published, the SDK can correlate the traces together under a common diagnostic id. This makes it possible to trace the flow of the message from publisher to consumer which is very helpful in a distributed system.

But I have to admit that these traces are not that helpful. The ProcessMessage provides details about the overall consumption of a message. If you have child activities, they are properly nested. The other 3 traces give telemetry on the time taken to send and receive the message. I think that these traces make sense in some scenarios, but not in all scenarios. I could see where having these disabled would be the desired state. If I was creating a custom library to handle messages, I would be better served to have my own custom telemetry, than to rely on the telemetry from the SDK.

Summary

Enabling OpenTelemetry with the Azure Service Bus SDK is possible. It provides telemetry on how the service bus is process the sending and receiving of messages. But it was not exactly what I wanted with respect to my application’s behavior. I think I’ll look at custom activities next.