tracing_subscriber offers a lot of configurability in how trace events are output. Here are some practical examples of of using multiple layers for use in a ratatui application.

Before the ratatui application starts and after it exists events are written to stdout. Also, tracing is used to submit OpenTelemetry traces.

Crate features

# …
[dependencies]

tracing = "0.1.40"
tracing-subscriber = {
    version = "0.3.18",
    features = ["env-filter", "time", "local-time"]
}

Toggleable stdout layer

Let's start with a layer that writes events to stdout that when an AtomicBool is true. This will allow us to turn the layer off when the ratatui application starts, then back on when it exits.

/// Convenience type alias
type BoxedLayer = Box<dyn Layer<Registry> + Send + Sync>;

/// Log to stdout when tui_active is false
fn stdout_layer(tui_active: &Arc<AtomicBool>) -> BoxedLayer {
    let timer = OffsetTime::local_rfc_3339().
        expect("could not get local offset!")

    let stdout = fmt::layer()
        .with_ansi(std::io::stdout().is_terminal())
        .with_timer(timer);

    let stdout_tui_active = tui_active.clone();

    stdout
        .with_filter(filter_fn(move |_| {
            !stdout_tui_active.load(Ordering::Relaxed)
        }))
        .boxed()
}

The fmt module offers many options for configuring stdout output. We then apply a filter that only enables the layer when tui_active is true and return a boxed layer to make it easy to combine types with any other layers.

Filtering stdout

The layer we've built will emit all events from all traced crates which will usually be more than desired. Adding an EnvFilter lets us set a good default and control what gets emitted at runtime. This filter stacks with the tui_active filter.

/// Filter from RUST_LOG environment variable that defaults to INFO if missing
/// or invalid
fn log_filter() -> EnvFilter {
    EnvFilter::builder()
        .with_default_directive(LevelFilter::INFO.into())    
        .from_env_lossy()
}

Now we need to compose the EnvFilter with the stdout_layer(). (We'll be updating this function as we add more layers.)

pub fn tracing() -> Arc<AtomicBool> {
    let tui_active = Arc::new(AtomicBool::new(false));

    let log_filter = log_filter();

    let log = log_layer(&tui_active).with_filter(log_filter());

    tracing_subscriber::registry()
        .with(log)
        .init();

    tui_active
}

OpenTelemetry tracing

Trace output sent to an OpenTelemetry collector can be much more verbose than output sent to the console to aid in debugging. To avoid storing too much data the traces can be sampled so only 1% of traces are recorded. For this example we'll output all traces.

Resource

To connect tracing to OpenTelemetry we need an SdkTracerProvider which needs a Resource describing our application:

use opentelemetry_semantic_conventions::{
    resource::SERVICE_VERSION,
    trace::PROCESS_PID,
    SCHEMA_URL,
};

pub fn resource() -> Resource {
    let pid = std::process::id();

    Resource::builder()
        .with_service_name(env!("CARGO_PKG_NAME"))
        .with_schema_url(
            [
                KeyValue::new(PROCESS_PID, Value::I64(pid.into())),
                KeyValue::new(SERVICE_VERSION, env!("CARGO_PKG_VERSION")),
            ],
            SCHEMA_URL,
        )
        .build()
}

There are many more fields in opentelemetry_semantic_conventions and a full guide on usage.

Tracer provider

The SdkTracerProvider process spans from the application and traced crates.

fn tracer_provider(resource: &Resource) -> SdkTracerProvider {
    // Allows connecting traces when acting as a server or client
    set_text_map_propagator(TraceContextPropagator::new());

    /// OTLP HTTP span exporter
    let exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_http()
        .build()
        .unwrap();

    SdkTracerProvider::builder()
        // Export every trace
        .with_sampler(Sampler::ParentBased(Box::new(Sampler::TraceIdRatioBased(
            1.0,
        ))))
        .with_id_generator(RandomIdGenerator::default())
        // The resource we created above
        .with_resource(resource.clone())
        .with_batch_exporter(exporter)
        .build()
}

The text map propagator allows connecting traces between services. (This is also used for connecting a service graph.)

The span exporter sets how you want spans to be sent to an Opentelemetry collector.

OpenTelemetry layer

To connect the tracing spans to OpenTelemetry we create an OpenTelemetryLayer to connect them:

pub fn tracer_layer(
    tracer_provider: &SdkTracerProvider,
) -> OpenTelemetryLayer<
    Layered<OtelFiltered, Layered<BoxedLayer, Registry>>,
    opentelemetry_sdk::trace::Tracer,
> {
    let tracer = tracer_provider.tracer(env!("CARGO_PKG_NAME"));
    OpenTelemetryLayer::new(tracer)
}

The OpenTelemetryLayer has a number of additional configuration options for controlling which information will be exported. The default is to export almost everything.

Register with tracing_subscriber

Now we can create the SdkTracerProvider, create a Layer from it, and register it with tracing_subscriber. The SdkTracerProvider must be held until the program exits, or is manually shut down, so it also needs to be returned.

pub fn tracing() -> (Arc<AtomicBool>, SdkTracerProvider) {
    let tui_active = Arc::new(AtomicBool::new(false));

    let log_filter = log_filter();

    let log = log_layer(&tui_active).with_filter(log_filter());

    let resource = resource();

    let otel_tracer_provider = tracer_provider(&resource);

    tracing_subscriber::registry()
        .with(log)
        .with(tracer_layer(&otel_tracer_provider))
        .init();

    (tui_active, otel_tracer_provider)
}

Shutdown

When you drop the SdkTracerProvider it will shut down automatically, but you can also shut it down manually to check for errors:

let (tui_active, otel_tracer_provider) = tracing();

// application code

if let Err(error) = otel_tracer_provider.shutdown() {
    warn!(?error, "OTEL tracer shutdown failed");
}