headroomlabs-ai/headroom · error

metrics encode error: {e}

Error message

metrics encode error: {e}

What it means

The Prometheus TextEncoder failed while rendering the gathered metric families into the scrape response. Return path is a 500 with this text plus a tracing::error! (event = metrics_encode_failed). Encoder failures at this stage are rare and indicate a metric was registered with data the text format cannot represent.

Source

Thrown at crates/headroom-proxy/src/observability/prometheus.rs:276

    rl_requests_gauge.with_label_values(&[INIT_SENTINEL]).set(0);
    rl_tokens_gauge.with_label_values(&[INIT_SENTINEL]).set(0);
    rl_input_gauge.with_label_values(&[INIT_SENTINEL]).set(0);
    rl_output_gauge.with_label_values(&[INIT_SENTINEL]).set(0);
    tier_counter.with_label_values(&[INIT_SENTINEL]).inc_by(0);
    status_counter.with_label_values(&[INIT_SENTINEL]).inc_by(0);

    let metric_families = registry().gather();
    let mut buffer = Vec::with_capacity(2048);
    let encoder = TextEncoder::new();
    if let Err(e) = encoder.encode(&metric_families, &mut buffer) {
        tracing::error!(
            event = "metrics_encode_failed",
            error = %e,
            "failed to encode Prometheus metrics scrape"
        );
        return Response::builder()
            .status(StatusCode::INTERNAL_SERVER_ERROR)
            .body(Body::from(format!("metrics encode error: {e}")))
            .expect("static error response");
    }
    Response::builder()
        .status(StatusCode::OK)
        .header(header::CONTENT_TYPE, encoder.format_type())
        .body(Body::from(buffer))
        .unwrap_or_else(|e| {
            Response::builder()
                .status(StatusCode::INTERNAL_SERVER_ERROR)
                .body(Body::from(format!("metrics response build error: {e}")))
                .expect("static error response")
        })
}

#[cfg(test)]
mod tests {
    use super::*;

View on GitHub (pinned to 322425c43b)

Solutions

1. Check the metrics_encode_failed error log — the encoder names the offending family.
2. Sanitize label values: replace newlines/quotes/backslashes and constrain label cardinality at registration time, not render time.
3. Reproduce locally by hitting /metrics and bisecting which registered collector produces the bad family (disable collectors one at a time).
4. Upgrade the prometheus crate if the failure comes from its own default collectors.

Example fix

// before: raw user-controlled label
counter.with_label_values(&[model_name]).inc();

// after: sanitize label values
fn sanitize_label(s: &str) -> String { s.replace(['\n', '"', '\\'], "_") }
counter.with_label_values(&[&sanitize_label(model_name)]).inc();

// Sanitize label values at registration time
fn safe_label(s: &str) -> String {
    s.chars().map(|c| if c.is_control() || matches!(c, '"' | '\\' | '\n') { '_' } else { c }).collect()
}

When it happens

Trigger: A metric family containing a label value or sample with characters/structure the Prometheus text exposition format rejects; a histogram/summary with NaN or negative counts; label values with embedded newlines; a custom metric registered with an invalid name.

Common situations: Label values derived from user input (model names, API keys) injected into metric labels without sanitization; a library registering metrics with names containing characters the encoder rejects; NaN counts from a race in a counter implementation.

Related errors

• metrics response build error: {e}
• magika session lock poisoned
• could not read pipeline config file: {0}
• bedrock_upstream_exception
• bedrock_eventstream_missing_event_type

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/7b333373e023ac1a. Report an issue: GitHub.