{"record":{"id":"7b333373e023ac1a","repo":"headroomlabs-ai/headroom","slug":"metrics-encode-error-e","errorCode":null,"errorMessage":"metrics encode error: {e}","messagePattern":"metrics encode error: (.+?)","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"crates/headroom-proxy/src/observability/prometheus.rs","lineNumber":276,"sourceCode":"    rl_requests_gauge.with_label_values(&[INIT_SENTINEL]).set(0);\n    rl_tokens_gauge.with_label_values(&[INIT_SENTINEL]).set(0);\n    rl_input_gauge.with_label_values(&[INIT_SENTINEL]).set(0);\n    rl_output_gauge.with_label_values(&[INIT_SENTINEL]).set(0);\n    tier_counter.with_label_values(&[INIT_SENTINEL]).inc_by(0);\n    status_counter.with_label_values(&[INIT_SENTINEL]).inc_by(0);\n\n    let metric_families = registry().gather();\n    let mut buffer = Vec::with_capacity(2048);\n    let encoder = TextEncoder::new();\n    if let Err(e) = encoder.encode(&metric_families, &mut buffer) {\n        tracing::error!(\n            event = \"metrics_encode_failed\",\n            error = %e,\n            \"failed to encode Prometheus metrics scrape\"\n        );\n        return Response::builder()\n            .status(StatusCode::INTERNAL_SERVER_ERROR)\n            .body(Body::from(format!(\"metrics encode error: {e}\")))\n            .expect(\"static error response\");\n    }\n    Response::builder()\n        .status(StatusCode::OK)\n        .header(header::CONTENT_TYPE, encoder.format_type())\n        .body(Body::from(buffer))\n        .unwrap_or_else(|e| {\n            Response::builder()\n                .status(StatusCode::INTERNAL_SERVER_ERROR)\n                .body(Body::from(format!(\"metrics response build error: {e}\")))\n                .expect(\"static error response\")\n        })\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n","sourceCodeStart":258,"sourceCodeEnd":294,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/crates/headroom-proxy/src/observability/prometheus.rs#L258-L294","documentation":"The Prometheus TextEncoder failed while rendering the gathered metric families into the scrape response. Return path is a 500 with this text plus a tracing::error! (event = metrics_encode_failed). Encoder failures at this stage are rare and indicate a metric was registered with data the text format cannot represent.","triggerScenarios":"A metric family containing a label value or sample with characters/structure the Prometheus text exposition format rejects; a histogram/summary with NaN or negative counts; label values with embedded newlines; a custom metric registered with an invalid name.","commonSituations":"Label values derived from user input (model names, API keys) injected into metric labels without sanitization; a library registering metrics with names containing characters the encoder rejects; NaN counts from a race in a counter implementation.","solutions":["Check the metrics_encode_failed error log — the encoder names the offending family.","Sanitize label values: replace newlines/quotes/backslashes and constrain label cardinality at registration time, not render time.","Reproduce locally by hitting /metrics and bisecting which registered collector produces the bad family (disable collectors one at a time).","Upgrade the prometheus crate if the failure comes from its own default collectors."],"exampleFix":"// before: raw user-controlled label\ncounter.with_label_values(&[model_name]).inc();\n\n// after: sanitize label values\nfn sanitize_label(s: &str) -> String { s.replace(['\\n', '\"', '\\\\'], \"_\") }\ncounter.with_label_values(&[&sanitize_label(model_name)]).inc();","handlingStrategy":"validation","validationCode":"// Sanitize label values at registration time\nfn safe_label(s: &str) -> String {\n    s.chars().map(|c| if c.is_control() || matches!(c, '\"' | '\\\\' | '\\n') { '_' } else { c }).collect()\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never put raw user-controlled strings into metric labels — sanitize at registration.","Bound label cardinality (allowlist of label values).","Scrape /metrics in integration tests so encoder failures surface in CI, not prod."],"tags":["prometheus","metrics","observability","encoding","rust"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}