databendlabs/databend · error

heartbeat request should contain auth header

Error message

heartbeat request should contain auth header

What it means

The HTTP heartbeat handler forwards heartbeat requests to other cluster nodes, and it copies the incoming request's `AUTHORIZATION` header into the forwarded headers via `.expect("heartbeat request should contain auth header")`. If the client's heartbeat request lacks an Authorization header, this panics inside the axum/poem handler. Databend assumes heartbeats always carry credentials because cluster communication is authenticated, so a bare heartbeat is treated as a bug in the caller.

Solutions

1. Add credentials to the heartbeat caller: use `curl -u user:password ...` or set basic-auth in the probe/script configuration.
2. Fix Kubernetes probes/monitoring to embed the auth header (or use a separate unauthenticated health endpoint if available).
3. Check any proxy/load balancer in front of Databend to ensure it does not strip the Authorization header.
4. Harden the handler to return 401 instead of panicking when the header is absent (wrap in `ok_or_else` → `ErrorCode`/HTTP error response).

Example fix

// before
req.headers()
    .get(http::header::AUTHORIZATION)
    .expect("heartbeat request should contain auth header")
    .to_owned(),

// after
req.headers()
    .get(http::header::AUTHORIZATION)
    .ok_or_else(|| ErrorCode::Unauthenticated(
        "heartbeat request should contain auth header"
    ))?
    .to_owned(),

// Client-side check before sending heartbeat
curl -s -o /dev/null -w "%{http_code}" -u "${DATABEND_USER}:${DATABEND_PASSWORD}" \
  http://host:8080/v1/heartbeat || echo "heartbeat auth missing/failed"

function heartbeatRequestHasAuth(headers) {
  return typeof headers.authorization === 'string' && headers.authorization.length > 0;
}

// Server-side hardening
let auth = match req.headers().get(http::header::AUTHORIZATION) {
    Some(h) => h.to_owned(),
    None => return Ok((StatusCode::UNAUTHORIZED, "heartbeat requires auth header")),
};

When it happens

Trigger: Sending `PUT /v1/heartbeat` (or the configured heartbeat endpoint) without an `Authorization` header, e.g., a custom health-check script, load balancer probe, or monitoring agent hitting the endpoint directly with no `--user`/auth configured.

Common situations: Kubernetes liveness/readiness probes configured without basic-auth; curl one-liners omitting `-u user:password`; ops scripts written against an older deployment where auth was disabled; reverse proxies stripping the Authorization header.

Related errors

• login_handler expect session id in ctx
• mysql handler should be authed when call
• {}
• Temp table id used up
• Invalid temp table desc

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/a58c879d7fb573be. Report an issue: GitHub.

Appendix: source

Thrown at src/query/service/src/servers/http/v1/http_query_handlers.rs:988

    if num_task > 0 {
        let mut tasks = Vec::with_capacity(num_task);
        let uri = req.uri().to_string();
        let method = req.method();
        let mut headers = HeaderMap::new();
        headers.insert(
            http::header::CONTENT_TYPE,
            HeaderValue::from_static("application/json"),
        );
        let agent = format!("databend-query/{}", ctx.version.semantic);
        headers.insert(
            http::header::USER_AGENT,
            HeaderValue::from_str(&agent).unwrap(),
        );
        headers.insert(
            http::header::AUTHORIZATION,
            req.headers()
                .get(http::header::AUTHORIZATION)
                .expect("heartbeat request should contain auth header")
                .to_owned(),
        );
        for (node, body) in nodes_to_forwards {
            let uri = uri.clone();
            let method = method.clone();
            let headers = headers.clone();

            tasks.push(async move {
                match forward_request_with_body(node, &uri, body, method, headers).await {
                    Ok(mut resp) => {
                        if resp.status() == StatusCode::OK {
                            Some(
                                resp.take_body()
                                    .into_json::<HeartBeatResponse>()
                                    .await
                                    .unwrap(),
                            )
                        } else {

View on GitHub (pinned to 288d84d76e)