databendlabs/databend · error · LookupError

ContextVar has no value

Error message

ContextVar {self.name} has no value

What it means

connect_in_process_grpc establishes an in-memory duplex transport for gRPC by sending the server half of the duplex through a channel to the endpoint task. If that channel's receiver has been dropped (endpoint shut down), the send fails and this io::Error (ConnectionRefused) is returned to the caller.

Solutions

  1. Ensure the endpoint task owning the receiver outlives all client connections.
  2. Check node shutdown ordering: stop clients before dropping the in-process endpoint.
  3. If this appears during shutdown, treat it as expected and retry against a healthy node.
  4. Investigate the endpoint drop path (e.g. handle closed early on error) if seen during normal operation.

Example fix

// before
drop(endpoint_handle);
let client = connect_in_process_grpc().await?;
// after
let client = connect_in_process_grpc().await?;
drop(endpoint_handle); // shut down after clients are done
Defensive patterns

Strategy: retry

Validate before calling

if endpoint_handle_is_dropped() { /* skip connecting */ } // no direct public probe; track endpoint liveness yourself

Try / catch

match connect_in_process_grpc().await { Err(e) if e.to_string().contains("in-process gRPC endpoint closed") && !shutting_down.load(Relaxed) => restart_endpoint_then_retry(), Err(e) => Err(e.into()), Ok(c) => Ok(c) }

Prevention

When it happens

Trigger: Calling connect() on an in-process gRPC client after the endpoint/server task has been dropped or shut down; a race where the endpoint is torn down while connections are still being established.

Common situations: Meta-service node shutting down while queries still hold a client; tests dropping the server handle before issuing RPCs; mis-ordered teardown of a standalone meta node.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/f782c071d383a4ab. Report an issue: GitHub.

Appendix: source

Thrown at src/query/script_udf_support/src/transform_udf_script.rs:639

                self.var = var
                self.old = old

        class ContextVar:
            __slots__ = ("name", "default", "_local")

            def __init__(self, name, *, default=_MISSING):
                self.name = name
                self.default = default
                self._local = threading.local()
                _REGISTERED_VARS.add(self)

            def get(self, default=_MISSING):
                value = getattr(self._local, "value", _MISSING)
                if value is _MISSING:
                    if default is not _MISSING:
                        return default
                    if self.default is _MISSING:
                        raise LookupError(f"ContextVar {self.name} has no value")
                    return self.default
                return value

            def set(self, value):
                old = getattr(self._local, "value", _MISSING)
                self._local.value = value
                return Token(self, old)

            def reset(self, token):
                if token.var is not self:
                    raise ValueError("Token does not belong to this ContextVar")
                if token.old is _MISSING:
                    if hasattr(self._local, "value"):
                        del self._local.value
                else:
                    self._local.value = token.old

        class Context:

View on GitHub (pinned to 288d84d76e)