error connecting to session recorders: %v; failure mode is '

Error message

error connecting to session recorders: %v; failure mode is 'fail closed'; closing connection.

What it means

Returned by the Kubernetes operator's session-recording proxy hijacker (k8s-operator/sessionrecording/hijacker.go:178) when connectToRecorder failed for all recorder addresses and the proxy is configured fail-closed (h.failOpen == false). Because recording is mandatory in this mode, the proxied exec/attach connection is torn down after attempting to send the warning message to the client. The returned error is the composed log-style message including the underlying dial error.

Source

Thrown at k8s-operator/sessionrecording/hijacker.go:178

	qp := h.req.URL.Query()
	container := strings.Join(qp[containerKey], "")
	var recorderAddr net.Addr
	trace := &httptrace.ClientTrace{
		GotConn: func(info httptrace.GotConnInfo) {
			recorderAddr = info.Conn.RemoteAddr()
		},
	}
	wc, _, errChan, err = h.connectToRecorder(httptrace.WithClientTrace(ctx, trace), h.addrs, h.ts.Dial)
	if err != nil {
		msg := fmt.Sprintf("error connecting to session recorders: %v", err)
		if h.failOpen {
			msg = msg + "; failure mode is 'fail open'; continuing session without recording."
			h.log.Warnf(msg)
			return conn, nil
		}
		msg = msg + "; failure mode is 'fail closed'; closing connection."
		if err := closeConnWithWarning(conn, msg); err != nil {
			return nil, errors.Join(errors.New(msg), err)
		}
		return nil, errors.New(msg)
	} else {
		h.log.Infof("%s session to container %q in Pod %q namespace %q will be recorded, the recording will be sent to a tsrecorder instance at %q", h.sessionType, container, h.pod, h.ns, recorderAddr)
	}

	cl := tstime.DefaultClock{}
	rec := tsrecorder.New(wc, cl, cl.Now(), h.failOpen, h.log)
	tty := strings.Join(qp[ttyKey], "")
	hasTerm := (tty == "true") // session has terminal attached
	ch := sessionrecording.CastHeader{
		Version:   asciicastv2,
		Timestamp: cl.Now().Unix(),
		Command:   strings.Join(qp[commandKey], " "),
		SrcNode:   strings.TrimSuffix(h.who.Node.Name, "."),
		SrcNodeID: h.who.Node.StableID,
		Kubernetes: &sessionrecording.Kubernetes{
			PodName:     h.pod,

View on GitHub (pinned to cfe32b8be6)

Solutions

Verify tsrecorder is running and the proxy can reach it: kubectl get pods for tsrecorder and test TLS to the address in the ProxyGroup recording config.
Fix the recorder addresses/TLS config in the ProxyGroup/proxy-class spec and ensure DNS resolves them from the operator/proxy pod.
Relax network policies/firewall rules so the operator can dial the recorder port.
If unrecorded sessions are acceptable in your threat model, set failureMode to fail-open (the source then logs and continues without recording) — an explicit risk decision.

Example fix

# before: recorder down/unreachable, fail-closed kills every exec
recording: true # failureMode defaults to fail-closed
# after: keep fail-closed but make the recorder reachable (preferred)
kubectl -n tsrecorder get pods # must be Ready
kubectl -n tailscale exec deploy/proxy -- wget -q -O- https://<recorder>:443/healthz
# only if unrecorded sessions are acceptable:
# set failureMode: fail-open in the ProxyGroup spec

Defensive patterns

Strategy: validation

Validate before calling

// Health-check recorders before enabling mandatory recording (fail-closed).
for _, addr := range recorderAddrs {
	c := tls.Dialer{Timeout: 3 * time.Second}
	if conn, err := c.DialContext(ctx, "tcp", addr); err == nil {
		conn.Close()
		continue
	}
	return fmt.Errorf("recorder %s unreachable; exec would be rejected in fail-closed mode", addr)
}

Try / catch

if err := proxy.Hijack(...); err != nil {
	var msg string
	if errors.As(err, &joined) { msg = joined.Error() } else { msg = err.Error() }
	if strings.Contains(msg, "fail closed") {
		// alert: recording is mandatory and recorders are down; fix recorder reachability
	}
}

Prevention

Run tsrecorder with >=2 replicas and monitor its availability like a critical dependency.
Smoke-test recorder reachability (TLS + healthz) from the operator namespace on deploy.
Explicitly decide failureMode per environment; fail-closed means recorder outage blocks exec.
Alert on the 'error connecting to session recorders' log line.

When it happens

Trigger: A kubectl exec/cp/attach request on a ProxyGroup with session recording configured (recording: true) and failureMode fail-closed (the default) while no tsrecorder instance is reachable: wrong recorder addresses, tsrecorder pod down, NetworkPolicy/firewall blocking, TLS/SNI mismatch, or DNS for the recorder headless service failing.

Common situations: tsrecorder deployment scaled to zero or crashed; proxy class pointing at stale recorder FQDNs; cert rotation on tsrecorder making connections fail; cluster network policy added after initial setup; first-time install where recorder was never verified.

Related errors

AI-assisted analysis of tailscale/tailscale@cfe32b8be6 (2026-08-15). Data as JSON: /api/errors/d42dc763395dec8a. Report an issue: GitHub.