thanos-io/thanos · error
failed to bootstrap capnp writer
Error message
failed to bootstrap capnp writer
What it means
connect() dials a capnp connection and bootstraps the remote Writer capability. After NewConn, it obtains the bootstrap capability via r.conn.Bootstrap(ctx) and calls writer.Resolve(ctx); if the remote side fails to serve the bootstrap interface (or the context dies before resolution), the connection is torn down and this wrapped error is returned.
Solutions
- Check the receiver address/port in the receive config points at the capnp writer endpoint and that the remote service is up
- Increase the context timeout passed to connect/writeWithReconnect
- Verify network connectivity (firewalls, load balancers) between sender and receiver
- Retry the write — writeWithReconnect will re-dial; ensure reconnect backoff is enabled
- Check receiver logs for bootstrap/rpc errors at the same timestamp
Example fix
// before ctx := context.Background() err := client.Write(ctx, req) // hangs until some inner timeout, then bootstrap fails // after ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) defer cancel() err := client.Write(ctx, req)
Defensive patterns
Strategy: retry
Validate before calling
// before writing, verify the endpoint is reachable
conn, err := net.DialTimeout("tcp", addr, 5*time.Second)
if err != nil { return fmt.Errorf("capnp endpoint %s unreachable: %w", addr, err) }
conn.Close() Try / catch
var be *errors.Error
if errors.As(err, &be) && strings.Contains(be.Error(), "failed to bootstrap capnp writer") {
// re-dial with backoff; check ctx deadline first
select { case <-ctx.Done(): return ctx.Err(); default: }
time.Sleep(backoff)
return retry()
} Prevention
- Use a context timeout that exceeds expected dial+bootstrap time
- Health-check the receiver before enabling writes
- Enable writeWithReconnect with exponential backoff
- Monitor receiver availability/logs for bootstrap failures
When it happens
Trigger: The remote capnp server (Cortex receiver) did not answer the bootstrap RPC within ctx's deadline, the remote returned an exception for the bootstrap capability, the connection was reset immediately after dial, or ctx was canceled before Resolve completed.
Common situations: Remote receiver not actually a capnp server on that port; wrong port in config; receiver overloaded or restarting; network firewall silently dropping the stream; context timeout too short under load.
Understand the failure class
Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.
Related errors
AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07).
Data as JSON: /api/errors/da76c28ea371ca0d.
Report an issue: GitHub.
Appendix: source
Thrown at pkg/receive/writecapnp/client.go:213
}
func (r *RemoteWriteClient) connect(ctx context.Context) error {
r.mu.Lock()
defer r.mu.Unlock()
if r.conn != nil {
return nil
}
conn, err := r.dialer.DialContext(ctx)
if err != nil {
return errors.Wrap(err, "failed to dial peer")
}
r.conn = rpc.NewConn(rpc.NewPackedStreamTransport(conn), nil)
writer := Writer(r.conn.Bootstrap(ctx))
if err := writer.Resolve(ctx); err != nil {
level.Warn(r.logger).Log("msg", "failed to bootstrap capnp writer, closing connection", "err", err)
r.closeUnlocked()
return errors.Wrap(err, "failed to bootstrap capnp writer")
}
r.writer = writer
return nil
}
func (r *RemoteWriteClient) Close() error {
r.mu.Lock()
r.closeUnlocked()
r.mu.Unlock()
return nil
}
func (r *RemoteWriteClient) closeUnlocked() {
if r.conn != nil {
conn := r.conn
r.conn = nil
go conn.Close()View on GitHub (pinned to 35b8b99117)