ZooKeeper event stream closed before a session was establish

Error message

ZooKeeper event stream closed before a session was established

What it means

The ZooKeeper client's event channel was closed before any session event arrived, so waitForZooKeeperSession cannot observe connection state and stops with this error. It indicates the underlying zk connection was shut down or the driver closed the event stream unexpectedly.

Source

Thrown at agents/drivers/hive-go/discovery.go:201

	}
	if len(cleaned) == 0 {
		return "/"
	}
	return "/" + strings.Join(cleaned, "/")
}

func waitForZooKeeperSession(ctx context.Context, events <-chan zk.Event, timeout time.Duration) error {
	timer := time.NewTimer(timeout)
	defer timer.Stop()
	for {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-timer.C:
			return errors.New("ZooKeeper connection timed out before a session was established")
		case event, ok := <-events:
			if !ok {
				return errors.New("ZooKeeper event stream closed before a session was established")
			}
			if event.Err != nil {
				return fmt.Errorf("ZooKeeper connection event: %w", event.Err)
			}
			switch event.State {
			case zk.StateHasSession:
				return nil
			case zk.StateAuthFailed:
				return errors.New("ZooKeeper authentication failed")
			case zk.StateExpired:
				return errors.New("ZooKeeper session expired during connection")
			}
		}
	}
}

func parseHiveServerRegistration(child string, data []byte) (endpoint, error) {
	candidates := []string{strings.TrimSpace(string(data)), strings.TrimSpace(child)}

View on GitHub (pinned to c0390bff16)

Solutions

Ensure the ZooKeeper connection is not closed while Endpoints is running (check for early defer Close or double Close)
Inspect for fatal client errors preceding the channel close and surface them
Upgrade/verify the go-zk dependency for known channel-close behavior changes
Serialize Endpoints calls so concurrent invocations don't tear down a shared connection

Example fix

// before
go func() { connection.Close() }() // closes while Endpoints waits
connection, events, _ := zk.Connect(...)
// after
connection, events, _ := zk.Connect(...)
defer connection.Close() // close only after Endpoints returns

Defensive patterns

Strategy: try-catch

Try / catch

endpoints, err := discovery.Endpoints(ctx)
if err != nil && strings.Contains(err.Error(), "event stream closed") {
    // recreate the zk connection and retry once
    return recreateConnectionAndDiscover(ctx)
}

Prevention

Never close the zk.Conn while Endpoints is in flight; use defer after all use
Guard against concurrent Endpoints calls on a shared connection
Check the go-zk version for known event-channel close semantics

When it happens

Trigger: Calling Endpoints when the zk.Conn is closed concurrently (Close called, or the client tears down after a fatal error) while the session wait is still consuming events.

Common situations: Parent context cancelation code path that closes the connection but not the events channel, double-Close of the discovery, or an older go-zk version closing the channel after connection errors.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/1bdbab989ded0dda. Report an issue: GitHub.