ZooKeeper event stream closed before a session was establish

Error message

ZooKeeper event stream closed before a session was established

What it means

While waiting for a ZooKeeper session, the events channel is closed before a session event is delivered. A closed event stream means the underlying ZooKeeper client stopped (connection closed or client terminated) without ever establishing a session, so the wait aborts with this error instead of blocking forever.

Source

Thrown at agents/drivers/argo-go/discovery.go:201

	}
	if len(cleaned) == 0 {
		return "/"
	}
	return "/" + strings.Join(cleaned, "/")
}

func waitForZooKeeperSession(ctx context.Context, events <-chan zk.Event, timeout time.Duration) error {
	timer := time.NewTimer(timeout)
	defer timer.Stop()
	for {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-timer.C:
			return errors.New("ZooKeeper connection timed out before a session was established")
		case event, ok := <-events:
			if !ok {
				return errors.New("ZooKeeper event stream closed before a session was established")
			}
			if event.Err != nil {
				return fmt.Errorf("ZooKeeper connection event: %w", event.Err)
			}
			switch event.State {
			case zk.StateHasSession:
				return nil
			case zk.StateAuthFailed:
				return errors.New("ZooKeeper authentication failed")
			case zk.StateExpired:
				return errors.New("ZooKeeper session expired during connection")
			}
		}
	}
}

func parseHiveServerRegistration(child string, data []byte) (endpoint, error) {
	candidates := []string{strings.TrimSpace(string(data)), strings.TrimSpace(child)}

View on GitHub (pinned to c0390bff16)

Solutions

Ensure nothing closes the ZooKeeper connection or client while Endpoints() is running
Re-create the ZooKeeper connection and retry discovery after an unexpected client shutdown
Check application shutdown ordering so discovery completes before teardown

Example fix

// before
go conn.Close() // closes while Endpoints is waiting
// after
endpoints, err := discovery.Endpoints(ctx) // then conn.Close() when done

Defensive patterns

Strategy: try-catch

Validate before calling

// ensure connection stays open for the duration
if conn == nil || conn.Closed() { return errors.New("zookeeper connection already closed") }

Try / catch

endpoints, err := discovery.Endpoints(ctx)
if err != nil {
	if strings.Contains(err.Error(), "event stream closed") {
		// reconnect and retry; do not close conn concurrently
	}
}

Prevention

Never call conn.Close() concurrently with Endpoints(); sequence discovery before shutdown
Use single-owner lifecycle management for the ZooKeeper connection
Add shutdown ordering: finish discovery, then close the client

When it happens

Trigger: Endpoints() -> waitForZooKeeperSession receives ok=false from the events channel, e.g. the connection was closed by another goroutine or the client shut down before a session was established.

Common situations: Calling Close() on the ZooKeeper connection concurrently with discovery; the ZooKeeper client process crashing or being torn down by a shutdown hook; premature context cleanup closing the connection.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/71e164d6870d395c. Report an issue: GitHub.