vitessio/vitess · error

local zk unhealthy: %v %v

Error message

local zk unhealthy: %v %v

What it means

Zkd.Start launches zookeeper and then polls the four-letter 'ruok' command; while any reply other than "imok" is received the error is overwritten with this message. It means the local zookeeper answered but is not healthy (or still starting) after the retry loop.

Source

Thrown at go/vt/zkctl/zkctl.go:115

	// give it some time to succeed - usually by the time the socket emerges
	// we are in good shape, but not always. So let's continue to retry until
	// we get an imok response from the socket or we timeout.
	timeout := time.Now().Add(startWaitTime)
	zkAddr := fmt.Sprintf(":%v", zkd.config.ClientPort)
	for time.Now().Before(timeout) {
		conn, connErr := net.Dial("tcp", zkAddr)
		if connErr != nil {
			err = connErr
		} else {
			conn.Write([]byte("ruok"))
			reply := make([]byte, 4)
			conn.Read(reply)
			conn.Close()
			if string(reply) == "imok" {
				err = nil
				break
			}
			err = fmt.Errorf("local zk unhealthy: %v %v", zkAddr, reply)
		}
		time.Sleep(time.Second)
	}
	if err != nil {
		return err
	}
	zkd.done = make(chan struct{})
	go func(done chan<- struct{}) {
		// wait so we don't get a bunch of defunct processes
		cmd.Wait()
		close(done)
	}(zkd.done)
	return err
}

// Shutdown kills a ZooKeeper server, but keeps its data dir intact.
func (zkd *Zkd) Shutdown() error {
	log.Info("zkctl.Shutdown")

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check the zk log files under $VTTOP/vt_<uid>/zk logs for the real failure
2. Delete/reinitialize a corrupted zk data directory (`zkctl teardown` then `init`)
3. Wait/retry if the ensemble was mid-election; verify quorum config (server list) is correct

conn, _ := net.Dial("tcp", zkAddr)
if conn != nil {
    conn.Write([]byte("ruok"))
    reply := make([]byte, 16)
    conn.Read(reply)
    if string(reply) != "imok" { return fmt.Errorf("zk not healthy yet: %s", zkAddr) }
}

if err := zkd.Start(); err != nil {
    if strings.Contains(err.Error(), "local zk unhealthy") {
        // check zk logs, wait for quorum, retry once
        time.Sleep(5 * time.Second)
        err = zkd.Start()
    }
}

When it happens

Trigger: Calling zkd.Start() (directly or via zkctl Init) when zookeeper starts but responds to 'ruok' with something other than 'imok' before the loop exhausts — e.g. zk in a broken/leaderless state or still electing.

Common situations: Zookeeper data dir corrupted or with permission problems; ensemble cannot reach quorum; very slow machine exceeding the startup poll window; wrong client port in zk config.

Related errors

• both the dry-run mode and actual buffering is enabled. To av
• failed to load flag for %s: %w
• failed to parse tablet-filters value %q: %v
• could not find tablet: %s
• health stats is not valid: %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/8f5ec30a12a94daa. Report an issue: GitHub.