t8y2/dbx · error

all HiveServer2 endpoints failed: %s

Error message

all HiveServer2 endpoints failed: %s

What it means

This error is returned by discoveryConnector.Connect (connector.go:167) after the driver tried every HiveServer2 endpoint returned by ZooKeeper-based service discovery and each connection attempt failed. The message aggregates all per-endpoint failures, joined with "; ", so the underlying causes (timeouts, refusals, TLS errors) are visible in the message.

Source

Thrown at agents/drivers/hive-go/connector.go:167

				rejected[target.address()] = true
				failures = append(failures, fmt.Sprintf("attempt %d %s: %v", attempt+1, target.address(), connectErr))
			}
			break
		}
		if attempt+1 < connector.retries && connector.retryInterval > 0 {
			timer := time.NewTimer(connector.retryInterval)
			select {
			case <-ctx.Done():
				timer.Stop()
				return nil, ctx.Err()
			case <-timer.C:
			}
		}
	}
	if len(failures) == 0 {
		return nil, errors.New("Hive discovery returned no endpoints")
	}
	return nil, fmt.Errorf("all HiveServer2 endpoints failed: %s", strings.Join(failures, "; "))
}

func (connector *discoveryConnector) Driver() driver.Driver {
	return connector.driver
}

func openHiveDatabase(config connectionConfig) *sql.DB {
	database := sql.OpenDB(newDiscoveryConnector(config))
	database.SetMaxOpenConns(1)
	database.SetMaxIdleConns(1)
	return database
}

func normalizeHiveAuth(value string) string {
	normalized := strings.ToUpper(strings.TrimSpace(value))
	switch normalized {
	case "":
		return "NONE"

View on GitHub (pinned to c0390bff16)

Solutions

1. Read the joined failure details to identify the common cause (timeout vs refused vs TLS).
2. Verify HiveServer2 instances are up and reachable (nc/curl the host:port).
3. Check network/firewall rules between the client and the Hive cluster.
4. Confirm ZooKeeper znodes contain current, healthy server addresses.

Example fix

// before (stale znode pointing at dead host)
zookeeper=zk1:2181/hiveserver2
// after (verify HS2 is healthy and re-registered)
hive --service hiveserver2 # restart so it re-registers in ZooKeeper

addrs := strings.Split(host, ":")
for _, a := range zkEndpoints {
    conn, err := net.DialTimeout("tcp", a, 3*time.Second)
    if err != nil {
        return fmt.Errorf("zk endpoint unreachable: %s", a)
    }
    conn.Close()
}

conn, err := db.Connect(ctx)
if err != nil && strings.Contains(err.Error(), "all HiveServer2 endpoints failed") {
    // inspect per-endpoint causes in err.Error(), then retry with backoff
    return retryWithBackoff(ctx, 3, db.Connect)
}

When it happens

Trigger: Using ZooKeeper discovery mode, discovery returns >=1 endpoints, every endpoint connect attempt fails, and the failures list is non-empty (an empty list yields 'Hive discovery returned no endpoints' instead).

Common situations: All HiveServer2 instances down or being restarted; network/firewall blocking the driver from the cluster; stale ZooKeeper node entries pointing at dead hosts; TLS configuration mismatch with all servers.

Related errors

• connect to ZooKeeper: %w
• ZooKeeper event stream closed before a session was establish
• ZooKeeper session closed because SASL authentication is requ
• ZooKeeper server list is empty
• ZooKeeper Kerberos SASL requires Hive Kerberos credentials

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/869a3f89634dacf1. Report an issue: GitHub.