t8y2/dbx · error

all HiveServer2 endpoints failed: %s

Error message

all HiveServer2 endpoints failed: %s

What it means

The discovery-based Hive connector tries every HiveServer2 endpoint returned by service discovery (ZooKeeper); if all connection attempts fail, it aggregates the per-endpoint failure messages and returns this error joined with '; '. The empty-failures case (no endpoints recorded) raises 'Hive discovery returned no endpoints' instead. This error tells the caller none of the discovered servers were reachable or accepted the connection.

Source

Thrown at agents/drivers/argo-go/connector.go:167

				rejected[target.address()] = true
				failures = append(failures, fmt.Sprintf("attempt %d %s: %v", attempt+1, target.address(), connectErr))
			}
			break
		}
		if attempt+1 < connector.retries && connector.retryInterval > 0 {
			timer := time.NewTimer(connector.retryInterval)
			select {
			case <-ctx.Done():
				timer.Stop()
				return nil, ctx.Err()
			case <-timer.C:
			}
		}
	}
	if len(failures) == 0 {
		return nil, errors.New("Hive discovery returned no endpoints")
	}
	return nil, fmt.Errorf("all HiveServer2 endpoints failed: %s", strings.Join(failures, "; "))
}

func (connector *discoveryConnector) Driver() driver.Driver {
	return connector.driver
}

func openHiveDatabase(config connectionConfig) *sql.DB {
	database := sql.OpenDB(newDiscoveryConnector(config))
	database.SetMaxOpenConns(1)
	database.SetMaxIdleConns(1)
	return database
}

func normalizeHiveAuth(value string) string {
	normalized := strings.ToUpper(strings.TrimSpace(value))
	switch normalized {
	case "":
		return "NONE"

View on GitHub (pinned to c0390bff16)

Solutions

  1. Inspect the joined failure details to see why each endpoint failed and fix the shared cause
  2. Verify HiveServer2 instances are running and reachable (nc/curl the host:port)
  3. Check firewall/security-group rules between client and the servers
  4. Refresh ZooKeeper service discovery state; remove stale endpoints or reconnect after cluster recovery
  5. Retry with backoff — transient cluster restarts resolve themselves

Example fix

// before
db, err := sql.Open("hive", "zookeeper://zk1:2181/hiveserver2") // all endpoints down
// after
// fix/restart HiveServer2, then:
db, err := sql.Open("hive", "zookeeper://zk1:2181/hiveserver2")
if err != nil && strings.Contains(err.Error(), "all HiveServer2 endpoints failed") {
    time.Sleep(5 * time.Second)
    db, err = sql.Open("hive", dsn) // retry
}
Defensive patterns

Strategy: retry

Validate before calling

addrs, err := net.LookupHost(zkHost)
if err != nil {
    return fmt.Errorf("discovery host unresolvable: %w", err)
}
// optionally probe a HiveServer2 port before connecting
for _, a := range addrs {
    c, err := net.DialTimeout("tcp", a+":10000", 2*time.Second)
    if err == nil { c.Close(); return nil }
}
return errors.New("no HiveServer2 endpoint currently reachable")

Try / catch

db, err := sql.Open("hive", dsn)
if err != nil { return err }
var pingErr error
for i := 0; i < 3; i++ {
    ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
    pingErr = db.PingContext(ctx)
    cancel()
    if pingErr == nil { break }
    time.Sleep(backoff(i))
}
if pingErr != nil && strings.Contains(pingErr.Error(), "all HiveServer2 endpoints failed") {
    // alert: cluster-wide outage; surface the joined per-endpoint causes
}

Prevention

When it happens

Trigger: Calling driver.Open/Connect on a discovery (zookeeper:// or hive discovery) DSN where every endpoint listed by discovery fails its individual connect (refused, timeout, auth rejected, TLS failure).

Common situations: HiveServer2 cluster down or being restarted; firewall/security group blocking the client; discovery returning stale node entries pointing at decommissioned hosts; wrong port in server registrations; Kerberos/TLS misconfiguration shared by all endpoints.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/1825a81a20adf211. Report an issue: GitHub.