{"record":{"id":"1825a81a20adf211","repo":"t8y2/dbx","slug":"all-hiveserver2-endpoints-failed-s","errorCode":null,"errorMessage":"all HiveServer2 endpoints failed: %s","messagePattern":"all HiveServer2 endpoints failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"agents/drivers/argo-go/connector.go","lineNumber":167,"sourceCode":"\t\t\t\trejected[target.address()] = true\n\t\t\t\tfailures = append(failures, fmt.Sprintf(\"attempt %d %s: %v\", attempt+1, target.address(), connectErr))\n\t\t\t}\n\t\t\tbreak\n\t\t}\n\t\tif attempt+1 < connector.retries && connector.retryInterval > 0 {\n\t\t\ttimer := time.NewTimer(connector.retryInterval)\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\ttimer.Stop()\n\t\t\t\treturn nil, ctx.Err()\n\t\t\tcase <-timer.C:\n\t\t\t}\n\t\t}\n\t}\n\tif len(failures) == 0 {\n\t\treturn nil, errors.New(\"Hive discovery returned no endpoints\")\n\t}\n\treturn nil, fmt.Errorf(\"all HiveServer2 endpoints failed: %s\", strings.Join(failures, \"; \"))\n}\n\nfunc (connector *discoveryConnector) Driver() driver.Driver {\n\treturn connector.driver\n}\n\nfunc openHiveDatabase(config connectionConfig) *sql.DB {\n\tdatabase := sql.OpenDB(newDiscoveryConnector(config))\n\tdatabase.SetMaxOpenConns(1)\n\tdatabase.SetMaxIdleConns(1)\n\treturn database\n}\n\nfunc normalizeHiveAuth(value string) string {\n\tnormalized := strings.ToUpper(strings.TrimSpace(value))\n\tswitch normalized {\n\tcase \"\":\n\t\treturn \"NONE\"","sourceCodeStart":149,"sourceCodeEnd":185,"githubUrl":"https://github.com/t8y2/dbx/blob/c0390bff16418b651f4728520d99adf8ce48829a/agents/drivers/argo-go/connector.go#L149-L185","documentation":"The discovery-based Hive connector tries every HiveServer2 endpoint returned by service discovery (ZooKeeper); if all connection attempts fail, it aggregates the per-endpoint failure messages and returns this error joined with '; '. The empty-failures case (no endpoints recorded) raises 'Hive discovery returned no endpoints' instead. This error tells the caller none of the discovered servers were reachable or accepted the connection.","triggerScenarios":"Calling driver.Open/Connect on a discovery (zookeeper:// or hive discovery) DSN where every endpoint listed by discovery fails its individual connect (refused, timeout, auth rejected, TLS failure).","commonSituations":"HiveServer2 cluster down or being restarted; firewall/security group blocking the client; discovery returning stale node entries pointing at decommissioned hosts; wrong port in server registrations; Kerberos/TLS misconfiguration shared by all endpoints.","solutions":["Inspect the joined failure details to see why each endpoint failed and fix the shared cause","Verify HiveServer2 instances are running and reachable (nc/curl the host:port)","Check firewall/security-group rules between client and the servers","Refresh ZooKeeper service discovery state; remove stale endpoints or reconnect after cluster recovery","Retry with backoff — transient cluster restarts resolve themselves"],"exampleFix":"// before\ndb, err := sql.Open(\"hive\", \"zookeeper://zk1:2181/hiveserver2\") // all endpoints down\n// after\n// fix/restart HiveServer2, then:\ndb, err := sql.Open(\"hive\", \"zookeeper://zk1:2181/hiveserver2\")\nif err != nil && strings.Contains(err.Error(), \"all HiveServer2 endpoints failed\") {\n    time.Sleep(5 * time.Second)\n    db, err = sql.Open(\"hive\", dsn) // retry\n}","handlingStrategy":"retry","validationCode":"addrs, err := net.LookupHost(zkHost)\nif err != nil {\n    return fmt.Errorf(\"discovery host unresolvable: %w\", err)\n}\n// optionally probe a HiveServer2 port before connecting\nfor _, a := range addrs {\n    c, err := net.DialTimeout(\"tcp\", a+\":10000\", 2*time.Second)\n    if err == nil { c.Close(); return nil }\n}\nreturn errors.New(\"no HiveServer2 endpoint currently reachable\")","typeGuard":null,"tryCatchPattern":"db, err := sql.Open(\"hive\", dsn)\nif err != nil { return err }\nvar pingErr error\nfor i := 0; i < 3; i++ {\n    ctx, cancel := context.WithTimeout(ctx, 10*time.Second)\n    pingErr = db.PingContext(ctx)\n    cancel()\n    if pingErr == nil { break }\n    time.Sleep(backoff(i))\n}\nif pingErr != nil && strings.Contains(pingErr.Error(), \"all HiveServer2 endpoints failed\") {\n    // alert: cluster-wide outage; surface the joined per-endpoint causes\n}","preventionTips":["Parse the joined per-endpoint failures to find the shared root cause","Monitor HiveServer2 health and ZooKeeper registrations","Clean stale ZK nodes; use session-based ephemeral znodes","Alert on 'all endpoints failed' as a cluster-level incident, not a per-node blip"],"tags":["hive","connection","discovery","failover"],"backgroundTag":"all-endpoints-failed","analyzedSha":"c0390bff16418b651f4728520d99adf8ce48829a","analyzedAt":"2026-09-05T23:05:10.900Z","contentChangedAt":"2026-09-05T23:05:10.900Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}