{"record":{"id":"f4dc5c511a9b64c8","repo":"vitessio/vitess","slug":"getkeyspaces-cluster-s-failed-w","errorCode":null,"errorMessage":"GetKeyspaces(cluster = %s) failed: %w","messagePattern":"GetKeyspaces\\(cluster = (.+?)\\) failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":743,"sourceCode":"\tif opts.IgnoreKeyspaces == nil {\n\t\topts.IgnoreKeyspaces = sets.New[string]()\n\t}\n\n\tif len(keyspaces) == 0 {\n\t\tspan, ctx := trace.NewSpan(ctx, \"Cluster.GetKeyspaces\")\n\t\tAnnotateSpan(c, span)\n\n\t\tif err := c.topoReadPool.Acquire(ctx); err != nil {\n\t\t\tspan.Finish()\n\t\t\treturn nil, fmt.Errorf(\"findWorkflows(keyspaces = %v, opts = %+v) failed to acquire topoReadPool: %w\", keyspaces, opts, err)\n\t\t}\n\n\t\tresp, err := c.Vtctld.GetKeyspaces(ctx, &vtctldatapb.GetKeyspacesRequest{})\n\t\tc.topoReadPool.Release()\n\n\t\tif err != nil {\n\t\t\tspan.Finish()\n\t\t\treturn nil, fmt.Errorf(\"GetKeyspaces(cluster = %s) failed: %w\", c.ID, err)\n\t\t}\n\n\t\tfor _, ks := range resp.Keyspaces {\n\t\t\tkeyspaces = append(keyspaces, ks.Name)\n\t\t}\n\n\t\tspan.Finish()\n\t} else if opts.IgnoreKeyspaces.Len() > 0 {\n\t\tlog.Warn(fmt.Sprintf(\"Cluster.findWorkflows: IgnoreKeyspaces was set, but Keyspaces was not empty; ignoring IgnoreKeyspaces in favor of explicitly checking everything in Keyspaces: (%s)\", strings.Join(keyspaces, \", \")))\n\t\topts.IgnoreKeyspaces = sets.New[string]()\n\t}\n\n\t// Annotate the parent span with some additional information about the call.\n\tif span, _ := trace.FromContext(ctx); span != nil {\n\t\tspan.Annotate(\"num_keyspaces\", len(keyspaces))\n\t\tspan.Annotate(\"keyspaces\", strings.Join(keyspaces, \",\"))\n\t\tspan.Annotate(\"num_ignore_keyspaces\", opts.IgnoreKeyspaces.Len())\n\t\tspan.Annotate(\"ignore_keyspaces\", strings.Join(sets.List(opts.IgnoreKeyspaces), \",\"))","sourceCodeStart":725,"sourceCodeEnd":761,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L725-L761","documentation":"FindWorkflows must enumerate every keyspace in the cluster to discover their workflows. It first calls vtctld's GetKeyspaces RPC, and if that call fails it wraps the underlying error with the cluster ID so the caller knows which Vitess cluster's topology could not be read. This is a topology-discovery failure, not a workflow-specific failure.","triggerScenarios":"Calling FindWorkflows, GetWorkflow, or GetWorkflows when the vtctld-backed GetKeyspaces RPC returns an error — e.g. vtctld unreachable, topology server down, or ctx canceled while waiting on c.topoReadPool.Acquire inside the goroutine that fetches keyspaces.","commonSituations":"Vitess topology (etcd2/zk2) is down or misconfigured; vtctld process restarted or OOMed; network partition between vtadmin and vtctld; cluster ID typo pointing vtadmin at a nonexistent cluster; request context deadline exceeded under load.","solutions":["Check connectivity from vtadmin to the cluster's vtctld (address/port in vtadmin config)","Verify the Vitess topology service (etcd/zk) is healthy and reachable by vtctld","Check vtctld logs at the time of the request for the root-cause RPC error","Retry the request; if pool exhaustion is the cause, raise the topo read pool size"],"exampleFix":"// before\nerr = fmt.Errorf(\"GetKeyspaces(cluster = %s) failed: %w\", c.ID, err)\n// after: ensure ctx deadline and pre-check topology health\nctx, cancel := context.WithTimeout(ctx, 10*time.Second)\ndefer cancel()\nresp, err := c.Vtctld.GetKeyspaces(ctx, &vtctldatapb.GetKeyspacesRequest{})\nif err != nil {\n    return nil, fmt.Errorf(\"GetKeyspaces(cluster = %s) failed: %w\", c.ID, err)\n}","handlingStrategy":"retry","validationCode":"// caller-side pre-check\nconn, err := net.DialTimeout(\"tcp\", vtctldAddr, 2*time.Second)\nif err != nil { return fmt.Errorf(\"vtctld %s unreachable: %w\", vtctldAddr, err) }","typeGuard":null,"tryCatchPattern":"workflows, err := cluster.FindWorkflows(ctx, req)\nif err != nil {\n    if errors.Is(err, context.DeadlineExceeded) {\n        // retry with longer deadline\n    }\n    log.Error(\"workflow discovery failed\", slog.Any(\"error\", err))\n    return err\n}","preventionTips":["Monitor vtctld and topology service health","Set generous request timeouts","Alert on vtadmin-to-vtctld connectivity","Pin correct cluster ID in vtadmin config"],"tags":["vtadmin","topology","vtctld","rpc"],"backgroundTag":"topology-rpc-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}