{"record":{"id":"d972abbf27b56a22","repo":"vitessio/vitess","slug":"getkeyspaces-cluster-s-w","errorCode":null,"errorMessage":"GetKeyspaces(cluster = %s): %w","messagePattern":"GetKeyspaces\\(cluster = (.+?)\\): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/api.go","lineNumber":1651,"sourceCode":"\t\t\tcontinue\n\t\t}\n\n\t\twg.Add(1)\n\n\t\tgo func(c *cluster.Cluster) {\n\t\t\tdefer wg.Done()\n\n\t\t\tspan, ctx := trace.NewSpan(ctx, \"Cluster.GetVSchemas\")\n\t\t\tdefer span.Finish()\n\n\t\t\tcluster.AnnotateSpan(c, span)\n\n\t\t\tgetKeyspacesSpan, getKeyspacesCtx := trace.NewSpan(ctx, \"Cluster.GetKeyspaces\")\n\t\t\tcluster.AnnotateSpan(c, getKeyspacesSpan)\n\n\t\t\tkeyspaces, err := c.Vtctld.GetKeyspaces(getKeyspacesCtx, &vtctldatapb.GetKeyspacesRequest{})\n\t\t\tif err != nil {\n\t\t\t\trec.RecordError(fmt.Errorf(\"GetKeyspaces(cluster = %s): %w\", c.ID, err))\n\t\t\t\tgetKeyspacesSpan.Finish()\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tgetKeyspacesSpan.Finish()\n\n\t\t\tvar (\n\t\t\t\tclusterM        sync.Mutex\n\t\t\t\tclusterWG       sync.WaitGroup\n\t\t\t\tclusterRec      concurrency.AllErrorRecorder\n\t\t\t\tclusterVSchemas = make([]*vtadminpb.VSchema, 0, len(keyspaces.Keyspaces))\n\t\t\t)\n\n\t\t\tfor _, keyspace := range keyspaces.Keyspaces {\n\t\t\t\tclusterWG.Add(1)\n\n\t\t\t\tgo func(keyspace *vtctldatapb.Keyspace) {\n\t\t\t\t\tdefer clusterWG.Done()","sourceCodeStart":1633,"sourceCodeEnd":1669,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/api.go#L1633-L1669","documentation":"Within GetVSchemas, each cluster's keyspaces are fetched through that cluster's vtctld (c.Vtctld.GetKeyspaces). A failure is recorded per cluster with this wrapper including the cluster ID and the underlying vtctld error; that cluster contributes no keyspaces and the overall request errors via the recorder.","triggerScenarios":"Calling GetVSchemas when a cluster's vtctld GetKeyspaces RPC fails — vtctld down, network partition, topo unavailable, or gRPC auth/context cancellation.","commonSituations":"vtctld restarted mid-request; firewall/network change between vtadmin and vtctld; topo store outage; context deadline exceeded under load.","solutions":["Check the wrapped cause for the root gRPC/topo error and the failing cluster ID","Confirm vtctld for that cluster is healthy (vtctldclient GetKeyspaces)","Verify network connectivity and timeouts between vtadmin and vtctld","Retry after restoring topo/vtctld health; consider per-cluster isolation if one bad cluster blocks aggregation"],"exampleFix":"// before\nkeyspaces, err := c.Vtctld.GetKeyspaces(getKeyspacesCtx, &vtctldatapb.GetKeyspacesRequest{})\nif err != nil {\n\trec.RecordError(fmt.Errorf(\"GetKeyspaces(cluster = %s): %w\", c.ID, err))\n\tgetKeyspacesSpan.Finish()\n\treturn\n}\n// after: ensure vtctld reachable; add per-cluster timeout\nkeyspaces, err := c.Vtctld.GetKeyspaces(getKeyspacesCtx, &vtctldatapb.GetKeyspacesRequest{})\nif err != nil {\n\trec.RecordError(fmt.Errorf(\"GetKeyspaces(cluster = %s): %w\", c.ID, err))\n\tgetKeyspacesSpan.Finish()\n\treturn\n}","handlingStrategy":"try-catch","validationCode":"// pre-check vtctld GetKeyspaces for the cluster\n_, err := c.Vtctld.GetKeyspaces(ctx, &vtctldatapb.GetKeyspacesRequest{})\nif err != nil {\n\tlog.Printf(\"cluster %s vtctld unavailable: %v\", c.ID, err)\n}","typeGuard":"func keyspaceFetchFailed(rec concurrency.AllErrorRecorder) bool { return rec.HasErrors() }","tryCatchPattern":"resp, err := client.GetVSchemas(ctx, req)\nif err != nil {\n\tif strings.Contains(err.Error(), \"GetKeyspaces(cluster = \") {\n\t\t// retry after vtctld/topo health restored\n\t}\n\treturn err\n}","preventionTips":["Monitor vtctld gRPC health per cluster","Set sane gRPC timeouts between vtadmin and vtctld","Retry with backoff on transient Unavailable errors","Alert on topo store outages"],"tags":["vtadmin","vtctld","keyspaces","grpc"],"backgroundTag":"vtctld-rpc-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}