{"record":{"id":"1e1fe131265bd84a","repo":"vitessio/vitess","slug":"reloadschemas-cluster-s-failed-w","errorCode":null,"errorMessage":"ReloadSchemas(cluster = %s) failed: %w","messagePattern":"ReloadSchemas\\(cluster = (.+?)\\) failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/api.go","lineNumber":2243,"sourceCode":"\t\tm    sync.Mutex\n\t\twg   sync.WaitGroup\n\t\trec  concurrency.AllErrorRecorder\n\t\tresp vtadminpb.ReloadSchemasResponse\n\t)\n\n\tfor _, c := range clusters {\n\t\tif !api.authz.IsAuthorized(ctx, c.ID, rbac.SchemaResource, rbac.ReloadAction) {\n\t\t\tcontinue\n\t\t}\n\n\t\twg.Add(1)\n\n\t\tgo func(c *cluster.Cluster) {\n\t\t\tdefer wg.Done()\n\n\t\t\tcr, err := c.ReloadSchemas(ctx, req)\n\t\t\tif err != nil {\n\t\t\t\trec.RecordError(fmt.Errorf(\"ReloadSchemas(cluster = %s) failed: %w\", c.ID, err))\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tm.Lock()\n\t\t\tdefer m.Unlock()\n\t\t\tresp.KeyspaceResults = append(resp.KeyspaceResults, cr.KeyspaceResults...)\n\t\t\tresp.ShardResults = append(resp.ShardResults, cr.ShardResults...)\n\t\t\tresp.TabletResults = append(resp.TabletResults, cr.TabletResults...)\n\t\t}(c)\n\t}\n\n\twg.Wait()\n\tif rec.HasErrors() {\n\t\treturn nil, rec.Error()\n\t}\n\n\treturn &resp, nil\n}","sourceCodeStart":2225,"sourceCodeEnd":2261,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/api.go#L2225-L2261","documentation":"During batch ReloadSchemas across clusters, VTAdmin records this error per cluster when c.ReloadSchemas fails; the wrapped error from the vtctld RPC is preserved. The overall call still returns other clusters' results, but this cluster's reload failed.","triggerScenarios":"Calling API.ReloadSchemas when the underlying vtctld ReloadSchemas RPC for one cluster fails (tablet unreachable, vtctld down, RPC timeout).","commonSituations":"A tablet was decommissioned or is down while its cluster is still registered in vtadmin; network partition between vtadmin and vtctld; MySQL restarted and tablet schema manager is unhealthy.","solutions":["Inspect the wrapped inner error to identify the failing tablet/cluster and fix connectivity to vtctld/tablets","Re-run ReloadSchemas for that cluster once the tablet is healthy","Remove stale/decommissioned clusters or tablets from the vtadmin cluster config"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// ensure the cluster is reachable before batch reload\nfor _, c := range clusters {\n    if _, err := c.GetVtctld(ctx); err != nil {\n        log.Printf(\"skipping unreachable cluster %s\", c.ID)\n    }\n}","typeGuard":null,"tryCatchPattern":"resp, err := client.ReloadSchemas(ctx, req)\nif err != nil {\n    var perCluster []string\n    if errors.As(err, &agg) { /* collect per-cluster failures */ }\n    log.Printf(\"ReloadSchemas partial failure: %v\", err)\n    // inspect resp.KeyspaceResults for clusters that succeeded and retry the rest\n}","preventionTips":["Monitor tablet health so reloads are not issued against dead tablets","Retry per-cluster reloads idempotently after transient vtctld failures","Keep vtadmin's cluster list in sync with decommissioned clusters"],"tags":["vtadmin","schema","vtctld","rpc"],"backgroundTag":"schema-reload-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}