{"record":{"id":"e9b2d5ffa4e2e1bd","repo":"vitessio/vitess","slug":"reloadschemashard-s-s-failed-w","errorCode":null,"errorMessage":"ReloadSchemaShard(%s/%s) failed: %w","messagePattern":"ReloadSchemaShard\\((.+?)/(.+?)\\) failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtadmin/cluster/cluster.go","lineNumber":2320,"sourceCode":"\t\t\t\tspan, ctx := trace.NewSpan(ctx, \"Cluster.reloadShardSchema\")\n\t\t\t\tdefer span.Finish()\n\n\t\t\t\tAnnotateSpan(c, span)\n\t\t\t\tspan.Annotate(\"keyspace\", keyspace)\n\t\t\t\tspan.Annotate(\"shard\", shard)\n\t\t\t\tspan.Annotate(\"concurrency\", req.Concurrency)\n\t\t\t\tspan.Annotate(\"include_primary\", req.IncludePrimary)\n\t\t\t\tspan.Annotate(\"wait_position\", req.WaitPosition)\n\n\t\t\t\tresp, err := c.Vtctld.ReloadSchemaShard(ctx, &vtctldatapb.ReloadSchemaShardRequest{\n\t\t\t\t\tKeyspace:       keyspace,\n\t\t\t\t\tShard:          shard,\n\t\t\t\t\tConcurrency:    req.Concurrency,\n\t\t\t\t\tIncludePrimary: req.IncludePrimary,\n\t\t\t\t\tWaitPosition:   req.WaitPosition,\n\t\t\t\t})\n\t\t\t\tif err != nil {\n\t\t\t\t\trec.RecordError(fmt.Errorf(\"ReloadSchemaShard(%s/%s) failed: %w\", keyspace, shard, err))\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\tm.Lock()\n\t\t\t\tdefer m.Unlock()\n\t\t\t\tresults = append(results, &vtadminpb.ReloadSchemasResponse_ShardResult{\n\t\t\t\t\tShard: &vtadminpb.Shard{\n\t\t\t\t\t\tCluster: cpb,\n\t\t\t\t\t\tShard: &vtctldatapb.Shard{\n\t\t\t\t\t\t\tKeyspace: keyspace,\n\t\t\t\t\t\t\tName:     shard,\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t\tEvents: resp.Events,\n\t\t\t\t})\n\t\t\t}(ks, shard)\n\t\t}\n\t}","sourceCodeStart":2302,"sourceCodeEnd":2338,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtadmin/cluster/cluster.go#L2302-L2338","documentation":"In reloadKeyspaceSchemas, when the request targets specific shards, each shard-level ReloadSchema call's error is wrapped as 'ReloadSchemaShard(%s/%s) failed' and recorded in the errgroup collector. The underlying cause is the vtctld ReloadSchemaShard RPC failure for that keyspace/shard.","triggerScenarios":"vtctld ReloadSchema for a specific keyspace/shard fails — tablets in that shard unreachable, RPC timeout, reload with WaitPosition/IncludePrimary options failing on a tablet, or context cancellation.","commonSituations":"Reload after DDL on a shard whose replica is lagging (WaitPosition unmet); shard with a down tablet; cell isolation preventing RPCs to that shard's tablets.","solutions":["Read the wrapped cause to identify the failing tablet in the shard","Verify all tablets in the shard are serving and reachable (vtctldclient GetTablets)","Retry the shard reload after restoring tablet health","Relax WaitPosition or IncludePrimary options if they cannot be satisfied"],"exampleFix":"// before\nreq := &vtadminpb.ReloadSchemasRequest{Keyspaces: []string{\"ks\"}, Shards: []string{\"-\"}}\n// after (retry only the failing shard once the tablet recovers)\nreq := &vtadminpb.ReloadSchemasRequest{KeyspacesAndShards: []*vtadminpb.ReloadSchemasRequest_KeyspaceShard{{Keyspace: \"ks\", Shard: \"-\"}}}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"res, err := c.ReloadSchemas(ctx, req)\nif err != nil {\n    if strings.Contains(err.Error(), \"ReloadSchemaShard(\") {\n        // parse keyspace/shard from message and retry that shard only\n    }\n}","preventionTips":["Check shard tablet health before shard-scoped reloads","Avoid WaitPosition values replicas cannot reach while lagging","Handle per-shard errors granularly rather than aborting the batch"],"tags":["vtadmin","schema-reload","shard","tablet-unreachable"],"backgroundTag":"schema-reload-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}