{"record":{"id":"e7c2f77d8318aab6","repo":"vitessio/vitess","slug":"writing-serving-data-failed-v","errorCode":null,"errorMessage":"writing serving data failed: %v","messagePattern":"writing serving data failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/topotools/rebuild_keyspace.go","lineNumber":174,"sourceCode":"\t\t\t}\n\t\t}\n\n\t\tif ki.KeyspaceType != topodatapb.KeyspaceType_SNAPSHOT || !allowPartial {\n\t\t\t// skip this check for SNAPSHOT keyspaces so that incomplete keyspaces can still serve\n\t\t\tif err := topo.OrderAndCheckPartitions(cell, srvKeyspace); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\t}\n\t// And then finally save the keyspace objects, in parallel.\n\trec := concurrency.AllErrorRecorder{}\n\twg := sync.WaitGroup{}\n\tfor cell, srvKeyspace := range srvKeyspaceMap {\n\t\twg.Add(1)\n\t\tgo func(cell string, srvKeyspace *topodatapb.SrvKeyspace) {\n\t\t\tdefer wg.Done()\n\t\t\tif err := ts.UpdateSrvKeyspace(ctx, cell, keyspace, srvKeyspace); err != nil {\n\t\t\t\trec.RecordError(fmt.Errorf(\"writing serving data failed: %v\", err))\n\t\t\t}\n\t\t}(cell, srvKeyspace)\n\t}\n\twg.Wait()\n\treturn rec.Error()\n}\n","sourceCodeStart":156,"sourceCodeEnd":181,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/topotools/rebuild_keyspace.go#L156-L181","documentation":"RebuildKeyspace writes the computed SrvKeyspace to every cell concurrently; any UpdateSrvKeyspace failure is wrapped as 'writing serving data failed: %v' and collected into the concurrent error recorder. The final error aggregates the underlying topo error (e.g. permission, timeout, or node-not-found from a specific cell).","triggerScenarios":"Calling RebuildKeyspace/RebuildKeyspaceLocked when ts.UpdateSrvKeyspace fails for at least one cell in the srvKeyspaceMap — topo server unreachable, cell-local topo (e.g. etcd2/zk2) down, or permissions issue on the serving graph path.","commonSituations":"A cell's local topo server is down or partitioned during a rebuild; misconfigured cell topo credentials; network flaps between vtctld and one cell's topo backend causing partial serving-graph updates.","solutions":["Read the wrapped underlying error to identify the failing cell, then check that cell's topo server health/connectivity.","Re-run RebuildKeyspace after the failing cell's topo is back; the write is idempotent and will overwrite the serving graph.","Verify topo credentials/ACLs for the serving graph path in the failing cell.","If partial serving data was written, a successful rebuild across all cells is required to restore consistency — always re-run rather than assuming one cell is enough."],"exampleFix":"// before: ignoring which cell failed\nif err := ts.UpdateSrvKeyspace(ctx, cell, keyspace, srvKeyspace); err != nil {\n    rec.RecordError(fmt.Errorf(\"writing serving data failed: %v\", err))\n}\n// after (operator-side retry):\nfor attempt := 0; attempt < 3; attempt++ {\n    if err := ts.RebuildKeyspace(ctx, cells, keyspace); err == nil {\n        break\n    }\n    time.Sleep(2 * time.Second)\n}","handlingStrategy":"try-catch","validationCode":"// probe each cell's topo before rebuilding\nfor _, cell := range cells {\n    if _, err := ts.GetSrvKeyspace(ctx, cell, keyspace); err != nil && !topo.IsErrType(err, topo.NoNode) {\n        return fmt.Errorf(\"cell %v topo unreachable: %w\", cell, err)\n    }\n}","typeGuard":"func isServingDataWriteErr(err error) bool { return err != nil && strings.Contains(err.Error(), \"writing serving data failed\") }","tryCatchPattern":"if err := ts.RebuildKeyspace(ctx, cells, keyspace); err != nil {\n    if isServingDataWriteErr(err) {\n        log.Warn(\"partial serving graph update; re-running rebuild\", slog.Any(\"error\", err))\n        return ts.RebuildKeyspace(ctx, cells, keyspace)\n    }\n    return err\n}","preventionTips":["Monitor topo server health in every cell that participates in serving","Re-run RebuildKeyspace after any failure — writes are idempotent and partial updates must be completed","Verify topo credentials/ACLs in all cells before bulk serving-graph operations"],"tags":["topo","serving-graph","rebuild-keyspace","network"],"backgroundTag":"topo-write-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}