What causes "writing serving data failed: %v" in vitessio/vitess?

Read the wrapped underlying error to identify the failing cell, then check that cell's topo server health/connectivity.

writing serving data failed: %v

Error message

writing serving data failed: %v

What it means

RebuildKeyspace writes the computed SrvKeyspace to every cell concurrently; any UpdateSrvKeyspace failure is wrapped as 'writing serving data failed: %v' and collected into the concurrent error recorder. The final error aggregates the underlying topo error (e.g. permission, timeout, or node-not-found from a specific cell).

Source

Thrown at go/vt/topotools/rebuild_keyspace.go:174

			}
		}

		if ki.KeyspaceType != topodatapb.KeyspaceType_SNAPSHOT || !allowPartial {
			// skip this check for SNAPSHOT keyspaces so that incomplete keyspaces can still serve
			if err := topo.OrderAndCheckPartitions(cell, srvKeyspace); err != nil {
				return err
			}
		}
	}
	// And then finally save the keyspace objects, in parallel.
	rec := concurrency.AllErrorRecorder{}
	wg := sync.WaitGroup{}
	for cell, srvKeyspace := range srvKeyspaceMap {
		wg.Add(1)
		go func(cell string, srvKeyspace *topodatapb.SrvKeyspace) {
			defer wg.Done()
			if err := ts.UpdateSrvKeyspace(ctx, cell, keyspace, srvKeyspace); err != nil {
				rec.RecordError(fmt.Errorf("writing serving data failed: %v", err))
			}
		}(cell, srvKeyspace)
	}
	wg.Wait()
	return rec.Error()
}

View on GitHub (pinned to 01a25a7d17)

Solutions

Read the wrapped underlying error to identify the failing cell, then check that cell's topo server health/connectivity.
Re-run RebuildKeyspace after the failing cell's topo is back; the write is idempotent and will overwrite the serving graph.
Verify topo credentials/ACLs for the serving graph path in the failing cell.
If partial serving data was written, a successful rebuild across all cells is required to restore consistency — always re-run rather than assuming one cell is enough.

Example fix

// before: ignoring which cell failed
if err := ts.UpdateSrvKeyspace(ctx, cell, keyspace, srvKeyspace); err != nil {
    rec.RecordError(fmt.Errorf("writing serving data failed: %v", err))
}
// after (operator-side retry):
for attempt := 0; attempt < 3; attempt++ {
    if err := ts.RebuildKeyspace(ctx, cells, keyspace); err == nil {
        break
    }
    time.Sleep(2 * time.Second)
}

Defensive patterns

Strategy: try-catch

Validate before calling

// probe each cell's topo before rebuilding
for _, cell := range cells {
    if _, err := ts.GetSrvKeyspace(ctx, cell, keyspace); err != nil && !topo.IsErrType(err, topo.NoNode) {
        return fmt.Errorf("cell %v topo unreachable: %w", cell, err)
    }
}

Type guard

func isServingDataWriteErr(err error) bool { return err != nil && strings.Contains(err.Error(), "writing serving data failed") }

Try / catch

if err := ts.RebuildKeyspace(ctx, cells, keyspace); err != nil {
    if isServingDataWriteErr(err) {
        log.Warn("partial serving graph update; re-running rebuild", slog.Any("error", err))
        return ts.RebuildKeyspace(ctx, cells, keyspace)
    }
    return err
}

Prevention

Monitor topo server health in every cell that participates in serving
Re-run RebuildKeyspace after any failure — writes are idempotent and partial updates must be completed
Verify topo credentials/ACLs in all cells before bulk serving-graph operations

When it happens

Trigger: Calling RebuildKeyspace/RebuildKeyspaceLocked when ts.UpdateSrvKeyspace fails for at least one cell in the srvKeyspaceMap — topo server unreachable, cell-local topo (e.g. etcd2/zk2) down, or permissions issue on the serving graph path.

Common situations: A cell's local topo server is down or partitioned during a rebuild; misconfigured cell topo credentials; network flaps between vtctld and one cell's topo backend causing partial serving-graph updates.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/e7c2f77d8318aab6. Report an issue: GitHub.