vitessio/vitess · error

error deleting ShardReplication object in cell %v: %w

Error message

error deleting ShardReplication object in cell %v: %w

What it means

removeShardCell wraps a failure from DeleteShardReplication when clearing the (now-empty) replication graph for the shard in a cell. NoNode is treated as success; any other error aborts cell removal so stale ShardReplication nodes are never left behind silently.

Source

Thrown at go/vt/vtctl/grpcvtctldserver/topo.go:314

		// We have tablets in the shard in this cell.
		if recursive {
			log.Info(fmt.Sprintf("Deleting all tablets in cell %v in shard %v/%v", cell, keyspace, shardName))
			for _, node := range replication.Nodes {
				// We don't care about scrapping or updating the replication
				// graph, because we're about to delete the entire replication
				// graph.
				log.Info(fmt.Sprintf("Deleting tablet %v", topoproto.TabletAliasString(node.TabletAlias)))
				if err := ts.DeleteTablet(ctx, node.TabletAlias); err != nil && !topo.IsErrType(err, topo.NoNode) {
					return fmt.Errorf("cannot delete tablet %v: %w", topoproto.TabletAliasString(node.TabletAlias), err)
				}
			}
		} else if len(replication.Nodes) > 0 {
			return vterrors.Errorf(vtrpc.Code_FAILED_PRECONDITION, "cell %v has %v possible tablets in replication graph", cell, len(replication.Nodes))
		}

		// Remove the empty replication graph.
		if err := ts.DeleteShardReplication(ctx, cell, keyspace, shardName); err != nil && !topo.IsErrType(err, topo.NoNode) {
			return fmt.Errorf("error deleting ShardReplication object in cell %v: %w", cell, err)
		}
	case topo.IsErrType(err, topo.NoNode):
		// No ShardReplication object. This is the expected path when there are
		// no tablets in the shard in that cell.
		err = nil
	default:
		// If we can't get the replication object out of the local topo, we
		// assume the topo server is down in that cell, so we'll only continue
		// if Force was specified.
		if !force {
			return err
		}

		log.Warn(fmt.Sprintf("Cannot get ShardReplication from cell %v; assuming cell topo server is down and forcing removal", cell))
	}

	// Finally, update the shard.

View on GitHub (pinned to 01a25a7d17)

Solutions

  1. Verify topo connectivity to the cell and retry RemoveKeyspaceCell/RemoveShardCell; the operation is idempotent (NoNode tolerated)
  2. Check topo ACLs for write access to the ShardReplication path
  3. If persistently stuck, delete the stale ShardReplication node directly via topo tooling (etcdctl/zkCli) with care

Example fix

// before
vtctldclient RemoveKeyspaceCell ks c1  # fails deleting ShardReplication
// after
# restore topo connectivity, then retry
vtctldclient RemoveKeyspaceCell ks c1
Defensive patterns

Strategy: retry

Validate before calling

// Confirm ShardReplication is empty/deletable before removal
sri, err := ts.GetShardReplication(ctx, cell, ks, shard)
if err == nil && len(sri.Nodes) > 0 {
	return fmt.Errorf("cell %s still has %d replication nodes", cell, len(sri.Nodes))
}

Try / catch

if err := removeKeyspaceCell(ctx, ks, cell); err != nil && strings.Contains(err.Error(), "ShardReplication") {
	// topo hiccup: retry; NoNode path is already success
}

Prevention

When it happens

Trigger: RemoveKeyspaceCell or RemoveShardCell after all tablets were deleted, but ts.DeleteShardReplication(ctx, cell, keyspace, shardName) fails with a topo RPC/ACL/timeout error.

Common situations: Topo outage during keyspace teardown; ACL preventing writes to the replication graph path; flaky cell topo (e.g., remote cell connection drop).

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/d3f743dd4f149f34. Report an issue: GitHub.