vitessio/vitess · error

cannot remove cell %v from shard %v/%v: %w

Error message

cannot remove cell %v from shard %v/%v: %w

What it means

When removing a cell from a keyspace, each shard's cell removal (removeShardCell) is attempted serially, stopping at the first failure. This error wraps the per-shard failure (e.g. tablets still serving in that cell without -force) with the cell, keyspace, and shard context.

Source

Thrown at go/vt/vtctl/grpcvtctldserver/server.go:3624

	defer span.Finish()

	defer panicHandler(&err)

	span.Annotate("keyspace", req.Keyspace)
	span.Annotate("cell", req.Cell)
	span.Annotate("force", req.Force)
	span.Annotate("recursive", req.Recursive)

	shards, err := s.ts.GetShardNames(ctx, req.Keyspace)
	if err != nil {
		return nil, err
	}

	// Remove all the shards, serially. Stop immediately if any fail.
	for _, shard := range shards {
		log.Info(fmt.Sprintf("Removing cell %v from shard %v/%v", req.Cell, req.Keyspace, shard))
		if err2 := removeShardCell(ctx, s.ts, req.Cell, req.Keyspace, shard, req.Recursive, req.Force); err2 != nil {
			err = fmt.Errorf("cannot remove cell %v from shard %v/%v: %w", req.Cell, req.Keyspace, shard, err2)
			return nil, err
		}
	}

	// Last, remove the SrvKeyspace object.
	log.Info(fmt.Sprintf("Removing cell %v keyspace %v SrvKeyspace object", req.Cell, req.Keyspace))
	if err = s.ts.DeleteSrvKeyspace(ctx, req.Cell, req.Keyspace); err != nil {
		err = fmt.Errorf("cannot delete SrvKeyspace from cell %v for keyspace %v: %w", req.Cell, req.Keyspace, err)
		return nil, err
	}

	return &vtctldatapb.RemoveKeyspaceCellResponse{}, nil
}

// RemoveShardCell is part of the vtctlservicepb.VtctldServer interface.
func (s *VtctldServer) RemoveShardCell(ctx context.Context, req *vtctldatapb.RemoveShardCellRequest) (resp *vtctldatapb.RemoveShardCellResponse, err error) {
	span, ctx := trace.NewSpan(ctx, "VtctldServer.RemoveShardCell")
	defer span.Finish()

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Re-run with `--recursive` (and `--force` if appropriate) so in-cell tablets are handled
2. Scrap/remove the remaining tablets in that cell first (`GetTablets` filtered by cell)
3. Verify no serving tablets remain in the cell, then retry RemoveKeyspaceCell
4. Check topo server health if deletion itself failed

Example fix

// before
vtctldclient RemoveKeyspaceCell --keyspace commerce zone2  # tablets remain
// after
vtctldclient RemoveKeyspaceCell --keyspace commerce --recursive --force zone2

// ensure no tablets remain in the cell before removal
for _, t := range tablets {
    if t.Keyspace == ks && t.Alias.Cell == cell {
        return fmt.Errorf("tablet %s still in cell %s", t.Alias, cell)
    }
}

// fall back to recursive/force removal on first failure
if strings.Contains(err.Error(), "cannot remove cell") {
    return removeKeyspaceCell(ctx, ks, cell, /*recursive*/ true, /*force*/ true)
}

When it happens

Trigger: vtctldclient RemoveKeyspaceCell where removeShardCell fails — serving tablets remain in the cell and Recursive/Force were not set, or topo deletion of the SrvVReverse links failed.

Common situations: Trying to decommission a cell that still hosts replicas for the keyspace; forgetting -recursive; a tablet in the cell is unresponsive so its records can't be cleaned.

Related errors

• no shards found in keyspace
• parse error
• BaseKeyspace is required for SNAPSHOT keyspaces
• SnapshotTime is required for SNAPSHOT keyspaces
• a POST request needs a keyspace in the URL

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/1a091e99e5c95fc5. Report an issue: GitHub.