cannot delete SrvKeyspace from cell %v for keyspace %v: %w

Error message

cannot delete SrvKeyspace from cell %v for keyspace %v: %w

What it means

This error wraps a failure from topo.DeleteSrvKeyspace during the RemoveKeyspaceCell vtctld RPC, after all tablets in the cell have been removed. It means the final step of purging the cell-local SrvKeyspace object failed, so the keyspace is only partially removed from that cell.

Source

Thrown at go/vt/vtctl/grpcvtctldserver/server.go:3632

	shards, err := s.ts.GetShardNames(ctx, req.Keyspace)
	if err != nil {
		return nil, err
	}

	// Remove all the shards, serially. Stop immediately if any fail.
	for _, shard := range shards {
		log.Info(fmt.Sprintf("Removing cell %v from shard %v/%v", req.Cell, req.Keyspace, shard))
		if err2 := removeShardCell(ctx, s.ts, req.Cell, req.Keyspace, shard, req.Recursive, req.Force); err2 != nil {
			err = fmt.Errorf("cannot remove cell %v from shard %v/%v: %w", req.Cell, req.Keyspace, shard, err2)
			return nil, err
		}
	}

	// Last, remove the SrvKeyspace object.
	log.Info(fmt.Sprintf("Removing cell %v keyspace %v SrvKeyspace object", req.Cell, req.Keyspace))
	if err = s.ts.DeleteSrvKeyspace(ctx, req.Cell, req.Keyspace); err != nil {
		err = fmt.Errorf("cannot delete SrvKeyspace from cell %v for keyspace %v: %w", req.Cell, req.Keyspace, err)
		return nil, err
	}

	return &vtctldatapb.RemoveKeyspaceCellResponse{}, nil
}

// RemoveShardCell is part of the vtctlservicepb.VtctldServer interface.
func (s *VtctldServer) RemoveShardCell(ctx context.Context, req *vtctldatapb.RemoveShardCellRequest) (resp *vtctldatapb.RemoveShardCellResponse, err error) {
	span, ctx := trace.NewSpan(ctx, "VtctldServer.RemoveShardCell")
	defer span.Finish()

	defer panicHandler(&err)

	span.Annotate("keyspace", req.Keyspace)
	span.Annotate("shard", req.ShardName)
	span.Annotate("cell", req.Cell)
	span.Annotate("force", req.Force)
	span.Annotate("recursive", req.Recursive)

View on GitHub (pinned to 01a25a7d17)

Solutions

Verify topo server connectivity and health (etcd/ZK) and retry RemoveKeyspaceCell
Check whether the SrvKeyspace was already deleted — if so the error is stale and the removal effectively succeeded
Inspect topo ACL/credentials for the vtctld identity
Manually verify/clean the leftover SrvKeyspace with topo tools before re-running

Example fix

// before
if err = s.ts.DeleteSrvKeyspace(ctx, req.Cell, req.Keyspace); err != nil {
    err = fmt.Errorf("cannot delete SrvKeyspace from cell %v for keyspace %v: %w", req.Cell, req.Keyspace, err)
    return nil, err
}
// after
if err = s.ts.DeleteSrvKeyspace(ctx, req.Cell, req.Keyspace); err != nil {
    if topo.IsErrType(err, topo.NoNode) {
        log.Warning("SrvKeyspace already deleted")
    } else {
        return nil, fmt.Errorf("cannot delete SrvKeyspace from cell %v for keyspace %v: %w", req.Cell, req.Keyspace, err)
    }
}

Defensive patterns

Strategy: try-catch

Validate before calling

// check SrvKeyspace exists in the target cell before removing
ks, err := topoServer.GetSrvKeyspace(ctx, cell, keyspace)
if err != nil { /* already gone or topo down — decide */ }

Try / catch

try {
  vtctldClient.removeKeyspaceCell(keyspace, cell, /* recursive */ true)
} catch (e) {
  if (String(e).includes('cannot delete SrvKeyspace')) {
    // verify topo health, then check if SrvKeyspace already deleted
    retryOrTreatAsDone()
  } else { throw e }
}

Prevention

Ensure topo (etcd/ZK) is healthy before decommissioning a cell
Use the --recursive flag so tablets are removed before the SrvKeyspace
Serialize cell-removal operations to avoid concurrent deletions
Verify topo ACLs grant vtctld write access to the cell's keyspace path

When it happens

Trigger: Calling `vtctldserver RemoveKeyspaceCell` (via `vtctldclient RemoveKeyspaceCell`) when the topo server cannot delete the SrvKeyspace record — e.g. topo connection failure, permission denied, or the node was concurrently deleted.

Common situations: Running a cell decommission while the topo (etcd/ZK) is unavailable or flaky; two operators removing the same cell concurrently so the SrvKeyspace is already gone; topo ACLs denying the vtctld identity write access.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/53fa843f4b663cf9. Report an issue: GitHub.