vitessio/vitess · error

FindAllShardsInKeyspace

Error message

FindAllShardsInKeyspace(cluster = %s, keyspace = %s) failed: %w

What it means

FindAllShardsInKeyspace delegates the actual topo read to c.Vtctld.FindAllShardsInKeyspace. Any RPC error is wrapped with the cluster ID and keyspace for context. This means the pool was acquired successfully but the vtctld-side lookup failed — e.g. the keyspace does not exist in the topo server, or vtctld/topo connectivity or storage problems.

Solutions

1. Verify the keyspace exists: vtctldclient GetKeyspace <name> before calling
2. Check vtctld connectivity and health from the vtadmin host
3. Check topo server (etcd/zookeeper) health and logs for the underlying error
4. If a concurrent deletion races reads, tolerate/retry on keyspace-not-found
5. Confirm the vtadmin cluster config points at the correct topo server

Example fix

// before
shards, err := cluster.FindAllShardsInKeyspace(ctx, ks, nil) // fails if ks deleted concurrently
// after
if _, err := cluster.GetKeyspace(ctx, ks); err != nil {
    return fmt.Errorf("keyspace %s not found, skipping shard lookup: %w", ks, err)
}
shards, err := cluster.FindAllShardsInKeyspace(ctx, ks, nil)

_, err := cluster.GetKeyspace(ctx, keyspace)
if err != nil {
    return fmt.Errorf("keyspace %s does not exist in cluster %s: %w", keyspace, clusterID, err)
}

shards, err := cluster.FindAllShardsInKeyspace(ctx, ks, nil)
if err != nil {
    if errors.Is(err, topo.ErrNoNode) || strings.Contains(err.Error(), "node doesn't exist") {
        return nil, fmt.Errorf("keyspace %s not found (deleted?)", ks)
    }
    return nil, err
}

When it happens

Trigger: Calling FindAllShardsInKeyspace for a keyspace that was never created or was already deleted; vtctld unreachable; topo server (etcd/zk) errors; stale cached cluster metadata pointing at the wrong topo.

Common situations: Race between a DeleteKeyspace and a concurrent GetWorkflows/GetKeyspace listing; typo'd keyspace name in automation; topo server outage or network partition; vtadmin pointed at a cluster where the keyspace lives under a different cell.

Related errors

• cannot delete SrvKeyspace from cell
• CreateKeyspace(%+v) failed to acquire topoRWPool
• CreateShard(%+v) failed to acquire topoRWPool
• Error setting tablet to read-only
• Error setting tablet to read-write

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/d7b67d9241856ea6. Report an issue: GitHub.

Appendix: source

Thrown at go/vt/vtadmin/cluster/cluster.go:632

func (c *Cluster) FindAllShardsInKeyspace(ctx context.Context, keyspace string, opts FindAllShardsInKeyspaceOptions) (map[string]*vtctldatapb.Shard, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.FindAllShardsInKeyspace")
	defer span.Finish()

	AnnotateSpan(c, span)
	span.Annotate("keyspace", keyspace)

	if !opts.skipPool {
		if err := c.topoReadPool.Acquire(ctx); err != nil {
			return nil, fmt.Errorf("FindAllShardsInKeyspace(%s) failed to acquire topoReadPool: %w", keyspace, err)
		}
		defer c.topoReadPool.Release()
	}

	resp, err := c.Vtctld.FindAllShardsInKeyspace(ctx, &vtctldatapb.FindAllShardsInKeyspaceRequest{
		Keyspace: keyspace,
	})
	if err != nil {
		return nil, fmt.Errorf("FindAllShardsInKeyspace(cluster = %s, keyspace = %s) failed: %w", c.ID, keyspace, err)
	}

	return resp.Shards, nil
}

// FindTablet returns the first tablet in a given cluster that satisfies the filter function.
func (c *Cluster) FindTablet(ctx context.Context, filter func(*vtadminpb.Tablet) bool) (*vtadminpb.Tablet, error) {
	span, ctx := trace.NewSpan(ctx, "Cluster.FindTablet")
	defer span.Finish()

	AnnotateSpan(c, span)

	tablets, err := c.findTablets(ctx, filter, 1)
	if err != nil {
		return nil, err
	}

	if len(tablets) != 1 {

View on GitHub (pinned to 01a25a7d17)