vitessio/vitess · error

GetKeyspaces(cluster = %s) failed: %w

Error message

GetKeyspaces(cluster = %s) failed: %w

What it means

FindWorkflows must enumerate every keyspace in the cluster to discover their workflows. It first calls vtctld's GetKeyspaces RPC, and if that call fails it wraps the underlying error with the cluster ID so the caller knows which Vitess cluster's topology could not be read. This is a topology-discovery failure, not a workflow-specific failure.

Source

Thrown at go/vt/vtadmin/cluster/cluster.go:743

	if opts.IgnoreKeyspaces == nil {
		opts.IgnoreKeyspaces = sets.New[string]()
	}

	if len(keyspaces) == 0 {
		span, ctx := trace.NewSpan(ctx, "Cluster.GetKeyspaces")
		AnnotateSpan(c, span)

		if err := c.topoReadPool.Acquire(ctx); err != nil {
			span.Finish()
			return nil, fmt.Errorf("findWorkflows(keyspaces = %v, opts = %+v) failed to acquire topoReadPool: %w", keyspaces, opts, err)
		}

		resp, err := c.Vtctld.GetKeyspaces(ctx, &vtctldatapb.GetKeyspacesRequest{})
		c.topoReadPool.Release()

		if err != nil {
			span.Finish()
			return nil, fmt.Errorf("GetKeyspaces(cluster = %s) failed: %w", c.ID, err)
		}

		for _, ks := range resp.Keyspaces {
			keyspaces = append(keyspaces, ks.Name)
		}

		span.Finish()
	} else if opts.IgnoreKeyspaces.Len() > 0 {
		log.Warn(fmt.Sprintf("Cluster.findWorkflows: IgnoreKeyspaces was set, but Keyspaces was not empty; ignoring IgnoreKeyspaces in favor of explicitly checking everything in Keyspaces: (%s)", strings.Join(keyspaces, ", ")))
		opts.IgnoreKeyspaces = sets.New[string]()
	}

	// Annotate the parent span with some additional information about the call.
	if span, _ := trace.FromContext(ctx); span != nil {
		span.Annotate("num_keyspaces", len(keyspaces))
		span.Annotate("keyspaces", strings.Join(keyspaces, ","))
		span.Annotate("num_ignore_keyspaces", opts.IgnoreKeyspaces.Len())
		span.Annotate("ignore_keyspaces", strings.Join(sets.List(opts.IgnoreKeyspaces), ","))

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check connectivity from vtadmin to the cluster's vtctld (address/port in vtadmin config)
2. Verify the Vitess topology service (etcd/zk) is healthy and reachable by vtctld
3. Check vtctld logs at the time of the request for the root-cause RPC error
4. Retry the request; if pool exhaustion is the cause, raise the topo read pool size

Example fix

// before
err = fmt.Errorf("GetKeyspaces(cluster = %s) failed: %w", c.ID, err)
// after: ensure ctx deadline and pre-check topology health
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
resp, err := c.Vtctld.GetKeyspaces(ctx, &vtctldatapb.GetKeyspacesRequest{})
if err != nil {
    return nil, fmt.Errorf("GetKeyspaces(cluster = %s) failed: %w", c.ID, err)
}

// caller-side pre-check
conn, err := net.DialTimeout("tcp", vtctldAddr, 2*time.Second)
if err != nil { return fmt.Errorf("vtctld %s unreachable: %w", vtctldAddr, err) }

workflows, err := cluster.FindWorkflows(ctx, req)
if err != nil {
    if errors.Is(err, context.DeadlineExceeded) {
        // retry with longer deadline
    }
    log.Error("workflow discovery failed", slog.Any("error", err))
    return err
}

When it happens

Trigger: Calling FindWorkflows, GetWorkflow, or GetWorkflows when the vtctld-backed GetKeyspaces RPC returns an error — e.g. vtctld unreachable, topology server down, or ctx canceled while waiting on c.topoReadPool.Acquire inside the goroutine that fetches keyspaces.

Common situations: Vitess topology (etcd2/zk2) is down or misconfigured; vtctld process restarted or OOMed; network partition between vtadmin and vtctld; cluster ID typo pointing vtadmin at a nonexistent cluster; request context deadline exceeded under load.

Related errors

• failed to GetCellInfoNames: %w
• GetKeyspaces(cluster = %s): %w
• invalid key:value pair
• parse error
• failed to delete tablet: %w

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/f4dc5c511a9b64c8. Report an issue: GitHub.