weaviate/weaviate · error

fetch meta of node %q: %w

Error message

fetch meta of node %q: %w

What it means

While fetching the backup's schema metadata during restore, the leader node's meta file could not be read from the object store. For raft-era backups the descriptor records a leader, and only the leader's meta is fetched (it alone carries the user/rbac snapshots). The underlying storage error is wrapped with the node name for diagnosis.

Source

Thrown at usecases/backup/scheduler.go:1154

	overridePath string,
	req *backup.DistributedBackupDescriptor,
) ([]backup.ClassDescriptor, []byte, []byte, error) {
	f := func(node string) (*backup.BackupDescriptor, error) {
		store, err := nodeBackend(node, s.backends, backend, req.ID, overrideBucket, overridePath)
		if err != nil {
			return nil, err
		}
		meta, err := store.Meta(ctx, req.ID, store.bucket, store.path)
		if err != nil {
			return nil, err
		}
		return meta, nil
	}

	if req.Leader != "" {
		meta, err := f(req.Leader) // raft version of the backup
		if err != nil {
			return nil, nil, nil, fmt.Errorf("fetch meta of node %q: %w", req.Leader, err)
		}
		return meta.Classes, meta.UserBackups, meta.RbacBackups, nil
	}

	// union
	m := make(map[string]backup.ClassDescriptor, 64)
	for k := range req.Nodes {
		meta, err := f(k)
		if err != nil {
			// Fail closed: a partial union would silently skip the missing
			// node's classes at restore time and blind the strip validation.
			return nil, nil, nil, fmt.Errorf("fetch meta of node %q: %w", k, err)
		}
		// guess the most up to date version
		for _, x := range meta.Classes {
			c, ok := m[x.Name]
			if !ok || len(x.ShardingState) > len(c.ShardingState) {
				m[x.Name] = x

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Verify the backup meta file exists in the object store under the leader node's prefix (bucket/path shown in the error)
2. Check storage backend credentials and bucket configuration used by the restore request (including overrideBucket/overridePath)
3. List or inspect the backup first (e.g. via the backup status/inspection API) to confirm it is intact before restoring
4. If the meta is unrecoverable, re-create the backup from a healthy cluster

// before restore, confirm meta is fetchable
meta, err := client.Backup().MetaGetter(backend).WithBackupID(id).Do(ctx)
if err != nil { return fmt.Errorf("backup %s meta not accessible: %w", id, err) }

if err := doRestore(ctx, req); err != nil {
    var transient bool
    if strings.Contains(err.Error(), "fetch meta of node") {
        transient = isRetryableStoreErr(err) // network/timeout/5xx from object store
    }
    if transient { backoffRetry(req) } else { return err }
}

When it happens

Trigger: Calling restore for a backup whose distributed descriptor has a non-empty Leader, and the backend store's Meta() call for that node fails: meta file absent, bucket misconfigured, credentials invalid, or the object-store request errored.

Common situations: Backup metadata deleted or expired in the object store (lifecycle rules); wrong bucket/override path passed to restore; S3/GCS/Azure credentials rotated after the backup; the leader node's prefix was partially removed by a prior failed restore cleanup.

Related errors

• there are no classes defined yet
• The V0 gRPC API is deprecated and will be removed in the nex
• could not cast generative result additional prop
• could not cast generative search params
• backup blocked: runtime-reindex in flight on this shard

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/137a19e1f8222d7e. Report an issue: GitHub.