{"record":{"id":"137a19e1f8222d7e","repo":"weaviate/weaviate","slug":"fetch-meta-of-node-q-w","errorCode":null,"errorMessage":"fetch meta of node %q: %w","messagePattern":"fetch meta of node %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"usecases/backup/scheduler.go","lineNumber":1154,"sourceCode":"\toverridePath string,\n\treq *backup.DistributedBackupDescriptor,\n) ([]backup.ClassDescriptor, []byte, []byte, error) {\n\tf := func(node string) (*backup.BackupDescriptor, error) {\n\t\tstore, err := nodeBackend(node, s.backends, backend, req.ID, overrideBucket, overridePath)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t\tmeta, err := store.Meta(ctx, req.ID, store.bucket, store.path)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t\treturn meta, nil\n\t}\n\n\tif req.Leader != \"\" {\n\t\tmeta, err := f(req.Leader) // raft version of the backup\n\t\tif err != nil {\n\t\t\treturn nil, nil, nil, fmt.Errorf(\"fetch meta of node %q: %w\", req.Leader, err)\n\t\t}\n\t\treturn meta.Classes, meta.UserBackups, meta.RbacBackups, nil\n\t}\n\n\t// union\n\tm := make(map[string]backup.ClassDescriptor, 64)\n\tfor k := range req.Nodes {\n\t\tmeta, err := f(k)\n\t\tif err != nil {\n\t\t\t// Fail closed: a partial union would silently skip the missing\n\t\t\t// node's classes at restore time and blind the strip validation.\n\t\t\treturn nil, nil, nil, fmt.Errorf(\"fetch meta of node %q: %w\", k, err)\n\t\t}\n\t\t// guess the most up to date version\n\t\tfor _, x := range meta.Classes {\n\t\t\tc, ok := m[x.Name]\n\t\t\tif !ok || len(x.ShardingState) > len(c.ShardingState) {\n\t\t\t\tm[x.Name] = x","sourceCodeStart":1136,"sourceCodeEnd":1172,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/usecases/backup/scheduler.go#L1136-L1172","documentation":"While fetching the backup's schema metadata during restore, the leader node's meta file could not be read from the object store. For raft-era backups the descriptor records a leader, and only the leader's meta is fetched (it alone carries the user/rbac snapshots). The underlying storage error is wrapped with the node name for diagnosis.","triggerScenarios":"Calling restore for a backup whose distributed descriptor has a non-empty Leader, and the backend store's Meta() call for that node fails: meta file absent, bucket misconfigured, credentials invalid, or the object-store request errored.","commonSituations":"Backup metadata deleted or expired in the object store (lifecycle rules); wrong bucket/override path passed to restore; S3/GCS/Azure credentials rotated after the backup; the leader node's prefix was partially removed by a prior failed restore cleanup.","solutions":["Verify the backup meta file exists in the object store under the leader node's prefix (bucket/path shown in the error)","Check storage backend credentials and bucket configuration used by the restore request (including overrideBucket/overridePath)","List or inspect the backup first (e.g. via the backup status/inspection API) to confirm it is intact before restoring","If the meta is unrecoverable, re-create the backup from a healthy cluster"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before restore, confirm meta is fetchable\nmeta, err := client.Backup().MetaGetter(backend).WithBackupID(id).Do(ctx)\nif err != nil { return fmt.Errorf(\"backup %s meta not accessible: %w\", id, err) }","typeGuard":null,"tryCatchPattern":"if err := doRestore(ctx, req); err != nil {\n    var transient bool\n    if strings.Contains(err.Error(), \"fetch meta of node\") {\n        transient = isRetryableStoreErr(err) // network/timeout/5xx from object store\n    }\n    if transient { backoffRetry(req) } else { return err }\n}","preventionTips":["Verify the backup with a meta/inspection call before restoring","Watch object-store lifecycle rules so backup prefixes are not auto-expired","Keep backend credentials valid for the entire retention period of backups","Pass explicit overrideBucket/overridePath only when you know the storage layout"],"tags":["backup","restore","object-store","meta","distributed"],"backgroundTag":"backup-meta-fetch-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}