{"record":{"id":"3b2e8634b8c5a42e","repo":"weaviate/weaviate","slug":"snapshot-shard-v-w","errorCode":null,"errorMessage":"snapshot shard %v: %w","messagePattern":"snapshot shard (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/backup.go","lineNumber":420,"sourceCode":"\t\t// holds the LazyLoadShard mutex.\n\t\treleaseBlock()\n\t}\n\n\t// Acquire preventShutdown before releasing shardCreateLocks: UnloadLocalShard\n\t// holds only shardCreateLocks (not backupLock), so without the refcount it could\n\t// call Shard.Shutdown between our release and CreateBackupSnapshot.\n\trelease, err := shard.preventShutdown()\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"prevent shutdown of shard %v: %w\", name, err)\n\t}\n\treleaseShard = release\n\n\ti.shardCreateLocks.Unlock(name)\n\tshardCreateLocksHeld = false\n\n\tfiles, err := shard.CreateBackupSnapshot(ctx, &sd, stagingRoot)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"snapshot shard %v: %w\", name, err)\n\t}\n\n\tif err := sd.FillFileInfo(files, shardBaseDescr, stagingRoot); err != nil {\n\t\treturn nil, fmt.Errorf(\"gather shard %v file info: %w\", name, err)\n\t}\n\n\treturn &sd, nil\n}\n\n// backupInactiveShardWithHardlinks backs up an inactive (unloaded) shard by reading\n// its files from disk and hardlinking them into the staging directory.\nfunc (i *Index) backupInactiveShardWithHardlinks(name string, sd *backup.ShardDescriptor, shardBaseDescr []backup.ShardAndID, stagingRoot string) error {\n\tshardDir := shardPath(i.path(), name)\n\tif _, err := os.Stat(shardDir); err != nil {\n\t\tif os.IsNotExist(err) {\n\t\t\t// FROZEN/OFFLOADED — no local data. Status is preserved in the\n\t\t\t// sharding state; omit from desc.Shards.\n\t\t\treturn errShardNoLocalData","sourceCodeStart":402,"sourceCodeEnd":438,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/backup.go#L402-L438","documentation":"This error wraps any failure from shard.CreateBackupSnapshot during a hard-link based shard backup. CreateBackupSnapshot creates a consistent snapshot of the shard's files (via checkpoint/descriptor files) and hard-links them into the staging directory. Weaviate wraps the underlying cause with the shard name so operators can identify which shard's snapshot failed mid-backup.","triggerScenarios":"A shard backup with hardlinks is in progress (backupShardWithHardlinks) and CreateBackupSnapshot fails — e.g. the shard's underlying store cannot freeze/flush its memtables, the descriptor cannot be written, or an I/O error occurs while linking files into stagingRoot. Commonly triggered when the shard is being shut down concurrently (the repo has a test named TestBackupShardWithHardlinks_PreventShutdownErrorReleasesLocks) or the staging directory is not writable.","commonSituations":"Backups racing with shard drops or server shutdown; disk-full or permission problems on the backup staging path; corrupted LSM state that prevents a clean snapshot; backup endpoints called while a shard is being loaded/unloaded.","solutions":["Check the wrapped inner error for the real cause (disk space, permissions, shutdown race) and fix that first","Retry the backup once the shard is active and no shutdown/drop is in flight","Verify the staging directory path is writable and has sufficient space","Check shard status (should be READY) before starting the backup"],"exampleFix":"// before\nfiles, err := shard.CreateBackupSnapshot(ctx, &sd, stagingRoot)\nif err != nil {\n    return nil, fmt.Errorf(\"snapshot shard %v: %w\", name, err)\n}\n// after — pre-check shard state and ctx before snapshotting\nif err := ctx.Err(); err != nil {\n    return nil, fmt.Errorf(\"snapshot shard %v cancelled: %w\", name, err)\n}\nfiles, err := shard.CreateBackupSnapshot(ctx, &sd, stagingRoot)\nif err != nil {\n    return nil, fmt.Errorf(\"snapshot shard %v: %w\", name, err)\n}","handlingStrategy":"try-catch","validationCode":"// Go: before requesting the backup, verify shard readiness and staging writability\nif err := ctx.Err(); err != nil { return err }\nif s, _ := os.Stat(stagingRoot); s == nil || !s.IsDir() {\n    return fmt.Errorf(\"staging root %s missing\", stagingRoot)\n}\ntest, err := os.CreateTemp(stagingRoot, \".probe*\")\nif err != nil { return err }\nos.Remove(test.Name()); test.Close()","typeGuard":"// errors.As to unwrap the snapshot failure\nvar pathErr *os.PathError\nif errors.As(err, &pathErr) {\n    // filesystem-level problem at pathErr.Path\n}","tryCatchPattern":"err := client.Backup(ctx, ...)\nif err != nil {\n    if strings.Contains(err.Error(), \"snapshot shard\") {\n        // check wrapped cause, verify disk/permissions, retry backup\n        var perr *fs.PathError\n        if errors.As(err, &perr) { log.Printf(\"fs failure on %s: %v\", perr.Path, perr.Err) }\n    }\n    return err\n}","preventionTips":["Avoid running backups concurrently with shard drops or server shutdown","Monitor disk space on both data and staging volumes before large backups","Ensure the backup staging path is writable by the Weaviate process user","Check shard status is READY before backing up active shards"],"tags":["backup","filesystem","shard","snapshot"],"backgroundTag":"snapshot-creation-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}