{"record":{"id":"2d0baffc49f752bc","repo":"weaviate/weaviate","slug":"hardlink-snapshot-w","errorCode":null,"errorMessage":"hardlink snapshot: %w","messagePattern":"hardlink snapshot: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/lsmkv/bucket_snapshot.go","lineNumber":165,"sourceCode":"\t\treturn \"\", fmt.Errorf(\"pause compaction: %w\", err)\n\t}\n\tdefer b.resumeCompaction(ctx)\n\n\t// Pause the flush cycle so no concurrent FlushAndSwitch can produce new\n\t// segment files while we enumerate and hard-link. Deactivate waits for\n\t// any in-progress flush to complete before returning.\n\tif err := b.flushCallbackCtrl.Deactivate(ctx); err != nil {\n\t\treturn \"\", fmt.Errorf(\"pause flush cycle: %w\", err)\n\t}\n\tdefer b.flushCallbackCtrl.Activate()\n\n\tif err := b.FlushMemtable(); err != nil {\n\t\treturn \"\", fmt.Errorf(\"flush memtable: %w\", err)\n\t}\n\n\tif err := snapshotBucketFiles(b.disk.dir, snapshotDir, false); err != nil {\n\t\tos.RemoveAll(snapshotDir)\n\t\treturn \"\", fmt.Errorf(\"hardlink snapshot: %w\", err)\n\t}\n\n\treturn snapshotDir, nil\n}\n\n// snapshotBucketFiles copies bucket files from srcDir into dstDir.\n// Immutable files (segments, bloom filters, count-net-additions) are\n// hard-linked for efficiency. WAL files are copied because they are\n// mutable — if the shard loads after the snapshot is taken, the original\n// WAL could be modified, which would corrupt the hard-linked snapshot.\n//\n// The caller must ensure no concurrent compaction or flush is modifying\n// srcDir — for loaded buckets this means pausing both cycles first; for\n// unloaded buckets this is inherently safe because no cycles are running.\n//\n// When includeWAL is false, .wal files are skipped. This is appropriate when\n// snapshotting a loaded bucket that has just been flushed (the WAL belongs to\n// the new empty memtable). When includeWAL is true, .wal files are copied.","sourceCodeStart":147,"sourceCodeEnd":183,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/lsmkv/bucket_snapshot.go#L147-L183","documentation":"The final step of CreateSnapshot hard-links every bucket segment file from the bucket directory into the snapshot directory via snapshotBucketFiles. On any failure (missing file, cross-device link, permission error) the partially created snapshot directory is removed with os.RemoveAll and this wrapped error is returned; compaction/flush resumption already happened via defers, and the hard-linked model stays valid because compaction only creates new files.","triggerScenarios":"snapshotBucketFiles fails because a source file disappeared mid-enumeration, the snapshots root is on a different filesystem (hard-link returns EXDEV), or permission/IO errors creating links; snapshotsRoot misconfigured.","commonSituations":"Backup target volume mounted on a different device than the data directory (EXDEV is the classic case); racing bucket deletion; permissions mismatch between data dir and snapshot root.","solutions":["Place snapshotsRoot on the same filesystem as the bucket directory — hard links cannot cross devices.","Check the wrapped error for EXDEV and move the snapshot root if so.","Verify read/write permissions on both bucket dir and snapshots root.","Ensure the bucket isn't deleted/compacted away mid-snapshot; retry after fixing storage conditions."],"exampleFix":"// before\nsnapshotDir := \"/mnt/backup-volume/snapshots\" // different FS than data dir\nbucket.CreateSnapshot(ctx, snapshotDir, name) // hardlink snapshot: EXDEV\n// after\nsnapshotDir := filepath.Join(dataRootParentSameFS, \"snapshots\")\nbucket.CreateSnapshot(ctx, snapshotDir, name)","handlingStrategy":"validation","validationCode":"func sameFilesystem(a, b string) bool {\n  sa, ea := syscall.Stat(a, &syscall.Stat_t{})\n  sb, eb := syscall.Stat(b, &syscall.Stat_t{})\n  return ea == nil && eb == nil && sa.Dev == sb.Dev\n}\nif !sameFilesystem(bucketDir, snapshotsRoot) {\n  return errors.New(\"snapshots root must be on the same filesystem as data (hard links)\")\n}","typeGuard":"func sameFilesystem(a, b string) bool {\n  var sa, sb syscall.Stat_t\n  if syscall.Stat(a, &sa) != nil || syscall.Stat(b, &sb) != nil { return false }\n  return sa.Dev == sb.Dev\n}","tryCatchPattern":"if _, err := bucket.CreateSnapshot(ctx, root, name); err != nil {\n  if strings.Contains(err.Error(), \"hardlink snapshot\") {\n    if errors.Is(err, syscall.EXDEV) || strings.Contains(err.Error(), \"invalid argument\") {\n      moveSnapshotRootToSameFS()\n    }\n  }\n}","preventionTips":["Keep snapshotsRoot on the same filesystem/volume as the data directory","Ensure permissions allow link creation in the snapshot root","Don't delete or aggressively compact the bucket mid-snapshot"],"tags":["snapshot","hardlink","filesystem","io","lsmkv"],"backgroundTag":"hardlink-cross-device","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}