{"record":{"id":"02b94d068a0e41e2","repo":"weaviate/weaviate","slug":"hardlink-inactive-shard-s-files-to-staging-w","errorCode":null,"errorMessage":"hardlink inactive shard %s files to staging: %w","messagePattern":"hardlink inactive shard (.+?) files to staging: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/backup.go","lineNumber":471,"sourceCode":"\tfor _, relPath := range files {\n\t\tsrc := filepath.Join(i.Config.RootPath, relPath)\n\t\tdst := filepath.Join(stagingRoot, relPath)\n\t\tif backup.IsImmutableFile(relPath) {\n\t\t\thardlinks = append(hardlinks, file.HardlinkPair{Src: src, Dst: dst})\n\t\t\tcontinue\n\t\t}\n\t\t// Mutable files are copied, not hard-linked — a shared inode would let\n\t\t// post-snapshot writes corrupt the staged copy. CopyFile, unlike\n\t\t// HardlinkFiles, doesn't create the destination dir.\n\t\tif err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {\n\t\t\treturn fmt.Errorf(\"create staging subdir for inactive shard %s file %s: %w\", name, relPath, err)\n\t\t}\n\t\tif err := file.CopyFile(src, dst); err != nil {\n\t\t\treturn fmt.Errorf(\"copy inactive shard %s file %s to staging: %w\", name, relPath, err)\n\t\t}\n\t}\n\tif err := file.HardlinkFiles(hardlinks); err != nil {\n\t\treturn fmt.Errorf(\"hardlink inactive shard %s files to staging: %w\", name, err)\n\t}\n\n\tif err := sd.FillFileInfo(files, shardBaseDescr, i.Config.RootPath); err != nil {\n\t\treturn fmt.Errorf(\"gather inactive shard %s file info: %w\", name, err)\n\t}\n\n\treturn nil\n}\n\n// descriptorWithoutHardlinks is the fallback path for filesystems that don't support\n// hardlinks. Compaction remains paused for the entire backup upload duration.\n//\n// Deprecated: NO-HARDLINK-BACKUP. Removed in v1.40; bugs here are not fixed.\nfunc (i *Index) descriptorWithoutHardlinks(ctx context.Context, backupID string, desc *backup.ClassDescriptor, classBaseDescrs []*backup.ClassDescriptor) (err error) {\n\tdefer func() {\n\t\tif err != nil {\n\t\t\t// closelock is hold by the caller\n\t\t\tenterrors.GoWrapper(func() { i.ReleaseBackup(ctx, backupID) }, i.logger)","sourceCodeStart":453,"sourceCodeEnd":489,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/backup.go#L453-L489","documentation":"Wraps a failure from file.HardlinkFiles when hardlinking shard files into the backup staging directory for an inactive shard. Hardlinks are used to snapshot files cheaply without copying; this fails on filesystems without hardlink support, cross-device targets, permission problems, or if a source file vanished.","triggerScenarios":"Backup descriptor creation for an inactive shard when HardlinkFiles fails: staging dir on a different filesystem (EXDEV), overlay/NFS/FAT filesystem without hardlink support, or EACCES/EPERM creating links.","commonSituations":"Docker overlayfs or network mounts (NFS/SMB) where hardlinks are disallowed or cross-device; staging path outside the data root on another mount.","solutions":["Ensure the backup staging directory is on the same filesystem as the shard data root","If the filesystem does not support hardlinks, use the no-hardlinks fallback path (descriptorWithoutHardlinks)","Check mount options and permissions on the staging directory","Read the wrapped OS error (EXDEV/EPERM) to confirm the cause"],"exampleFix":"// before: hardlinks attempted regardless of FS support\nif err := file.HardlinkFiles(hardlinks); err != nil {\n\treturn fmt.Errorf(\"hardlink inactive shard %s files to staging: %w\", name, err)\n}\n// after: detect cross-device/link failure and fall back to copying\nif err := file.HardlinkFiles(hardlinks); err != nil {\n\tif errors.Is(err, syscall.EXDEV) || errors.Is(err, syscall.EPERM) {\n\t\treturn i.backupInactiveShardWithoutHardlinks(ctx, name) // copy-based fallback\n\t}\n\treturn fmt.Errorf(\"hardlink inactive shard %s files to staging: %w\", name, err)\n}","handlingStrategy":"fallback","validationCode":"// detect hardlink support on the target filesystem\nif err := os.Link(probeSrc, probeDst); err != nil {\n\t// use the no-hardlinks backup path\n}","typeGuard":null,"tryCatchPattern":"if err := backup(); err != nil {\n\tif strings.Contains(err.Error(), \"hardlink inactive shard\") {\n\t\t// retry with the no-hardlinks fallback path\n\t}\n\treturn err\n}","preventionTips":["Store backups on the same filesystem/device as the data","Avoid overlayfs/NFS for staging if hardlinks are required","Know your filesystem's hardlink limitations before enabling the hardlink path"],"tags":["filesystem","backup","hardlink"],"backgroundTag":"hardlink-not-supported","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}