{"record":{"id":"cf1e3c2e7cf838e9","repo":"thanos-io/thanos","slug":"hard-link-file-s","errorCode":null,"errorMessage":"hard link file %s","messagePattern":"hard link file (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":601,"sourceCode":"\t\treturn errors.Wrap(err, \"create chunks dir\")\n\t}\n\n\tfis, err := os.ReadDir(filepath.Join(src, block.ChunksDirname))\n\tif err != nil {\n\t\treturn errors.Wrap(err, \"read chunk dir\")\n\t}\n\tfiles := make([]string, 0, len(fis))\n\tfor _, fi := range fis {\n\t\tfiles = append(files, fi.Name())\n\t}\n\tfor i, fn := range files {\n\t\tfiles[i] = filepath.Join(block.ChunksDirname, fn)\n\t}\n\tfiles = append(files, block.MetaFilename, block.IndexFilename)\n\n\tfor _, fn := range files {\n\t\tif err := os.Link(filepath.Join(src, fn), filepath.Join(dst, fn)); err != nil {\n\t\t\treturn errors.Wrapf(err, \"hard link file %s\", fn)\n\t\t}\n\t}\n\treturn nil\n}\n\n// Meta defines the format thanos.shipper.json file that the shipper places in the data directory.\ntype Meta struct {\n\tVersion  int         `json:\"version\"`\n\tUploaded []ulid.ULID `json:\"uploaded\"`\n}\n\nconst (\n\t// DefaultMetaFilename is the default JSON filename for meta information.\n\tDefaultMetaFilename = \"thanos.shipper.json\"\n\n\t// MetaVersion1 represents 1 version of meta.\n\tMetaVersion1 = 1\n)","sourceCodeStart":583,"sourceCodeEnd":619,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L583-L619","documentation":"hardlinkBlock hardlinks every chunk file plus meta.json and index.json from the source block into the staging directory using os.Link. If linking any individual file fails, the upload aborts with \"hard link file <name>\". Hardlinks require both paths on the same filesystem, so this commonly fails when the staging/temp dir and the block data dir live on different mounts.","triggerScenarios":"Sync()/upload() calling os.Link(src/<fn>, dst/<fn>) for a chunk file, meta.json, or index.json when the link fails — cross-device link, missing source file, or permissions.","commonSituations":"Data dir and temp/staging dir on different filesystems or Docker volumes (EXDEV: invalid cross-device link); chunk file deleted by concurrent compaction mid-upload; read-only or permission-restricted source; disk full on destination (though hardlinks themselves need no space, the earlier mkdir would).","solutions":["Ensure the shipper's local data directory and its temporary/staging directory are on the same filesystem (hardlinks cannot cross mounts — EXDEV).","Check the named file still exists in the source block; if it vanished, another process (compaction/cleanup) is racing the shipper — exclude the dir or pause it.","Verify write permission on the destination staging directory and read permission on the source block.","If the filesystem does not support hardlinks (some NFS/FUSE/CIFS setups), configure Thanos to avoid hardlinking or move data onto a POSIX filesystem."],"exampleFix":"// before: temp dir on a different volume than data dir\n--tsdb.path=/data  --tmp.dir=/tmp\n// after: same filesystem so os.Link succeeds\n--tsdb.path=/data  --tmp.dir=/data/tmp","handlingStrategy":"validation","validationCode":"srcFi, _ := os.Stat(dataDir)\ndstFi, _ := os.Stat(tmpDir)\nif !sameDevice(srcFi, dstFi) {\n\treturn fmt.Errorf(\"data dir and tmp dir must be on the same filesystem for hardlinks\")\n}\nfunc sameDevice(a, b os.FileInfo) bool { return a.Sys().(*syscall.Stat_t).Dev == b.Sys().(*syscall.Stat_t).Dev }","typeGuard":null,"tryCatchPattern":"var pathErr *os.LinkError\nif errors.As(err, &pathErr) {\n\tlevel.Error(logger).Log(\"msg\", \"hardlink failed (cross-device?)\", \"err\", err)\n}","preventionTips":["Keep the TSDB data dir and any temp/staging dir on the same filesystem/mount.","In containers, mount both paths from the same volume, not separate volume types.","Do not run compaction/cleanup concurrently with shipping on the same directory.","Prefer local POSIX filesystems over NFS/FUSE that may not support hardlinks."],"tags":["filesystem","hardlink","upload","cross-device"],"backgroundTag":"file-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}