{"record":{"id":"ca0bfd86dc1a49a4","repo":"thanos-io/thanos","slug":"hard-link-block","errorCode":null,"errorMessage":"hard link block","messagePattern":"hard link block","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":501,"sourceCode":"\tupdir := filepath.Join(\"thanos\", \"upload\", meta.ULID.String())\n\n\t// Remove updir just in case.\n\tif err := s.dir.RemoveAll(updir); err != nil {\n\t\treturn errors.Wrap(err, \"clean upload directory\")\n\t}\n\tif err := s.dir.MkdirAll(updir, 0750); err != nil {\n\t\treturn errors.Wrap(err, \"create upload dir\")\n\t}\n\tdefer func() {\n\t\tif err := s.dir.RemoveAll(updir); err != nil {\n\t\t\tlevel.Error(s.logger).Log(\"msg\", \"failed to clean upload directory\", \"err\", err)\n\t\t}\n\t}()\n\n\tabsUpdir := filepath.Join(s.dir.Name(), updir)\n\tdir := filepath.Join(s.dir.Name(), meta.ULID.String())\n\tif err := hardlinkBlock(dir, absUpdir); err != nil {\n\t\treturn errors.Wrap(err, \"hard link block\")\n\t}\n\t// Attach current labels and write a new meta file with Thanos extensions.\n\tif lset := s.labels(); !lset.IsEmpty() {\n\t\tlset.Range(func(l labels.Label) {\n\t\t\tmeta.Thanos.Labels[l.Name] = l.Value\n\t\t})\n\t}\n\tmeta.Thanos.Source = s.source\n\tmeta.Thanos.SegmentFiles = block.GetSegmentFiles(absUpdir)\n\tif err := meta.WriteToDir(s.logger, absUpdir); err != nil {\n\t\treturn errors.Wrap(err, \"write meta file\")\n\t}\n\tvar uploadOptions []objstore.UploadOption\n\tif s.uploadConcurrency > 0 {\n\t\tuploadOptions = append(uploadOptions, objstore.WithUploadConcurrency(s.uploadConcurrency))\n\t}\n\treturn block.Upload(ctx, s.logger, s.bucket, absUpdir, s.hashFunc, uploadOptions...)\n}","sourceCodeStart":483,"sourceCodeEnd":519,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L483-L519","documentation":"This error wraps any failure from hardlinkBlock during Shipper.Sync's upload of a local Prometheus TSDB block to object storage. hardlinkBlock creates hard links of the uplid block directory (s.dir.Name()/updir) into a new directory named by the block ULID, so the block gets a stable ULID-based path before meta/label modifications. It fails when the source dir is unreadable, the destination dir already exists, or the filesystem does not support hard links (e.g. cross-device link, overlayfs/NFS restrictions).","triggerScenarios":"Calling Shipper.Sync when: a previous partially-failed upload left a directory named meta.ULID.String() inside the shipper dir (hardlink destination already exists); the updir source directory was deleted or renamed concurrently; or the shipper's local dir and the block dir are on different filesystems/devices so link(2) returns EXDEV.","commonSituations":"Prometheus running on container overlay filesystems or NFS mounts that forbid hard links; crash-recovery after a killed Sync leaving stale ULID directories; disk-level issues like permissions on the TSDB blocks dir; running Prometheus and Thanos sidecar with blocks dir bind-mounted differently so paths resolve across devices.","solutions":["Remove the stale destination directory named by the block ULID inside the shipper dir and re-run Sync.","Ensure the Prometheus blocks dir and shipper dir are on the same local filesystem that supports hard links (avoid NFS/overlayfs; use a bind mount of the actual data dir).","Check filesystem permissions on the shipper dir so the process can create directories and links.","Verify the source updir still exists on disk (no concurrent compaction/deletion); adjust Prometheus retention so blocks are not removed mid-sync."],"exampleFix":"// before: stale partial upload dir blocks hardlink\n$ ls data/shipper/\n01F8Z...  (leftover from previous failed upload)\n\n// after: clean stale ULID dir and retry\n$ rm -rf data/shipper/01F8Z...\n$ # or prevent EXDEV: mount blocks dir directly, not a copied volume","handlingStrategy":"validation","validationCode":"dest := filepath.Join(shipperDir, blockULID)\nif _, err := os.Stat(dest); err == nil {\n    if err := os.RemoveAll(dest); err != nil {\n        return fmt.Errorf(\"stale upload dir %s: %w\", dest, err)\n    }\n}\nif err := unix.Stat(srcParent, &st); err == nil && isDifferentDevice(st, destSt) {\n    return errors.New(\"blocks dir and shipper dir must be on the same filesystem\")\n}","typeGuard":"func hardlinkSupported(dir string) bool {\n    a, b := filepath.Join(dir, \".hl_a\"), filepath.Join(dir, \".hl_b\")\n    os.WriteFile(a, []byte(\"x\"), 0o644)\n    defer os.Remove(a); defer os.Remove(b)\n    return os.Link(a, b) == nil\n}","tryCatchPattern":"if err := hardlinkBlock(dir, absUpdir); err != nil {\n    if os.IsExist(err) {\n        os.RemoveAll(dir)\n        err = hardlinkBlock(dir, absUpdir)\n    }\n    if err != nil {\n        return errors.Wrap(err, \"hard link block\")\n    }\n}","preventionTips":["Mount the Prometheus data dir directly (bind mount), never via a copy or overlay layer.","Clean up leftover ULID-named directories in the shipper dir after failed syncs.","Keep shipper dir and block dir on the same local filesystem that supports hard links.","Monitor disk health; hardlink failures often co-occur with I/O errors."],"tags":["go","filesystem","hardlink","shipper","object-storage"],"backgroundTag":"file-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}