thanos-io/thanos · error

hard link block

Error message

hard link block

What it means

This error wraps any failure from hardlinkBlock during Shipper.Sync's upload of a local Prometheus TSDB block to object storage. hardlinkBlock creates hard links of the uplid block directory (s.dir.Name()/updir) into a new directory named by the block ULID, so the block gets a stable ULID-based path before meta/label modifications. It fails when the source dir is unreadable, the destination dir already exists, or the filesystem does not support hard links (e.g. cross-device link, overlayfs/NFS restrictions).

Solutions

1. Remove the stale destination directory named by the block ULID inside the shipper dir and re-run Sync.
2. Ensure the Prometheus blocks dir and shipper dir are on the same local filesystem that supports hard links (avoid NFS/overlayfs; use a bind mount of the actual data dir).
3. Check filesystem permissions on the shipper dir so the process can create directories and links.
4. Verify the source updir still exists on disk (no concurrent compaction/deletion); adjust Prometheus retention so blocks are not removed mid-sync.

Example fix

// before: stale partial upload dir blocks hardlink
$ ls data/shipper/
01F8Z...  (leftover from previous failed upload)

// after: clean stale ULID dir and retry
$ rm -rf data/shipper/01F8Z...
$ # or prevent EXDEV: mount blocks dir directly, not a copied volume

dest := filepath.Join(shipperDir, blockULID)
if _, err := os.Stat(dest); err == nil {
    if err := os.RemoveAll(dest); err != nil {
        return fmt.Errorf("stale upload dir %s: %w", dest, err)
    }
}
if err := unix.Stat(srcParent, &st); err == nil && isDifferentDevice(st, destSt) {
    return errors.New("blocks dir and shipper dir must be on the same filesystem")
}

func hardlinkSupported(dir string) bool {
    a, b := filepath.Join(dir, ".hl_a"), filepath.Join(dir, ".hl_b")
    os.WriteFile(a, []byte("x"), 0o644)
    defer os.Remove(a); defer os.Remove(b)
    return os.Link(a, b) == nil
}

if err := hardlinkBlock(dir, absUpdir); err != nil {
    if os.IsExist(err) {
        os.RemoveAll(dir)
        err = hardlinkBlock(dir, absUpdir)
    }
    if err != nil {
        return errors.Wrap(err, "hard link block")
    }
}

When it happens

Trigger: Calling Shipper.Sync when: a previous partially-failed upload left a directory named meta.ULID.String() inside the shipper dir (hardlink destination already exists); the updir source directory was deleted or renamed concurrently; or the shipper's local dir and the block dir are on different filesystems/devices so link(2) returns EXDEV.

Common situations: Prometheus running on container overlay filesystems or NFS mounts that forbid hard links; crash-recovery after a killed Sync leaving stale ULID directories; disk-level issues like permissions on the TSDB blocks dir; running Prometheus and Thanos sidecar with blocks dir bind-mounted differently so paths resolve across devices.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• write meta file
• open dir
• read dir
• stat block
• create meta fetcher

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/ca0bfd86dc1a49a4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:501

	updir := filepath.Join("thanos", "upload", meta.ULID.String())

	// Remove updir just in case.
	if err := s.dir.RemoveAll(updir); err != nil {
		return errors.Wrap(err, "clean upload directory")
	}
	if err := s.dir.MkdirAll(updir, 0750); err != nil {
		return errors.Wrap(err, "create upload dir")
	}
	defer func() {
		if err := s.dir.RemoveAll(updir); err != nil {
			level.Error(s.logger).Log("msg", "failed to clean upload directory", "err", err)
		}
	}()

	absUpdir := filepath.Join(s.dir.Name(), updir)
	dir := filepath.Join(s.dir.Name(), meta.ULID.String())
	if err := hardlinkBlock(dir, absUpdir); err != nil {
		return errors.Wrap(err, "hard link block")
	}
	// Attach current labels and write a new meta file with Thanos extensions.
	if lset := s.labels(); !lset.IsEmpty() {
		lset.Range(func(l labels.Label) {
			meta.Thanos.Labels[l.Name] = l.Value
		})
	}
	meta.Thanos.Source = s.source
	meta.Thanos.SegmentFiles = block.GetSegmentFiles(absUpdir)
	if err := meta.WriteToDir(s.logger, absUpdir); err != nil {
		return errors.Wrap(err, "write meta file")
	}
	var uploadOptions []objstore.UploadOption
	if s.uploadConcurrency > 0 {
		uploadOptions = append(uploadOptions, objstore.WithUploadConcurrency(s.uploadConcurrency))
	}
	return block.Upload(ctx, s.logger, s.bucket, absUpdir, s.hashFunc, uploadOptions...)
}

View on GitHub (pinned to 35b8b99117)