thanos-io/thanos · error

hard link file

Error message

hard link file %s

What it means

hardlinkBlock hardlinks every chunk file plus meta.json and index.json from the source block into the staging directory using os.Link. If linking any individual file fails, the upload aborts with "hard link file <name>". Hardlinks require both paths on the same filesystem, so this commonly fails when the staging/temp dir and the block data dir live on different mounts.

Solutions

1. Ensure the shipper's local data directory and its temporary/staging directory are on the same filesystem (hardlinks cannot cross mounts — EXDEV).
2. Check the named file still exists in the source block; if it vanished, another process (compaction/cleanup) is racing the shipper — exclude the dir or pause it.
3. Verify write permission on the destination staging directory and read permission on the source block.
4. If the filesystem does not support hardlinks (some NFS/FUSE/CIFS setups), configure Thanos to avoid hardlinking or move data onto a POSIX filesystem.

Example fix

// before: temp dir on a different volume than data dir
--tsdb.path=/data  --tmp.dir=/tmp
// after: same filesystem so os.Link succeeds
--tsdb.path=/data  --tmp.dir=/data/tmp

srcFi, _ := os.Stat(dataDir)
dstFi, _ := os.Stat(tmpDir)
if !sameDevice(srcFi, dstFi) {
	return fmt.Errorf("data dir and tmp dir must be on the same filesystem for hardlinks")
}
func sameDevice(a, b os.FileInfo) bool { return a.Sys().(*syscall.Stat_t).Dev == b.Sys().(*syscall.Stat_t).Dev }

var pathErr *os.LinkError
if errors.As(err, &pathErr) {
	level.Error(logger).Log("msg", "hardlink failed (cross-device?)", "err", err)
}

When it happens

Trigger: Sync()/upload() calling os.Link(src/<fn>, dst/<fn>) for a chunk file, meta.json, or index.json when the link fails — cross-device link, missing source file, or permissions.

Common situations: Data dir and temp/staging dir on different filesystems or Docker volumes (EXDEV: invalid cross-device link); chunk file deleted by concurrent compaction mid-upload; read-only or permission-restricted source; disk full on destination (though hardlinks themselves need no space, the earlier mkdir would).

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• is not a directory
• clean upload directory
• create upload dir
• hard link block
• create chunks dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/cf1e3c2e7cf838e9. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:601

		return errors.Wrap(err, "create chunks dir")
	}

	fis, err := os.ReadDir(filepath.Join(src, block.ChunksDirname))
	if err != nil {
		return errors.Wrap(err, "read chunk dir")
	}
	files := make([]string, 0, len(fis))
	for _, fi := range fis {
		files = append(files, fi.Name())
	}
	for i, fn := range files {
		files[i] = filepath.Join(block.ChunksDirname, fn)
	}
	files = append(files, block.MetaFilename, block.IndexFilename)

	for _, fn := range files {
		if err := os.Link(filepath.Join(src, fn), filepath.Join(dst, fn)); err != nil {
			return errors.Wrapf(err, "hard link file %s", fn)
		}
	}
	return nil
}

// Meta defines the format thanos.shipper.json file that the shipper places in the data directory.
type Meta struct {
	Version  int         `json:"version"`
	Uploaded []ulid.ULID `json:"uploaded"`
}

const (
	// DefaultMetaFilename is the default JSON filename for meta information.
	DefaultMetaFilename = "thanos.shipper.json"

	// MetaVersion1 represents 1 version of meta.
	MetaVersion1 = 1
)

View on GitHub (pinned to 35b8b99117)