thanos-io/thanos · error

read chunk dir

Error message

read chunk dir

What it means

hardlinkBlock reads the source block's chunks directory (os.ReadDir on <src>/chunks) to enumerate the chunk files to hardlink into the staging area. Failure here means the source block directory is missing its chunks subdirectory or is unreadable, so the block cannot be staged for upload and the error is wrapped as "read chunk dir".

Solutions

1. Verify the source block directory contains a readable chunks/ subdirectory (ls <block>/chunks).
2. Remove the corrupted/incomplete block from the data dir so the source re-materializes it, or restore it from a healthy replica.
3. Check for cleanup jobs or concurrent compaction deleting chunk files while the shipper uploads; pause them or exclude the shipper's dir.
4. Check mount health and permissions on the data directory if ReadDir fails with EACCES/ESTALE.

chunksDir := filepath.Join(blockDir, block.ChunksDirname)
if fi, err := os.Stat(chunksDir); err != nil || !fi.IsDir() {
	return fmt.Errorf("block %s missing chunks dir", blockDir)
}

func hasChunksDir(blockDir string) bool {
	fi, err := os.Stat(filepath.Join(blockDir, "chunks"))
	return err == nil && fi.IsDir()
}

if err := sh.Sync(ctx); err != nil {
	if strings.Contains(err.Error(), "read chunk dir") {
		level.Warn(logger).Log("msg", "block missing chunks, removing", "err", err)
	}
	return err
}

When it happens

Trigger: Sync()/upload() staging a block whose <block-dir>/chunks directory does not exist, was deleted, or cannot be read due to permissions or I/O error.

Common situations: A partially written or manually truncated block (missing chunks dir); external tooling (cleanup scripts, compaction) removing files while the shipper runs; wrong permissions after copying blocks between hosts; NFS stale handles.

Related errors

• is not a directory
• clean upload directory
• create upload dir
• read metadata for block
• create chunks dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/b9a748d834db94b6. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:588

		return metas[i].MinTime < metas[j].MinTime
	})

	if len(failedBlocks) > 0 {
		err = ErrorSyncBlockCorrupted
	}
	return metas, failedBlocks, err
}

func hardlinkBlock(src, dst string) error {
	chunkDir := filepath.Join(dst, block.ChunksDirname)

	if err := os.MkdirAll(chunkDir, 0750); err != nil {
		return errors.Wrap(err, "create chunks dir")
	}

	fis, err := os.ReadDir(filepath.Join(src, block.ChunksDirname))
	if err != nil {
		return errors.Wrap(err, "read chunk dir")
	}
	files := make([]string, 0, len(fis))
	for _, fi := range fis {
		files = append(files, fi.Name())
	}
	for i, fn := range files {
		files[i] = filepath.Join(block.ChunksDirname, fn)
	}
	files = append(files, block.MetaFilename, block.IndexFilename)

	for _, fn := range files {
		if err := os.Link(filepath.Join(src, fn), filepath.Join(dst, fn)); err != nil {
			return errors.Wrapf(err, "hard link file %s", fn)
		}
	}
	return nil
}

View on GitHub (pinned to 35b8b99117)