thanos-io/thanos · error

create chunks dir

Error message

create chunks dir

What it means

During upload, the Shipper stages each block via hardlinkBlock, which first creates the chunks directory (<staging dst>/chunks) with os.MkdirAll. If that mkdir fails, the upload of the block cannot proceed and the error is wrapped with "create chunks dir". This is almost always an OS-level problem: permissions, a full disk, or the destination path not being writable.

Solutions

1. Check permissions and ownership of the shipper's local data directory and ensure the process UID can create directories there (0750 needs write on parent).
2. Verify the volume backing the data dir is writable and not full (df -h; mount | grep <path>).
3. In Kubernetes, add an emptyDir/PVC for the data dir and drop readOnlyRootFilesystem or add writableVolumeMounts.
4. Retry after fixing; if mkdir fails transiently due to NFS, remount the share.

func canCreateDir(parent string) error {
	fi, err := os.Stat(parent)
	if err != nil {
		return err
	}
	if !fi.IsDir() {
		return fmt.Errorf("%s is not a directory", parent)
	}
	return os.Access(parent, os.O_RDWR)
}

if err := sh.Sync(ctx); err != nil {
	if strings.Contains(err.Error(), "create chunks dir") {
		level.Error(logger).Log("msg", "staging dir unwritable", "err", err)
	}
	return err
}

When it happens

Trigger: Calling upload() (via Sync) when os.MkdirAll(<dst>/chunks, 0750) fails because the staging/destination parent dir is unwritable, the path is invalid, or the filesystem is full/read-only.

Common situations: Data directory volume mounted read-only (e.g. k8s readOnlyRootFilesystem); disk full during large compaction uploads; wrong ownership/permissions on the local data dir after running the container as a different UID; SELinux/AppArmor blocking writes.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• clean upload directory
• create upload dir
• create working compact directory
• create working downsample directory
• create dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/9a0ddea357b93d4a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:583

			return nil, nil, errors.Wrapf(err, "read metadata for block %v", dir)
		}
		metas = append(metas, m)
	}
	sort.Slice(metas, func(i, j int) bool {
		return metas[i].MinTime < metas[j].MinTime
	})

	if len(failedBlocks) > 0 {
		err = ErrorSyncBlockCorrupted
	}
	return metas, failedBlocks, err
}

func hardlinkBlock(src, dst string) error {
	chunkDir := filepath.Join(dst, block.ChunksDirname)

	if err := os.MkdirAll(chunkDir, 0750); err != nil {
		return errors.Wrap(err, "create chunks dir")
	}

	fis, err := os.ReadDir(filepath.Join(src, block.ChunksDirname))
	if err != nil {
		return errors.Wrap(err, "read chunk dir")
	}
	files := make([]string, 0, len(fis))
	for _, fi := range fis {
		files = append(files, fi.Name())
	}
	for i, fn := range files {
		files[i] = filepath.Join(block.ChunksDirname, fn)
	}
	files = append(files, block.MetaFilename, block.IndexFilename)

	for _, fn := range files {
		if err := os.Link(filepath.Join(src, fn), filepath.Join(dst, fn)); err != nil {
			return errors.Wrapf(err, "hard link file %s", fn)

View on GitHub (pinned to 35b8b99117)