thanos-io/thanos · error

open chunk writer

Error message

open chunk writer

What it means

block.Repair wraps the error from chunks.NewWriter, which creates the chunk writer for the new block directory chunks/<resid>/chunks/. Failure is almost always an OS-level problem creating the destination directory/files — missing parent dir, permission denied, or ENOSPC — not data corruption. Wrapped as 'open chunk writer: <cause>'.

Solutions

1. Ensure the target dir exists and is writable by the process running Repair (mkdir -p and chown/chmod as needed).
2. Free disk space if the wrapped error is ENOSPC.
3. Remove leftover partial result directories from earlier failed repairs before retrying.
4. Copy blocks to a local writable scratch dir, repair there, then upload the result instead of repairing in place on read-only mounts.

Example fix

// before
if _, err := block.Repair(ctx, logger, os.Getenv("MOUNT"), id, source, fn) { ... } // MOUNT may be read-only
// after
workdir := "/var/tmp/thanos-repair"
if err := os.MkdirAll(workdir, 0o750); err != nil {
    return err
}
if err := copyBlock(filepath.Join(workdir, id.String()), dir, id); err != nil {
    return err
}
if _, err := block.Repair(ctx, logger, workdir, id, source, fn); err != nil {
    return err
}

if fi, err := os.Stat(dir); err != nil || !fi.IsDir() {
    return fmt.Errorf("repair target dir %s unavailable", dir)
}
test := filepath.Join(dir, ".repair-write-test")
if err := os.WriteFile(test, []byte("ok"), 0o600); err != nil {
    return fmt.Errorf("dir %s not writable: %w", dir, err)
}
os.Remove(test)

resid, err := block.RepairIssue347(ctx, logger, dir, id, source)
if err != nil && strings.Contains(err.Error(), "open chunk writer") {
    return resid, fmt.Errorf("cannot write repaired block into %s: %w (check permissions/free space)", dir, err)
}

When it happens

Trigger: Repair on a dir where the new block's parent directory cannot be created (dir doesn't exist or is read-only); filesystem full; path too long; stale partial resdir from a previous failed repair run conflicting with permissions.

Common situations: Running repair against an object-storage-mounted (read-only FUSE) path; container running with a read-only rootfs or non-writable workdir; disk full during a large repair; repairing into a dir owned by a different user than the repair process.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• create working compact directory
• create working downsample directory
• retention failed
• create dir
• create default tenant data dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e6787b384f417feb. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:454

	defer runutil.CloseWithErrCapture(&err, b, "repair block reader")

	indexr, err := b.Index()
	if err != nil {
		return resid, errors.Wrap(err, "open index")
	}
	defer runutil.CloseWithErrCapture(&err, indexr, "repair index reader")

	chunkr, err := b.Chunks()
	if err != nil {
		return resid, errors.Wrap(err, "open chunks")
	}
	defer runutil.CloseWithErrCapture(&err, chunkr, "repair chunk reader")

	resdir := filepath.Join(dir, resid.String())

	chunkw, err := chunks.NewWriter(filepath.Join(resdir, ChunksDirname))
	if err != nil {
		return resid, errors.Wrap(err, "open chunk writer")
	}
	defer runutil.CloseWithErrCapture(&err, chunkw, "repair chunk writer")

	indexw, err := index.NewWriter(context.TODO(), filepath.Join(resdir, IndexFilename))
	if err != nil {
		return resid, errors.Wrap(err, "open index writer")
	}
	defer runutil.CloseWithErrCapture(&err, indexw, "repair index writer")

	// TODO(fabxc): adapt so we properly handle the version once we update to an upstream
	// that has multiple.
	resmeta := *meta
	resmeta.ULID = resid
	resmeta.Stats = tsdb.BlockStats{} // Reset stats.
	resmeta.Thanos.Source = source    // Update source.

	if err := rewrite(ctx, logger, indexr, chunkr, indexw, chunkw, &resmeta, ignoreChkFns); err != nil {
		return resid, errors.Wrap(err, "rewrite block")

View on GitHub (pinned to 35b8b99117)