thanos-io/thanos · error

write meta

Error message

write meta

What it means

processDownsampling wraps meta.WriteToDir failures as "write meta". The enriched metadata (with index stats chunk/series max sizes) cannot be written to the output block's meta.json. This blocks the subsequent upload of the downsampled block.

Solutions

1. Verify --data-dir has free space and is writable by the thanos process user; clean stale block dirs.
2. In Kubernetes, size emptyDir/PV adequately and check kubelet eviction logs.
3. Ensure a single downsampler instance owns the data dir; remove MAC (SELinux) denials from audit logs if present.
4. Re-run after cleanup — the operation is idempotent per block and will regenerate the output.

// preflight before writing
if err := syscall.Access(resdir, os.O_WRONLY); err != nil {
    return fmt.Errorf("output dir %s not writable: %w", resdir, err)
}

if err := meta.WriteToDir(logger, resdir); err != nil {
    return errors.Wrap(err, "write meta") // check disk space/permissions before rerunning
}

When it happens

Trigger: meta.WriteToDir(logger, resdir) fails: permission denied on resdir, disk full, resdir removed mid-run, or filesystem I/O error while serializing meta.json.

Common situations: --data-dir on a full or read-only volume; quota exceeded on ephemeral storage in Kubernetes; concurrent processes racing over the same data dir; SELinux/AppArmor blocking writes.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• read meta file
• failed to read
• create meta fetcher
• create working compact directory
• create working downsample directory

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/94d49f5c8c48c6b6. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:416

		err = stats.AnyErr()
	}
	if err != nil && !acceptMalformedIndex {
		return errors.Wrap(err, "output block index not valid")
	}

	meta, err := metadata.ReadFromDir(resdir)
	if err != nil {
		return errors.Wrap(err, "read meta")
	}

	if stats.ChunkMaxSize > 0 {
		meta.Thanos.IndexStats.ChunkMaxSize = stats.ChunkMaxSize
	}
	if stats.SeriesMaxSize > 0 {
		meta.Thanos.IndexStats.SeriesMaxSize = stats.SeriesMaxSize
	}
	if err := meta.WriteToDir(logger, resdir); err != nil {
		return errors.Wrap(err, "write meta")
	}

	begin = time.Now()

	err = block.Upload(ctx, logger, bkt, resdir, hashFunc, objstore.WithUploadConcurrency(blockFilesConcurrency))
	if err != nil {
		return compact.NewRetryError(errors.Wrapf(err, "upload downsampled block %s", id))
	}

	level.Info(logger).Log("msg", "uploaded block", "id", id, "duration", time.Since(begin), "duration_ms", time.Since(begin).Milliseconds())

	// It is not harmful if these fails.
	if err := os.RemoveAll(bdir); err != nil {
		level.Warn(logger).Log("msg", "failed to clean directory", "dir", bdir, "err", err)
	}
	if err := os.RemoveAll(resdir); err != nil {
		level.Warn(logger).Log("msg", "failed to clean directory", "resdir", bdir, "err", err)
	}

View on GitHub (pinned to 35b8b99117)