thanos-io/thanos · error

read meta

Error message

read meta

What it means

processDownsampling wraps metadata.ReadFromDir(resdir) failures as "read meta". The meta.json of the freshly downsampled output block (resdir) cannot be read or parsed. This normally means the file is missing or not valid JSON/meta schema.

Solutions

1. Delete the partial output directory for that block under --data-dir and let the next cycle regenerate it.
2. Check free disk space and dmesg for OOM/IO errors during the downsampling run.
3. Ensure nothing else (cron cleanup, shared volume users) mutates --data-dir while thanos downsample runs.

metaPath := filepath.Join(resdir, "meta.json")
if _, err := os.Stat(metaPath); err != nil {
    os.RemoveAll(resdir) // incomplete output; force regeneration
}

meta, err := metadata.ReadFromDir(resdir)
if err != nil {
    os.RemoveAll(resdir)
    return errors.Wrap(err, "read meta")
}

When it happens

Trigger: metadata.ReadFromDir on the result directory fails because meta.json was not written by the downsample step (e.g. process killed mid-write), JSON is malformed, or the directory was removed concurrently.

Common situations: Disk full during block write so meta.json is truncated; operator/scripts deleting files in --data-dir while downsampling runs; OOM kill between block creation and meta write.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• meta.json corrupted
• file: ; err
• could not sync metas
• open block
• write meta

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/1d3194d0e10dad98. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:406

	}
	resdir := filepath.Join(dir, id.String())

	downsampleDuration := time.Since(begin)
	level.Info(logger).Log("msg", "downsampled block",
		"from", m.ULID, "to", id, "duration", downsampleDuration, "duration_ms", downsampleDuration.Milliseconds())
	metrics.downsampleDuration.WithLabelValues(m.Thanos.ResolutionString()).Observe(downsampleDuration.Seconds())

	stats, err := block.GatherIndexHealthStats(ctx, logger, filepath.Join(resdir, block.IndexFilename), m.MinTime, m.MaxTime)
	if err == nil {
		err = stats.AnyErr()
	}
	if err != nil && !acceptMalformedIndex {
		return errors.Wrap(err, "output block index not valid")
	}

	meta, err := metadata.ReadFromDir(resdir)
	if err != nil {
		return errors.Wrap(err, "read meta")
	}

	if stats.ChunkMaxSize > 0 {
		meta.Thanos.IndexStats.ChunkMaxSize = stats.ChunkMaxSize
	}
	if stats.SeriesMaxSize > 0 {
		meta.Thanos.IndexStats.SeriesMaxSize = stats.SeriesMaxSize
	}
	if err := meta.WriteToDir(logger, resdir); err != nil {
		return errors.Wrap(err, "write meta")
	}

	begin = time.Now()

	err = block.Upload(ctx, logger, bkt, resdir, hashFunc, objstore.WithUploadConcurrency(blockFilesConcurrency))
	if err != nil {
		return compact.NewRetryError(errors.Wrapf(err, "upload downsampled block %s", id))
	}

View on GitHub (pinned to 35b8b99117)