thanos-io/thanos · error

output block index not valid

Error message

output block index not valid

What it means

After downsample.Downsample produces a new block, processDownsampling runs block.GatherIndexHealthStats on the output index and wraps failures as "output block index not valid" unless acceptMalformedIndex is set. The newly generated downsampled block's index fails health verification (e.g. stats.AnyErr() reports issues like oversized or inconsistent chunks).

Solutions

1. Re-run downsampling after removing the intermediate directory for that block in --data-dir to force a clean regeneration.
2. Upgrade Thanos: several downsampler index-quality bugs (chunk max size, resolution handling) were fixed over time.
3. Use --accept-malformed-index deliberately if the identified issue is known-benign for your setup.
4. Check the parent block's index health first; a bad input produces a bad output.

stats, err := block.GatherIndexHealthStats(ctx, logger, filepath.Join(resdir, block.IndexFilename), m.MinTime, m.MaxTime)
if err == nil { err = stats.AnyErr() }
if err != nil {
    os.RemoveAll(resdir) // discard bad output before upload
}

When it happens

Trigger: GatherIndexHealthStats returns an error or stats.AnyErr() is non-nil, and --accept-malformed-index is false: downsampled index has postings/timestamps outside m.MinTime..m.MaxTime or other health-stat failures.

Common situations: Downsampling blocks that already contained marginal/corrupt data; historic Thanos downsampler bugs producing indexes with too-large chunks; verifying output against the parent block's time range when output legitimately differs.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• input block index not valid
• gather index issues for block
• raw resolution must be higher than the minimum block size…
• 5m resolution retention must be higher than the minimum…
• sync before first pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/cd40ff76df148a67. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:401

	defer runutil.CloseWithLogOnErr(log.With(logger, "outcome", "potential left mmap file handlers left"), b, "tsdb reader")

	id, err := downsample.Downsample(ctx, logger, m, b, dir, resolution)
	if err != nil {
		return errors.Wrapf(err, "downsample block %s to window %d", m.ULID, resolution)
	}
	resdir := filepath.Join(dir, id.String())

	downsampleDuration := time.Since(begin)
	level.Info(logger).Log("msg", "downsampled block",
		"from", m.ULID, "to", id, "duration", downsampleDuration, "duration_ms", downsampleDuration.Milliseconds())
	metrics.downsampleDuration.WithLabelValues(m.Thanos.ResolutionString()).Observe(downsampleDuration.Seconds())

	stats, err := block.GatherIndexHealthStats(ctx, logger, filepath.Join(resdir, block.IndexFilename), m.MinTime, m.MaxTime)
	if err == nil {
		err = stats.AnyErr()
	}
	if err != nil && !acceptMalformedIndex {
		return errors.Wrap(err, "output block index not valid")
	}

	meta, err := metadata.ReadFromDir(resdir)
	if err != nil {
		return errors.Wrap(err, "read meta")
	}

	if stats.ChunkMaxSize > 0 {
		meta.Thanos.IndexStats.ChunkMaxSize = stats.ChunkMaxSize
	}
	if stats.SeriesMaxSize > 0 {
		meta.Thanos.IndexStats.SeriesMaxSize = stats.SeriesMaxSize
	}
	if err := meta.WriteToDir(logger, resdir); err != nil {
		return errors.Wrap(err, "write meta")
	}

	begin = time.Now()

View on GitHub (pinned to 35b8b99117)