thanos-io/thanos · error

block id , try running with --debug.accept-malformed-index

Error message

block id %s, try running with --debug.accept-malformed-index

What it means

The index has out-of-order labels (a known corruption class) and the compactor is not running with --debug.accept-malformed-index. Because this malformed index could produce wrong query results, compaction fails with an error advising the escape-hatch flag. Unlike issue347, this is not auto-repaired — the operator must either fix the block or explicitly accept the malformed index.

Solutions

1. Repair or delete the offending block: run 'thanos tools bucket verify --repair' to remove malformed blocks
2. If the data is acceptable for your use case, start compactor with --debug.accept-malformed-index to skip this check
3. Re-upload the block from source Prometheus so a healthy index is written
4. Check Thanos release notes — validation strictness may have changed with your upgrade

Example fix

// before
# thanos compact --data-dir /data --objstore.bucket prom // fails on out-of-order labels
// after
# thanos compact --data-dir /data --objstore.bucket prom --debug.accept-malformed-index

// Scan bucket for malformed indexes before compaction runs:
thanos tools bucket verify --repair --objstore.bucket prom

if err != nil && strings.Contains(err.Error(), "--debug.accept-malformed-index") {
    log.Error(err, "malformed index detected; repair the block or enable the debug flag")
}

When it happens

Trigger: stats.OutOfOrderLabelsErr() returns non-nil while cg.acceptMalformedIndex is false, during block verification after download.

Common situations: Legacy buckets with blocks written by old buggy Prometheus writers; operators unaware the --debug.accept-malformed-index flag exists; after upgrading Thanos which started validating this condition.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• block with not healthy index found
• iterate series
• repaired block is invalid
• gather index issues for block
• blocks with out-of-order chunks are dropped from compaction

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4859c981833df4be. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1246

					return e
				}, opentracing.Tags{"block.id": meta.ULID}); err != nil {
					return errors.Wrapf(err, "gather index issues for block %s", bdir)
				}

				if err := stats.CriticalErr(); err != nil {
					return halt(errors.Wrapf(err, "block with not healthy index found %s; Compaction level %v; Labels: %v", bdir, meta.Compaction.Level, meta.Thanos.Labels))
				}

				if err := stats.OutOfOrderChunksErr(); err != nil {
					return outOfOrderChunkError(errors.Wrapf(err, "blocks with out-of-order chunks are dropped from compaction:  %s", bdir), meta.ULID)
				}

				if err := stats.Issue347OutsideChunksErr(); err != nil {
					return issue347Error(errors.Wrapf(err, "invalid, but reparable block %s", bdir), meta.ULID)
				}

				if err := stats.OutOfOrderLabelsErr(); !cg.acceptMalformedIndex && err != nil {
					return errors.Wrapf(err,
						"block id %s, try running with --debug.accept-malformed-index", meta.ULID)
				}
				level.Debug(cg.logger).Log("msg", "verified block", "block", meta.ULID.String(), "duration", time.Since(start), "duration_ms", time.Since(start).Milliseconds())
				return nil
			})
		}(errCtx, m)

		toCompactDirs = append(toCompactDirs, bdir)
	}
	sourceBlockStr := fmt.Sprintf("%v", toCompactDirs)

	if err := g.Wait(); err != nil {
		return false, nil, err
	}

	level.Info(cg.logger).Log("msg", "downloaded and verified blocks; compacting blocks", "duration", time.Since(begin), "duration_ms", time.Since(begin).Milliseconds(), "plan", sourceBlockStr)

	begin = time.Now()

View on GitHub (pinned to 35b8b99117)