{"record":{"id":"4859c981833df4be","repo":"thanos-io/thanos","slug":"block-id-s-try-running-with-debug-accept-malfo","errorCode":null,"errorMessage":"block id %s, try running with --debug.accept-malformed-index","messagePattern":"block id (.+?), try running with --debug\\.accept-malformed-index","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/compact/compact.go","lineNumber":1246,"sourceCode":"\t\t\t\t\treturn e\n\t\t\t\t}, opentracing.Tags{\"block.id\": meta.ULID}); err != nil {\n\t\t\t\t\treturn errors.Wrapf(err, \"gather index issues for block %s\", bdir)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.CriticalErr(); err != nil {\n\t\t\t\t\treturn halt(errors.Wrapf(err, \"block with not healthy index found %s; Compaction level %v; Labels: %v\", bdir, meta.Compaction.Level, meta.Thanos.Labels))\n\t\t\t\t}\n\n\t\t\t\tif err := stats.OutOfOrderChunksErr(); err != nil {\n\t\t\t\t\treturn outOfOrderChunkError(errors.Wrapf(err, \"blocks with out-of-order chunks are dropped from compaction:  %s\", bdir), meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.Issue347OutsideChunksErr(); err != nil {\n\t\t\t\t\treturn issue347Error(errors.Wrapf(err, \"invalid, but reparable block %s\", bdir), meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.OutOfOrderLabelsErr(); !cg.acceptMalformedIndex && err != nil {\n\t\t\t\t\treturn errors.Wrapf(err,\n\t\t\t\t\t\t\"block id %s, try running with --debug.accept-malformed-index\", meta.ULID)\n\t\t\t\t}\n\t\t\t\tlevel.Debug(cg.logger).Log(\"msg\", \"verified block\", \"block\", meta.ULID.String(), \"duration\", time.Since(start), \"duration_ms\", time.Since(start).Milliseconds())\n\t\t\t\treturn nil\n\t\t\t})\n\t\t}(errCtx, m)\n\n\t\ttoCompactDirs = append(toCompactDirs, bdir)\n\t}\n\tsourceBlockStr := fmt.Sprintf(\"%v\", toCompactDirs)\n\n\tif err := g.Wait(); err != nil {\n\t\treturn false, nil, err\n\t}\n\n\tlevel.Info(cg.logger).Log(\"msg\", \"downloaded and verified blocks; compacting blocks\", \"duration\", time.Since(begin), \"duration_ms\", time.Since(begin).Milliseconds(), \"plan\", sourceBlockStr)\n\n\tbegin = time.Now()","sourceCodeStart":1228,"sourceCodeEnd":1264,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1228-L1264","documentation":"The index has out-of-order labels (a known corruption class) and the compactor is not running with --debug.accept-malformed-index. Because this malformed index could produce wrong query results, compaction fails with an error advising the escape-hatch flag. Unlike issue347, this is not auto-repaired — the operator must either fix the block or explicitly accept the malformed index.","triggerScenarios":"stats.OutOfOrderLabelsErr() returns non-nil while cg.acceptMalformedIndex is false, during block verification after download.","commonSituations":"Legacy buckets with blocks written by old buggy Prometheus writers; operators unaware the --debug.accept-malformed-index flag exists; after upgrading Thanos which started validating this condition.","solutions":["Repair or delete the offending block: run 'thanos tools bucket verify --repair' to remove malformed blocks","If the data is acceptable for your use case, start compactor with --debug.accept-malformed-index to skip this check","Re-upload the block from source Prometheus so a healthy index is written","Check Thanos release notes — validation strictness may have changed with your upgrade"],"exampleFix":"// before\n# thanos compact --data-dir /data --objstore.bucket prom // fails on out-of-order labels\n// after\n# thanos compact --data-dir /data --objstore.bucket prom --debug.accept-malformed-index","handlingStrategy":"validation","validationCode":"// Scan bucket for malformed indexes before compaction runs:\nthanos tools bucket verify --repair --objstore.bucket prom","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"--debug.accept-malformed-index\") {\n    log.Error(err, \"malformed index detected; repair the block or enable the debug flag\")\n}","preventionTips":["Repair legacy blocks proactively rather than using the accept-malformed escape hatch","Document the flag's data-correctness risk before anyone uses it","Alert on this error — it indicates corrupted index data in the bucket"],"tags":["compaction","index","corruption","thanos"],"backgroundTag":"checksum-mismatch","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}