{"record":{"id":"8cfb53365722bc92","repo":"thanos-io/thanos","slug":"gather-index-issues-for-block-s","errorCode":null,"errorMessage":"gather index issues for block %s","messagePattern":"gather index issues for block (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/compact/compact.go","lineNumber":1230,"sourceCode":"\t\tbdir := filepath.Join(dir, m.ULID.String())\n\t\tfunc(ctx context.Context, meta *metadata.Meta) {\n\t\t\tg.Go(func() error {\n\t\t\t\tstart := time.Now()\n\t\t\t\tif err := tracing.DoInSpanWithErr(ctx, \"compaction_block_download\", func(ctx context.Context) error {\n\t\t\t\t\treturn block.Download(ctx, cg.logger, cg.bkt, meta.ULID, bdir, objstore.WithFetchConcurrency(cg.blockFilesConcurrency))\n\t\t\t\t}, opentracing.Tags{\"block.id\": meta.ULID}); err != nil {\n\t\t\t\t\treturn retry(errors.Wrapf(err, \"download block %s\", meta.ULID))\n\t\t\t\t}\n\t\t\t\tlevel.Debug(cg.logger).Log(\"msg\", \"downloaded block\", \"block\", meta.ULID.String(), \"duration\", time.Since(start), \"duration_ms\", time.Since(start).Milliseconds())\n\n\t\t\t\tstart = time.Now()\n\t\t\t\t// Ensure all input blocks are valid.\n\t\t\t\tvar stats block.HealthStats\n\t\t\t\tif err := tracing.DoInSpanWithErr(ctx, \"compaction_block_health_stats\", func(ctx context.Context) (e error) {\n\t\t\t\t\tstats, e = block.GatherIndexHealthStats(ctx, cg.logger, filepath.Join(bdir, block.IndexFilename), meta.MinTime, meta.MaxTime)\n\t\t\t\t\treturn e\n\t\t\t\t}, opentracing.Tags{\"block.id\": meta.ULID}); err != nil {\n\t\t\t\t\treturn errors.Wrapf(err, \"gather index issues for block %s\", bdir)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.CriticalErr(); err != nil {\n\t\t\t\t\treturn halt(errors.Wrapf(err, \"block with not healthy index found %s; Compaction level %v; Labels: %v\", bdir, meta.Compaction.Level, meta.Thanos.Labels))\n\t\t\t\t}\n\n\t\t\t\tif err := stats.OutOfOrderChunksErr(); err != nil {\n\t\t\t\t\treturn outOfOrderChunkError(errors.Wrapf(err, \"blocks with out-of-order chunks are dropped from compaction:  %s\", bdir), meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.Issue347OutsideChunksErr(); err != nil {\n\t\t\t\t\treturn issue347Error(errors.Wrapf(err, \"invalid, but reparable block %s\", bdir), meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.OutOfOrderLabelsErr(); !cg.acceptMalformedIndex && err != nil {\n\t\t\t\t\treturn errors.Wrapf(err,\n\t\t\t\t\t\t\"block id %s, try running with --debug.accept-malformed-index\", meta.ULID)\n\t\t\t\t}","sourceCodeStart":1212,"sourceCodeEnd":1248,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1212-L1248","documentation":"After downloading, Thanos validates each block's index with block.GatherIndexHealthStats. If scanning the index file for structural issues fails (e.g. file unreadable, truncated, or the health-stat walker errors), the error is wrapped as 'gather index issues for block <dir>'. This happens before any data is modified.","triggerScenarios":"block.GatherIndexHealthStats on filepath.Join(bdir, block.IndexFilename) returns an error: truncated/corrupt index file on disk, I/O failure reading the downloaded index, or internal validation walk failing.","commonSituations":"Incomplete block downloads (index.toml present but index truncated); disk corruption; version mismatch where index format is not understood; out-of-disk conditions during concurrent fetches.","solutions":["Delete the block's local directory under --data-dir and let compaction re-download it","Verify the block in the bucket (thanos tools bucket verify) — if corrupt remotely, run bucket repair or delete the bad block","Check local disk health and free space on the compactor node","Upgrade Thanos if the index was produced by a newer Thanos/Prometheus with format changes"],"exampleFix":"// before\n// corrupt block dir cached locally\n# rm -rf /data/compact/0ULID.../\n// after\n// re-run compactor; block is re-downloaded and re-validated","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"gather index issues\") {\n    // purge local cache dir for that block and retry once\n    os.RemoveAll(blockLocalDir)\n}","preventionTips":["Use healthy disks (watch SMART/dmesg) on compactor nodes","Don't kill the compactor mid-download; allow graceful shutdown","Verify blocks in the bucket periodically with 'thanos tools bucket verify'"],"tags":["compaction","index","validation","thanos"],"backgroundTag":"file-read-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}