{"record":{"id":"f6c3fdb62211e0c0","repo":"thanos-io/thanos","slug":"compact-blocks-v","errorCode":null,"errorMessage":"compact blocks %v","messagePattern":"compact blocks (.+?)","errorType":"exception","errorClass":"halt","httpStatus":null,"severity":"critical","filePath":"pkg/compact/compact.go","lineNumber":1274,"sourceCode":"\tsourceBlockStr := fmt.Sprintf(\"%v\", toCompactDirs)\n\n\tif err := g.Wait(); err != nil {\n\t\treturn false, nil, err\n\t}\n\n\tlevel.Info(cg.logger).Log(\"msg\", \"downloaded and verified blocks; compacting blocks\", \"duration\", time.Since(begin), \"duration_ms\", time.Since(begin).Milliseconds(), \"plan\", sourceBlockStr)\n\n\tbegin = time.Now()\n\tvar compIDs []ulid.ULID\n\tif err := tracing.DoInSpanWithErr(ctx, \"compaction\", func(ctx context.Context) (e error) {\n\t\tpopulateBlockFunc, e := compactionLifecycleCallback.GetBlockPopulator(ctx, cg.logger, cg)\n\t\tif e != nil {\n\t\t\treturn e\n\t\t}\n\t\tcompIDs, e = comp.CompactWithBlockPopulator(dir, toCompactDirs, nil, populateBlockFunc)\n\t\treturn e\n\t}); err != nil {\n\t\treturn false, nil, halt(errors.Wrapf(err, \"compact blocks %v\", toCompactDirs))\n\t}\n\tif len(compIDs) == 0 {\n\t\t// No compacted blocks means all compacted blocks are of no sample.\n\t\tlevel.Info(cg.logger).Log(\"msg\", \"no compacted blocks, deleting source blocks\", \"blocks\", sourceBlockStr)\n\t\tfor _, meta := range toCompact {\n\t\t\tif meta.Stats.NumSamples == 0 {\n\t\t\t\tif err := cg.deleteBlock(meta.ULID, filepath.Join(dir, meta.ULID.String()), blockDeletableChecker); err != nil {\n\t\t\t\t\tlevel.Warn(cg.logger).Log(\"msg\", \"failed to mark for deletion an empty block found during compaction\", \"block\", meta.ULID)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t// Even though no compacted blocks, there may be more work to do.\n\t\treturn true, nil, nil\n\t}\n\tcg.compactions.Inc()\n\tif overlappingBlocks {\n\t\tcg.verticalCompactions.Inc()\n\t}","sourceCodeStart":1256,"sourceCodeEnd":1292,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1256-L1292","documentation":"The actual merge of source blocks via comp.CompactWithBlockPopulator (TSDB compaction) failed. Thanos wraps this in a halt error including the source block directories, because a failed compaction may have left partial state that requires operator attention before retrying safely.","triggerScenarios":"comp.CompactWithBlockPopulator(dir, toCompactDirs, nil, populateBlockFunc) returns error: merged-OOO histogram issues, disk full, corrupted local block data, panics inside TSDB merge, or index writing failures during merge.","commonSituations":"Insufficient disk space in --data-dir for the merged output; very large compactions exhausting memory (mmap'd index) or hitting TSDB merge bugs; corrupted downloaded data that passed health checks; OOM-killed compactor leaving partial temp dirs.","solutions":["Check disk space and memory on the compactor; enlarge --data-dir and re-run","Delete the partially written compaction output/ULID directories under the group's data dir and let compaction retry","Inspect the wrapped TSDB error; upgrade Thanos/Prometheus if it matches a known merge bug (e.g. histogram or symbol merge panics)","Reduce compaction concurrency (--compact.concurrency) and compaction level sizes if OOM/timeout is the cause"],"exampleFix":"// before\n// halt: disk full during merge\n// after\n# df -h /data && rm -rf /data/compact/<partial-ULID>\n# thanos compact --data-dir /data --objstore.bucket prom","handlingStrategy":"retry","validationCode":"// Ensure enough headroom before a big compaction:\ndf -h /data  # keep > 2x largest level block size free\n","typeGuard":null,"tryCatchPattern":"if err := compactor.Compact(ctx); err != nil {\n    if strings.Contains(err.Error(), \"compact blocks\") {\n        // halt error: clean partial output dirs, free disk, then re-run\n        cleanupPartialCompactions(dataDir)\n    }\n}","preventionTips":["Size --data-dir for worst-case compaction output plus source blocks","Set memory requests/limits generously; TSDB merging is memory hungry","Keep Thanos/Prometheus current to get TSDB merge bug fixes","Clean partial output dirs after OOM kills before re-running"],"tags":["compaction","tsdb","merge","thanos"],"backgroundTag":"database-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}