{"record":{"id":"e9e4d570bbb1cff0","repo":"thanos-io/thanos","slug":"second-pass-of-downsampling-failed","errorCode":null,"errorMessage":"second pass of downsampling failed","messagePattern":"second pass of downsampling failed","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/compact.go","lineNumber":509,"sourceCode":"\t\t\tfilteredMetas = sy.Metas()\n\t\t\tnoDownsampleBlocks = noDownsampleMarkerFilter.NoDownsampleMarkedBlocks()\n\t\t\tfor ul := range noDownsampleBlocks {\n\t\t\t\tdelete(filteredMetas, ul)\n\t\t\t}\n\n\t\t\tif err := downsampleBucket(\n\t\t\t\tctx,\n\t\t\t\tlogger,\n\t\t\t\tdownsampleMetrics,\n\t\t\t\tinsBkt,\n\t\t\t\tfilteredMetas,\n\t\t\t\tdownsamplingDir,\n\t\t\t\tconf.downsampleConcurrency,\n\t\t\t\tconf.blockFilesConcurrency,\n\t\t\t\tmetadata.HashFunc(conf.hashFunc),\n\t\t\t\tconf.acceptMalformedIndex,\n\t\t\t); err != nil {\n\t\t\t\treturn errors.Wrap(err, \"second pass of downsampling failed\")\n\t\t\t}\n\n\t\t\tlevel.Info(logger).Log(\"msg\", \"downsampling iterations done\")\n\t\t} else {\n\t\t\tlevel.Info(logger).Log(\"msg\", \"downsampling was explicitly disabled\")\n\t\t}\n\n\t\t// TODO(bwplotka): Find a way to avoid syncing if no op was done.\n\t\tif err := sy.SyncMetas(ctx); err != nil {\n\t\t\treturn errors.Wrap(err, \"sync before retention\")\n\t\t}\n\n\t\tif err := compact.ApplyRetentionPolicyByResolution(ctx, logger, insBkt, sy.Metas(), retentionByResolution, compactMetrics.blocksMarked.WithLabelValues(metadata.DeletionMarkFilename, \"\")); err != nil {\n\t\t\treturn errors.Wrap(err, \"retention failed\")\n\t\t}\n\n\t\treturn cleanPartialMarked()\n\t}","sourceCodeStart":491,"sourceCodeEnd":527,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/compact.go#L491-L527","documentation":"Wrapper error from the second downsampling pass in runCompact (cmd/thanos/compact.go:509) around the downsampling bucket operation. Same failure mode as the first pass but for 1h-resolution work triggered by 5m downsamples created earlier.","triggerScenarios":"Downsampling job failure in pass two: corrupt block index/chunks, hash function mismatch, IO failure writing downsampled blocks, context cancellation from shutdown or concurrency limits.","commonSituations":"Disk exhaustion on --data-dir after the first pass generated large intermediates, malformed blocks discovered only in second pass, node shutdown (SIGTERM) mid-work.","solutions":["Read the wrapped inner error for the failing block/stage","Check disk free space on the data/downsampling directory","Verify --hash-func matches block upload hash functions","Retry after transient issues; downsampling resumes where it left off","Gracefully shut down rather than killing, to avoid partial local artifacts"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := downsampleBucket(...); err != nil {\n\tlevel.Error(logger).Log(\"msg\", \"second pass downsampling failed\", \"err\", err)\n\treturn errors.Wrap(err, \"second pass of downsampling failed\")\n}","preventionTips":["Guarantee disk headroom after pass one (pass two writes 1h blocks)","Keep hash function config stable across upgrades","Graceful shutdown on SIGTERM to avoid partial artifacts"],"tags":["thanos","downsampling","disk"],"backgroundTag":"file-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}