{"record":{"id":"2c601775a276e108","repo":"thanos-io/thanos","slug":"downsampling-failed","errorCode":null,"errorMessage":"downsampling failed","messagePattern":"downsampling failed","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/downsample.go","lineNumber":140,"sourceCode":"\n\t\tg.Add(func() error {\n\t\t\tdefer runutil.CloseWithLogOnErr(logger, insBkt, \"bucket client\")\n\t\t\tstatusProber.Ready()\n\n\t\t\treturn runutil.Repeat(waitInterval, ctx.Done(), func() error {\n\t\t\t\tlevel.Info(logger).Log(\"msg\", \"start first pass of downsampling\")\n\t\t\t\tmetas, _, err := metaFetcher.Fetch(ctx)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"sync before first pass of downsampling\")\n\t\t\t\t}\n\n\t\t\t\tfor _, meta := range metas {\n\t\t\t\t\tresolutionLabel := meta.Thanos.ResolutionString()\n\t\t\t\t\tmetrics.downsamples.WithLabelValues(resolutionLabel)\n\t\t\t\t\tmetrics.downsampleFailures.WithLabelValues(resolutionLabel)\n\t\t\t\t}\n\t\t\t\tif err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"downsampling failed\")\n\t\t\t\t}\n\n\t\t\t\tlevel.Info(logger).Log(\"msg\", \"start second pass of downsampling\")\n\t\t\t\tmetas, _, err = metaFetcher.Fetch(ctx)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"sync before second pass of downsampling\")\n\t\t\t\t}\n\t\t\t\tif err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"downsampling failed\")\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t})\n\t\t}, func(error) {\n\t\t\tcancel()\n\t\t})\n\t}\n\n\tsrv := httpserver.New(logger, reg, comp, httpProbe,","sourceCodeStart":122,"sourceCodeEnd":158,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/downsample.go#L122-L158","documentation":"After the first metadata sync, RunDownsample calls downsampleBucket to actually produce downsampled blocks. Any failure inside that pipeline (loading/compacting blocks, writing new blocks, hash verification) is wrapped as \"downsampling failed\" and returned to the Repeat loop for retry.","triggerScenarios":"downsampleBucket returns an error: block data unreadable in the bucket, insufficient disk space in --data-dir, concurrency limits exceeded, or checksum/hashfunc mismatch while downloading block files.","commonSituations":"Full data-dir disk on the downsampler node; corrupted block in object storage; object-store timeouts on large segment downloads; downsample concurrency too high causing OOM or fd exhaustion.","solutions":["Read the wrapped root cause: identify the specific block/ULID and failure (download, compact, upload).","Free disk space or enlarge the volume behind --data-dir; clean stale partial downsampling dirs.","Re-run; Repeat retries periodically, and healthy blocks are skipped on subsequent passes.","Lower --downsample.concurrency and --block-files-concurrency if resource exhaustion caused the failure."],"exampleFix":"// before\nthanos downsample --data-dir=/small-vol --downsample.concurrency=10 ...\n// after\nthanos downsample --data-dir=/big-vol --downsample.concurrency=2 ...","handlingStrategy":"retry","validationCode":"// Go: ensure data-dir has headroom before starting\nif st, err := os.Stat(dataDir); err != nil || !st.IsDir() {\n    return fmt.Errorf(\"data dir missing: %s\", dataDir)\n}","typeGuard":null,"tryCatchPattern":"// Go\nif err := downsampleBucket(...); err != nil {\n    log.Printf(\"downsampling failed: %+v\", err) // root cause shows the block and op\n}","preventionTips":["Provision --data-dir with at least a few GiB of free space per concurrent downsample.","Set --downsample.concurrency conservatively (1-4) on small nodes.","Run exactly one downsampler per bucket to avoid concurrent block writes.","Alert on repeated 'downsampling failed' wrap occurrences rather than a single one."],"tags":["downsample","blocks","object-storage","disk"],"backgroundTag":"database-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}