{"record":{"id":"83c624f59697d1ff","repo":"thanos-io/thanos","slug":"downsampling-to-5-min","errorCode":null,"errorMessage":"downsampling to 5 min","messagePattern":"downsampling to 5 min","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/downsample.go","lineNumber":267,"sourceCode":"\t\terrCh                   = make(chan error, downsampleConcurrency)\n\t\tworkerCtx, workerCancel = context.WithCancel(ctx)\n\t)\n\n\tdefer workerCancel()\n\n\tlevel.Debug(logger).Log(\"msg\", \"downsampling bucket\", \"concurrency\", downsampleConcurrency)\n\tfor range downsampleConcurrency {\n\t\twg.Go(func() {\n\t\t\tfor m := range metaCh {\n\t\t\t\tresolution := downsample.ResLevel1\n\t\t\t\terrMsg := \"downsampling to 5 min\"\n\t\t\t\tif m.Thanos.Downsample.Resolution == downsample.ResLevel1 {\n\t\t\t\t\tresolution = downsample.ResLevel2\n\t\t\t\t\terrMsg = \"downsampling to 60 min\"\n\t\t\t\t}\n\t\t\t\tif err := processDownsampling(workerCtx, logger, bkt, m, dir, resolution, hashFunc, metrics, acceptMalformedIndex, blockFilesConcurrency); err != nil {\n\t\t\t\t\tmetrics.downsampleFailures.WithLabelValues(m.Thanos.ResolutionString()).Inc()\n\t\t\t\t\terrCh <- errors.Wrap(err, errMsg)\n\n\t\t\t\t}\n\t\t\t\tmetrics.downsamples.WithLabelValues(m.Thanos.ResolutionString()).Inc()\n\t\t\t}\n\t\t})\n\t}\n\n\t// Workers scheduled, distribute blocks.\nmetaSendLoop:\n\tfor _, mk := range metasULIDS {\n\t\tm := metas[mk]\n\n\t\tswitch m.Thanos.Downsample.Resolution {\n\t\tcase downsample.ResLevel2:\n\t\t\tcontinue\n\n\t\tcase downsample.ResLevel0:\n\t\t\tmissing := false","sourceCodeStart":249,"sourceCodeEnd":285,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/downsample.go#L249-L285","documentation":"When a 5m-resolution block fails its second-stage downsampling, downsampleBucket pushes errors.Wrap(err, \"downsampling to 60 min\") onto errCh; \"downsampling to 5 min\" is the analogous wrapper for the first stage. It is a context wrapper around processDownsampling failures, recording which resolution stage failed, and increments the downsampleFailures metric.","triggerScenarios":"Any failure inside processDownsampling (download, index verification, open, downsample, upload) for a block being downsampled to ResLevel1 (5 min); the raw error is wrapped with errMsg \"downsampling to 5 min\" and sent to errCh.","commonSituations":"Transient object-store errors during download/upload (network blips, throttling); corrupt source block index; OOM or disk full in the data dir during downsampling; bucket with malformed blocks.","solutions":["Inspect the wrapped cause beneath \"downsampling to 5 min\" — retry errors (compact.NewRetryError) will be retried on the next compactor/downsampler cycle automatically.","Fix the underlying object store access issue (credentials, throttling, network) indicated by the inner error.","Delete a persistently corrupt source block after confirming it is unrecoverable, so the pipeline stops retrying it.","Increase downsample concurrency limits or data-dir disk space if failures stem from resource exhaustion."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"var rerr *compact.RetryError\nif errors.As(err, &rerr) {\n    // transient: rely on automatic retry next cycle\n    log.Printf(\"retryable downsampling failure (%s): %v\", stage, err)\n} else {\n    // halt-worthy: page/inspect\n    log.Printf(\"fatal downsampling failure: %v\", err)\n}","preventionTips":["Monitor the thanos_downsample_failures_total metric per resolution.","Harden object store connectivity (retries, timeouts, credential rotation) in the downsampler deployment.","Keep --data-dir sized generously to avoid failure cascades from disk pressure."],"tags":["downsampling","object-storage","pipeline"],"backgroundTag":"api-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}