thanos-io/thanos · error

downsampling to 5 min

Error message

downsampling to 5 min

What it means

When a 5m-resolution block fails its second-stage downsampling, downsampleBucket pushes errors.Wrap(err, "downsampling to 60 min") onto errCh; "downsampling to 5 min" is the analogous wrapper for the first stage. It is a context wrapper around processDownsampling failures, recording which resolution stage failed, and increments the downsampleFailures metric.

Solutions

1. Inspect the wrapped cause beneath "downsampling to 5 min" — retry errors (compact.NewRetryError) will be retried on the next compactor/downsampler cycle automatically.
2. Fix the underlying object store access issue (credentials, throttling, network) indicated by the inner error.
3. Delete a persistently corrupt source block after confirming it is unrecoverable, so the pipeline stops retrying it.
4. Increase downsample concurrency limits or data-dir disk space if failures stem from resource exhaustion.

var rerr *compact.RetryError
if errors.As(err, &rerr) {
    // transient: rely on automatic retry next cycle
    log.Printf("retryable downsampling failure (%s): %v", stage, err)
} else {
    // halt-worthy: page/inspect
    log.Printf("fatal downsampling failure: %v", err)
}

When it happens

Trigger: Any failure inside processDownsampling (download, index verification, open, downsample, upload) for a block being downsampled to ResLevel1 (5 min); the raw error is wrapped with errMsg "downsampling to 5 min" and sent to errCh.

Common situations: Transient object-store errors during download/upload (network blips, throttling); corrupt source block index; OOM or disk full in the data dir during downsampling; bucket with malformed blocks.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• sync before first pass of downsampling
• sync before second pass of downsampling
• create meta fetcher
• create syncer
• raw resolution must be higher than the minimum block size…

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/83c624f59697d1ff. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:267

		errCh                   = make(chan error, downsampleConcurrency)
		workerCtx, workerCancel = context.WithCancel(ctx)
	)

	defer workerCancel()

	level.Debug(logger).Log("msg", "downsampling bucket", "concurrency", downsampleConcurrency)
	for range downsampleConcurrency {
		wg.Go(func() {
			for m := range metaCh {
				resolution := downsample.ResLevel1
				errMsg := "downsampling to 5 min"
				if m.Thanos.Downsample.Resolution == downsample.ResLevel1 {
					resolution = downsample.ResLevel2
					errMsg = "downsampling to 60 min"
				}
				if err := processDownsampling(workerCtx, logger, bkt, m, dir, resolution, hashFunc, metrics, acceptMalformedIndex, blockFilesConcurrency); err != nil {
					metrics.downsampleFailures.WithLabelValues(m.Thanos.ResolutionString()).Inc()
					errCh <- errors.Wrap(err, errMsg)

				}
				metrics.downsamples.WithLabelValues(m.Thanos.ResolutionString()).Inc()
			}
		})
	}

	// Workers scheduled, distribute blocks.
metaSendLoop:
	for _, mk := range metasULIDS {
		m := metas[mk]

		switch m.Thanos.Downsample.Resolution {
		case downsample.ResLevel2:
			continue

		case downsample.ResLevel0:
			missing := false

View on GitHub (pinned to 35b8b99117)