What causes "download block %s" in thanos-io/thanos?

Let the retry mechanism work — RetryError failures are retried on subsequent cycles; just confirm the downsampler stays up.

thanos-io/thanos · error · RetryError

download block

Error message

download block %s

What it means

processDownsampling wraps block.Download failures with compact.NewRetryError(errors.Wrapf(err, "download block %s", m.ULID)). Downloading the TSDB block from the object store failed. Because it is a RetryError, the compactor treats it as transient and retries the block on a later cycle instead of halting.

Solutions

Let the retry mechanism work — RetryError failures are retried on subsequent cycles; just confirm the downsampler stays up.
Check object store credentials and network reachability from the downsampler pod to the bucket endpoint.
Ensure only one downsampler operates against the same bucket and that block deletion/gc is not racing downloads.
Reduce --block-files-concurrency if the store throttles concurrent GETs; inspect inner error for 404 vs 403 vs 5xx.

Defensive patterns

Strategy: retry

Validate before calling

// before running downsampler, check reachability
_, err := bkt.Iter(ctx, "", func(string) error { return nil })
if err != nil { log.Fatalf("object store unreachable: %v", err) }

Try / catch

if compact.IsRetryable(err) { /* schedule retry with backoff */ } else { /* alert: bucket auth/config problem */ }

Prevention

Use exponential backoff and retry-friendly object store client settings.
Alert on object-store 404s to detect racing deletions between compactors/downsamplers.
Verify credentials and bucket policy in staging with the same flags used in production.

When it happens

Trigger: block.Download for block m.ULID fails: object not found (block deleted concurrently by compactor), network timeouts, S3/GCS 5xx or throttling, wrong credentials, or fetch-concurrency issues.

Common situations: Two downsamplers/compactors racing so one deletes a block the other is downloading; bucket rate limits under high block download concurrency; expired/misconfigured object store credentials; proxy or VPC endpoint misconfiguration.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/9f73dae189a04651. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:362

func processDownsampling(
	ctx context.Context,
	logger log.Logger,
	bkt objstore.Bucket,
	m *metadata.Meta,
	dir string,
	resolution int64,
	hashFunc metadata.HashFunc,
	metrics *DownsampleMetrics,
	acceptMalformedIndex bool,
	blockFilesConcurrency int,
) error {
	begin := time.Now()
	bdir := filepath.Join(dir, m.ULID.String())

	err := block.Download(ctx, logger, bkt, m.ULID, bdir, objstore.WithFetchConcurrency(blockFilesConcurrency))
	if err != nil {
		return compact.NewRetryError(errors.Wrapf(err, "download block %s", m.ULID))
	}
	level.Info(logger).Log("msg", "downloaded block", "id", m.ULID, "duration", time.Since(begin), "duration_ms", time.Since(begin).Milliseconds())

	if err := block.VerifyIndex(ctx, logger, filepath.Join(bdir, block.IndexFilename), m.MinTime, m.MaxTime); err != nil && !acceptMalformedIndex {
		return errors.Wrap(err, "input block index not valid")
	}

	begin = time.Now()

	var pool chunkenc.Pool
	if m.Thanos.Downsample.Resolution == 0 {
		pool = chunkenc.NewPool()
	} else {
		pool = downsample.NewPool()
	}

	b, err := tsdb.OpenBlock(logutil.GoKitLogToSlog(logger), bdir, pool, nil)
	if err != nil {

View on GitHub (pinned to 35b8b99117)