thanos-io/thanos · error

downsampling failed

Error message

downsampling failed

What it means

After the first metadata sync, RunDownsample calls downsampleBucket to actually produce downsampled blocks. Any failure inside that pipeline (loading/compacting blocks, writing new blocks, hash verification) is wrapped as "downsampling failed" and returned to the Repeat loop for retry.

Solutions

1. Read the wrapped root cause: identify the specific block/ULID and failure (download, compact, upload).
2. Free disk space or enlarge the volume behind --data-dir; clean stale partial downsampling dirs.
3. Re-run; Repeat retries periodically, and healthy blocks are skipped on subsequent passes.
4. Lower --downsample.concurrency and --block-files-concurrency if resource exhaustion caused the failure.

Example fix

// before
thanos downsample --data-dir=/small-vol --downsample.concurrency=10 ...
// after
thanos downsample --data-dir=/big-vol --downsample.concurrency=2 ...

// Go: ensure data-dir has headroom before starting
if st, err := os.Stat(dataDir); err != nil || !st.IsDir() {
    return fmt.Errorf("data dir missing: %s", dataDir)
}

// Go
if err := downsampleBucket(...); err != nil {
    log.Printf("downsampling failed: %+v", err) // root cause shows the block and op
}

When it happens

Trigger: downsampleBucket returns an error: block data unreadable in the bucket, insufficient disk space in --data-dir, concurrency limits exceeded, or checksum/hashfunc mismatch while downloading block files.

Common situations: Full data-dir disk on the downsampler node; corrupted block in object storage; object-store timeouts on large segment downloads; downsample concurrency too high causing OOM or fd exhaustion.

Related errors

• create meta fetcher
• sync before first pass of downsampling
• sync before second pass of downsampling
• meta.json unmarshal
• unexpected meta file

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/2c601775a276e108. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:140

		g.Add(func() error {
			defer runutil.CloseWithLogOnErr(logger, insBkt, "bucket client")
			statusProber.Ready()

			return runutil.Repeat(waitInterval, ctx.Done(), func() error {
				level.Info(logger).Log("msg", "start first pass of downsampling")
				metas, _, err := metaFetcher.Fetch(ctx)
				if err != nil {
					return errors.Wrap(err, "sync before first pass of downsampling")
				}

				for _, meta := range metas {
					resolutionLabel := meta.Thanos.ResolutionString()
					metrics.downsamples.WithLabelValues(resolutionLabel)
					metrics.downsampleFailures.WithLabelValues(resolutionLabel)
				}
				if err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {
					return errors.Wrap(err, "downsampling failed")
				}

				level.Info(logger).Log("msg", "start second pass of downsampling")
				metas, _, err = metaFetcher.Fetch(ctx)
				if err != nil {
					return errors.Wrap(err, "sync before second pass of downsampling")
				}
				if err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {
					return errors.Wrap(err, "downsampling failed")
				}
				return nil
			})
		}, func(error) {
			cancel()
		})
	}

	srv := httpserver.New(logger, reg, comp, httpProbe,

View on GitHub (pinned to 35b8b99117)