What causes "first pass of downsampling failed" in thanos-io/thanos?

Read the wrapped inner error to identify the failing block and stage

first pass of downsampling failed

Error message

first pass of downsampling failed

What it means

Wrapper error from the first downsampling pass in runCompact (cmd/thanos/compact.go:481) around downsample.DownsampleBucket (or equivalent). An individual block downsampling job failed — the specific block, hash function, or concurrency worker error is wrapped inside.

Solutions

Read the wrapped inner error to identify the failing block and stage
Free disk space or fix volume permissions on the downsampling/data directory
Re-download or delete the corrupt source block (after verifying) so compaction can regenerate it
Match --hash-func with the hash function used when blocks were uploaded
Set --accept-malformed-index only if you understand the malformed index risk

Defensive patterns

Strategy: try-catch

Try / catch

if err := downsampleBucket(...); err != nil {
	level.Error(logger).Log("msg", "first pass downsampling failed", "err", err)
	return errors.Wrap(err, "first pass of downsampling failed")
}

Prevention

Monitor free disk space on --data-dir (downsampling needs headroom for intermediates)
Keep --hash-func consistent with upload-time hash functions
Avoid killing the process mid-downsample; use graceful shutdown

When it happens

Trigger: Downsampling a specific block fails: corrupt or unreadable index/chunks files in the local downsampling dir, unsupported hash function, IO errors writing the downsampled block, concurrent workers exhausted context.

Common situations: Disk full on --data-dir volume, malformed blocks flagged by --accept-malformed-index not set, hash function mismatch (e.g. SHA256 vs XXHash) with objects in the bucket.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/fb49a4686d6c8575. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:481

			for _, meta := range filteredMetas {
				resolutionLabel := meta.Thanos.ResolutionString()
				downsampleMetrics.downsamples.WithLabelValues(resolutionLabel)
				downsampleMetrics.downsampleFailures.WithLabelValues(resolutionLabel)
			}

			if err := downsampleBucket(
				ctx,
				logger,
				downsampleMetrics,
				insBkt,
				filteredMetas,
				downsamplingDir,
				conf.downsampleConcurrency,
				conf.blockFilesConcurrency,
				metadata.HashFunc(conf.hashFunc),
				conf.acceptMalformedIndex,
			); err != nil {
				return errors.Wrap(err, "first pass of downsampling failed")
			}

			level.Info(logger).Log("msg", "start second pass of downsampling")
			if err := sy.SyncMetas(ctx); err != nil {
				return errors.Wrap(err, "sync before second pass of downsampling")
			}

			// Regenerate the filtered list of blocks after the sync,
			// to include the blocks created by the first pass.
			filteredMetas = sy.Metas()
			noDownsampleBlocks = noDownsampleMarkerFilter.NoDownsampleMarkedBlocks()
			for ul := range noDownsampleBlocks {
				delete(filteredMetas, ul)
			}

			if err := downsampleBucket(
				ctx,
				logger,

View on GitHub (pinned to 35b8b99117)