What causes "second pass of downsampling failed" in thanos-io/thanos?

Read the wrapped inner error for the failing block/stage

second pass of downsampling failed

Error message

second pass of downsampling failed

What it means

Wrapper error from the second downsampling pass in runCompact (cmd/thanos/compact.go:509) around the downsampling bucket operation. Same failure mode as the first pass but for 1h-resolution work triggered by 5m downsamples created earlier.

Solutions

Read the wrapped inner error for the failing block/stage
Check disk free space on the data/downsampling directory
Verify --hash-func matches block upload hash functions
Retry after transient issues; downsampling resumes where it left off
Gracefully shut down rather than killing, to avoid partial local artifacts

Defensive patterns

Strategy: try-catch

Try / catch

if err := downsampleBucket(...); err != nil {
	level.Error(logger).Log("msg", "second pass downsampling failed", "err", err)
	return errors.Wrap(err, "second pass of downsampling failed")
}

Prevention

Guarantee disk headroom after pass one (pass two writes 1h blocks)
Keep hash function config stable across upgrades
Graceful shutdown on SIGTERM to avoid partial artifacts

When it happens

Trigger: Downsampling job failure in pass two: corrupt block index/chunks, hash function mismatch, IO failure writing downsampled blocks, context cancellation from shutdown or concurrency limits.

Common situations: Disk exhaustion on --data-dir after the first pass generated large intermediates, malformed blocks discovered only in second pass, node shutdown (SIGTERM) mid-work.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e9e4d570bbb1cff0. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:509

			filteredMetas = sy.Metas()
			noDownsampleBlocks = noDownsampleMarkerFilter.NoDownsampleMarkedBlocks()
			for ul := range noDownsampleBlocks {
				delete(filteredMetas, ul)
			}

			if err := downsampleBucket(
				ctx,
				logger,
				downsampleMetrics,
				insBkt,
				filteredMetas,
				downsamplingDir,
				conf.downsampleConcurrency,
				conf.blockFilesConcurrency,
				metadata.HashFunc(conf.hashFunc),
				conf.acceptMalformedIndex,
			); err != nil {
				return errors.Wrap(err, "second pass of downsampling failed")
			}

			level.Info(logger).Log("msg", "downsampling iterations done")
		} else {
			level.Info(logger).Log("msg", "downsampling was explicitly disabled")
		}

		// TODO(bwplotka): Find a way to avoid syncing if no op was done.
		if err := sy.SyncMetas(ctx); err != nil {
			return errors.Wrap(err, "sync before retention")
		}

		if err := compact.ApplyRetentionPolicyByResolution(ctx, logger, insBkt, sy.Metas(), retentionByResolution, compactMetrics.blocksMarked.WithLabelValues(metadata.DeletionMarkFilename, "")); err != nil {
			return errors.Wrap(err, "retention failed")
		}

		return cleanPartialMarked()
	}

View on GitHub (pinned to 35b8b99117)