What causes "sync before second pass of downsampling" in thanos-io/thanos?

Check the wrapped cause; treat context canceled as normal shutdown.

sync before second pass of downsampling

Error message

sync before second pass of downsampling

What it means

The second pass re-fetches metadata (metas, _, err = metaFetcher.Fetch(ctx)) so newly created 5m-resolution blocks can be downsampled to 1h. A Fetch error here is wrapped as "sync before second pass of downsampling". Same failure modes as the first sync: object-store access issues, corrupt metadata, or canceled context.

Solutions

Check the wrapped cause; treat context canceled as normal shutdown.
Verify object-store connectivity and credentials; retry — the loop repeats automatically.
Inspect the ULID reported in the error and validate/repair its meta.json.
Ensure only one downsampler operates on the bucket to avoid racing metadata writes.

Defensive patterns

Strategy: retry

Validate before calling

// Go: check object store reachability between passes
_, err := bkt.Attributes(ctx, "")
return err

Try / catch

// Go
if err := metaFetcher.Fetch(ctx); err != nil {
    if ctx.Err() != nil { return nil } // shutdown, not a real failure
    log.Printf("second-pass sync failed, retrying: %+v", err)
}

Prevention

Use STS/refreshable credentials sized for multi-hour downsample jobs.
Ensure bucket listing consistency; avoid aggressive negative-caching proxies in front of S3.
Handle SIGTERM so the job drains instead of canceling mid-fetch.

When it happens

Trigger: metaFetcher.Fetch fails between the two downsampleBucket passes — network blip to the object store, expired credentials, invalid meta.json of a block just uploaded, or shutdown canceling the context.

Common situations: Bucket eventually-consistent listing missing/including fresh blocks; transient S3 5xx/SlowDown; operator restart during the long second phase; blocks written by another instance concurrently.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/214ae5011ec013d3. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:146

				level.Info(logger).Log("msg", "start first pass of downsampling")
				metas, _, err := metaFetcher.Fetch(ctx)
				if err != nil {
					return errors.Wrap(err, "sync before first pass of downsampling")
				}

				for _, meta := range metas {
					resolutionLabel := meta.Thanos.ResolutionString()
					metrics.downsamples.WithLabelValues(resolutionLabel)
					metrics.downsampleFailures.WithLabelValues(resolutionLabel)
				}
				if err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {
					return errors.Wrap(err, "downsampling failed")
				}

				level.Info(logger).Log("msg", "start second pass of downsampling")
				metas, _, err = metaFetcher.Fetch(ctx)
				if err != nil {
					return errors.Wrap(err, "sync before second pass of downsampling")
				}
				if err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {
					return errors.Wrap(err, "downsampling failed")
				}
				return nil
			})
		}, func(error) {
			cancel()
		})
	}

	srv := httpserver.New(logger, reg, comp, httpProbe,
		httpserver.WithListen(httpBindAddr),
		httpserver.WithGracePeriod(httpGracePeriod),
		httpserver.WithTLSConfig(httpTLSConfig),
	)

	g.Add(func() error {

View on GitHub (pinned to 35b8b99117)