thanos-io/thanos · error

sync before first pass of downsampling

Error message

sync before first pass of downsampling

What it means

In the first downsampling loop, RunDownsample repeatedly calls metaFetcher.Fetch(ctx) to sync block metadata from the bucket. Any Fetch failure (context canceled, network/object-store error, malformed meta.json) is wrapped as "sync before first pass of downsampling" and retried after waitInterval by runutil.Repeat.

Solutions

1. Inspect the wrapped cause: context canceled means shutdown — just exit; object-store errors need credential/endpoint checks.
2. Check bucket health and permissions; retry after provider rate limits clear.
3. Find and re-upload or delete the corrupt block meta.json reported in the error chain.
4. Verify network egress/firewall allows the object-store endpoint.

// Go: preflight a cheap listing before starting the long job
if err := bkt.Iter(ctx, "", func(string) error { return nil }); err != nil {
    return err
}

// Go
cerr := errors.Cause(err)
if cerr == context.Canceled { return nil } // normal shutdown
// otherwise rely on runutil.Repeat backoff or alert on repeated failures

When it happens

Trigger: metaFetcher.Fetch returns an error because the object store is unreachable, a block's meta.json is corrupt/invalid, listing fails, or the context was canceled during shutdown.

Common situations: S3/GCS credentials expired mid-run; rate limiting (SlowDown) from the provider; partially uploaded/corrupt meta.json in the bucket; operator Ctrl-C (context canceled) surfacing as this wrap.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• create meta fetcher
• downsampling failed
• sync before second pass of downsampling
• upload file to bucket
• stat

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/78b3565e4d27b63e. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:131

	statusProber := prober.Combine(
		httpProbe,
		prober.NewInstrumentation(comp, logger, extprom.WrapRegistererWithPrefix("thanos_", reg)),
	)

	metrics := newDownsampleMetrics(reg)
	// Start cycle of syncing blocks from the bucket and garbage collecting the bucket.
	{
		ctx, cancel := context.WithCancel(context.Background())

		g.Add(func() error {
			defer runutil.CloseWithLogOnErr(logger, insBkt, "bucket client")
			statusProber.Ready()

			return runutil.Repeat(waitInterval, ctx.Done(), func() error {
				level.Info(logger).Log("msg", "start first pass of downsampling")
				metas, _, err := metaFetcher.Fetch(ctx)
				if err != nil {
					return errors.Wrap(err, "sync before first pass of downsampling")
				}

				for _, meta := range metas {
					resolutionLabel := meta.Thanos.ResolutionString()
					metrics.downsamples.WithLabelValues(resolutionLabel)
					metrics.downsampleFailures.WithLabelValues(resolutionLabel)
				}
				if err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {
					return errors.Wrap(err, "downsampling failed")
				}

				level.Info(logger).Log("msg", "start second pass of downsampling")
				metas, _, err = metaFetcher.Fetch(ctx)
				if err != nil {
					return errors.Wrap(err, "sync before second pass of downsampling")
				}
				if err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {
					return errors.Wrap(err, "downsampling failed")

View on GitHub (pinned to 35b8b99117)