thanos-io/thanos · error · RetryError

upload downsampled block

Error message

upload downsampled block %s

What it means

This error wraps a failure to upload a downsampled Prometheus TSDB block to object storage in Thanos. It is produced in processDownsampling when block.Upload fails, and it is wrapped in compact.NewRetryError so the compaction/worker loop retries it. The underlying cause (object storage connectivity, permissions, partial files) is in the wrapped error.

Solutions

1. Check the wrapped cause in the error for the object-store failure and verify bucket credentials/endpoint/permissions with the same config via `thanos tools bucket verify` or objstore client.
2. Retry: the error is a compact.RetryError, so transient network failures resolve on the next compaction iteration; fix permanent causes (creds, bucket policy) so retries succeed.
3. Confirm the downsampled block directory (resdir) is complete and readable before upload; re-run downsample for that block if files are corrupt.
4. Increase upload concurrency/timeout settings (blockFilesConcurrency, ctx deadline) if uploads are timing out on large blocks.

Example fix

// before
err = block.Upload(ctx, logger, bkt, resdir, hashFunc, objstore.WithUploadConcurrency(blockFilesConcurrency))
// after: fail fast with clearer diagnostics
if err := objstore.UploadProbe(ctx, bkt); err != nil {
    level.Error(logger).Log("msg", "object store unreachable, skipping downsample upload", "err", err)
    return compact.NewRetryError(errors.Wrapf(err, "upload downsampled block %s", id))
}
err = block.Upload(ctx, logger, bkt, resdir, hashFunc, objstore.WithUploadConcurrency(blockFilesConcurrency))

if err := bkt.Iter(ctx, "/", func(string) error { return nil }); err != nil { return fmt.Errorf("object store unreachable before upload: %w", err) }

func isRetryable(err error) bool { var re *compact.RetryError; return errors.As(err, &re) }

err := block.Upload(ctx, logger, bkt, resdir, hashFunc, opts)
var re *compact.RetryError
if errors.As(err, &re) { logTransient(err); return err } // retried by compactor
if errors.Is(err, context.Canceled) { return err }
logPermanent(err); return err

When it happens

Trigger: block.Upload returns a non-nil error: object store unreachable or credentials invalid, remote write of a block file fails mid-transfer, context cancelled during upload, or the hashed temp dir contents are unreadable.

Common situations: Misconfigured S3/GCS/Azure credentials or endpoint in the objstore config; network outage between Thanos Compactor and bucket; bucket permission (no put-object); too-aggressive ctx timeout killing long uploads of large downsampled blocks.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• upload of failed
• upload of failed
• compaction
• sync before first pass of downsampling
• sync before second pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/d54cff93f8070bdb. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:423

	if err != nil {
		return errors.Wrap(err, "read meta")
	}

	if stats.ChunkMaxSize > 0 {
		meta.Thanos.IndexStats.ChunkMaxSize = stats.ChunkMaxSize
	}
	if stats.SeriesMaxSize > 0 {
		meta.Thanos.IndexStats.SeriesMaxSize = stats.SeriesMaxSize
	}
	if err := meta.WriteToDir(logger, resdir); err != nil {
		return errors.Wrap(err, "write meta")
	}

	begin = time.Now()

	err = block.Upload(ctx, logger, bkt, resdir, hashFunc, objstore.WithUploadConcurrency(blockFilesConcurrency))
	if err != nil {
		return compact.NewRetryError(errors.Wrapf(err, "upload downsampled block %s", id))
	}

	level.Info(logger).Log("msg", "uploaded block", "id", id, "duration", time.Since(begin), "duration_ms", time.Since(begin).Milliseconds())

	// It is not harmful if these fails.
	if err := os.RemoveAll(bdir); err != nil {
		level.Warn(logger).Log("msg", "failed to clean directory", "dir", bdir, "err", err)
	}
	if err := os.RemoveAll(resdir); err != nil {
		level.Warn(logger).Log("msg", "failed to clean directory", "resdir", bdir, "err", err)
	}

	return nil
}

View on GitHub (pinned to 35b8b99117)