thanos-io/thanos · error

download

Error message

download %v

What it means

Wraps an error from block.Download, which fetches the block's index/chunks/meta files from the object store into tmpDir during `tools bucket rewrite`. The error means the block's data could not be fully downloaded, either because it does not exist (deleted concurrently, wrong bucket), or due to an object store/network failure.

Solutions

1. Verify the block ID still exists with `thanos tools bucket ls` before rewriting and re-check after the failure.
2. Pause/drain the compactor for the bucket so it does not delete the block mid-rewrite.
3. Check object store connectivity/credentials and retry; check for rate-limit errors in the wrapped cause.

Example fix

// before (compactor deleting concurrently)
thanos tools bucket rewrite --objstore.bucket=b ID
// after (run with compactor drained for the bucket)
thanos tools bucket rewrite --objstore.bucket=b --consistent-delay ... ID # and no compactor running

// confirm block exists before rewrite
thanos tools bucket ls --objstore.bucket=b | grep -q "$ID" || exit 1

if err := block.Download(ctx, logger, bkt, id, dir); err != nil {
    if errors.Is(err, context.Canceled) { return err }
    level.Warn(logger).Log("msg", "download failed, retrying", "err", err)
    time.Sleep(backoff)
}

When it happens

Trigger: Running rewrite when the block was deleted by compaction/garbage collection between listing and downloading; wrong bucket or credentials; transient network failure or object-store rate limiting during download.

Common situations: Rewriting an old block concurrently with a running compactor that removed it; misconfigured --objstore.prefix or bucket name pointing elsewhere; S3/GCS throttling; paused network connectivity in CI.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• upload
• read meta.json for block
• check exists in bucket
• get meta file
• read meta file

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/2f45cd931eb94867. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/tools_bucket.go:1250

		}

		if err := os.RemoveAll(tbc.tmpDir); err != nil {
			return err
		}
		if err := os.MkdirAll(tbc.tmpDir, os.ModePerm); err != nil {
			return err
		}

		ctx, cancel := context.WithCancel(context.Background())
		g.Add(func() error {
			chunkPool := chunkenc.NewPool()
			changeLog := compactv2.NewChangeLog(io.Discard)
			stubCounter := promauto.With(nil).NewCounter(prometheus.CounterOpts{})
			for _, id := range ids {
				// Delete series from block & modify.
				level.Info(logger).Log("msg", "downloading block", "source", id)
				if err := block.Download(ctx, logger, insBkt, id, filepath.Join(tbc.tmpDir, id.String())); err != nil {
					return errors.Wrapf(err, "download %v", id)
				}

				meta, err := metadata.ReadFromDir(filepath.Join(tbc.tmpDir, id.String()))
				if err != nil {
					return errors.Wrapf(err, "read meta of %v", id)
				}
				b, err := tsdb.OpenBlock(logutil.GoKitLogToSlog(logger), filepath.Join(tbc.tmpDir, id.String()), chunkPool, nil)
				if err != nil {
					return errors.Wrapf(err, "open block %v", id)
				}

				p := compactv2.NewProgressLogger(logger, int(b.Meta().Stats.NumSeries))
				newID := ulid.MustNew(ulid.Now(), rand.Reader)
				meta.ULID = newID
				meta.Thanos.Rewrites = append(meta.Thanos.Rewrites, metadata.Rewrite{
					Sources:          meta.Compaction.Sources,
					DeletionsApplied: deletions,
					RelabelsApplied:  relabels,

View on GitHub (pinned to 35b8b99117)