thanos-io/thanos · error

download block

Error message

download block %s

What it means

During RepairIssue347, the affected block is re-downloaded from the object store bucket into a temp dir before it can be rebuilt. This error wraps any failure of block.Download for the corrupted block ID — network failures, missing object parts, or bucket access errors — and is passed to retry() so the whole repair can be retried.

Solutions

1. Verify the bucket credentials/permissions allow reading the block prefix (meta.json, index, chunks/*)
2. Check the block actually exists in the bucket (`thanos tools bucket ls` / verify); if partially uploaded, remove or re-upload it
3. Rely on the built-in retry — for persistent failures check object-store connectivity (endpoint, DNS, TLS) and rate limits
4. Free disk space on the compactor tmp volume if the download fails writing locally
5. If the object is unrecoverable, delete/mark the corrupted block for deletion so compaction proceeds

Example fix

// before: wrong credentials
--objstore.bucket=s3 --s3.access-key=<readonly-on-other-bucket>
// after: grant read on the bucket
--objstore.bucket=s3 --s3.access-key=<key-with-bucket-read>

// before repair, check the block is downloadable
rc, err := bkt.Get(ctx, path.Join(ie.id.String(), "meta.json"))
if err != nil {
    return fmt.Errorf("block %s not readable in bucket: %w", ie.id, err)
}
rc.Close()
// and check local space
if free(tmpdir) < minBlockBytes {
    return fmt.Errorf("insufficient disk space for download")
}

// Go
err := block.Download(ctx, logger, bkt, id, bdir)
if err != nil {
    if isRetryable(err) { // net.Error, 5xx, throttling
        backoff.Retry(func() error { return block.Download(ctx, logger, bkt, id, bdir) }, ...)
    }
    return errors.Wrapf(err, "download block %s", id)
}

When it happens

Trigger: block.Download(ctx, logger, bkt, ie.id, bdir) fails: object missing (404) in the bucket, transient network/S3 errors, permission denied on bucket read, or local disk full/write error while writing chunk/index files into tmpdir.

Common situations: Bucket credentials lacking read permission for the block prefix; object store throttling (S3 503/slow-down); partially uploaded block missing meta.json or index; DNS/proxy outages between compactor and object storage; tmp disk quota exceeded.

Related errors

• download block
• upload downsampled block
• write index header
• generate index header
• write index header

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f7d0a41aa5cbd39b. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1131

		return errors.Errorf("Given error is not an issue347 error: %v", issue347Err)
	}

	level.Info(logger).Log("msg", "Repairing block broken by https://github.com/prometheus/tsdb/issues/347", "id", ie.id, "err", issue347Err)

	tmpdir, err := os.MkdirTemp("", fmt.Sprintf("repair-issue-347-id-%s-", ie.id))
	if err != nil {
		return err
	}

	defer func() {
		if err := os.RemoveAll(tmpdir); err != nil {
			level.Warn(logger).Log("msg", "failed to remote tmpdir", "err", err, "tmpdir", tmpdir)
		}
	}()

	bdir := filepath.Join(tmpdir, ie.id.String())
	if err := block.Download(ctx, logger, bkt, ie.id, bdir); err != nil {
		return retry(errors.Wrapf(err, "download block %s", ie.id))
	}

	meta, err := metadata.ReadFromDir(bdir)
	if err != nil {
		return errors.Wrapf(err, "read meta from %s", bdir)
	}

	resid, err := block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk)
	if err != nil {
		return errors.Wrapf(err, "repair failed for block %s", ie.id)
	}

	// Verify repaired id before uploading it.
	if err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {
		return errors.Wrapf(err, "repaired block is invalid %s", resid)
	}

	level.Info(logger).Log("msg", "uploading repaired block", "newID", resid)

View on GitHub (pinned to 35b8b99117)