thanos-io/thanos · error

download from source

Error message

download from source

What it means

BackupAndDelete downloads the block to a local temp dir with block.Download before backing it up; a failure is wrapped as "download from source". The broken block could not be fully retrieved from the source bucket, so neither backup nor delete happened.

Solutions

1. Read the wrapped cause: missing-object errors indicate an incomplete block in the source bucket.
2. Check temp disk space and permissions (os.MkdirTemp target filesystem).
3. Retry after transient object-store/network issues.
4. If the block is incomplete in the bucket, restore from backup or delete it manually — repair/delete cannot proceed without a full download.

Example fix

// before: small temp filesystem filling up
TMPDIR=/run/user/tmp
// after: temp dir on a volume with enough free space
TMPDIR=/var/tmp th (...verifier run...)

// ensure temp volume has room for the block before download
if st, err := os.Stat(tempdir); err == nil && syscall.Statfs(tempdir).Bavail*uint64(st.Size()) < blockSize { ... }
// simpler: confirm all block objects exist in source bucket first

if err := verifier.VerifyRepair(...); err != nil {
    if strings.Contains(err.Error(), "download from source") {
        // retry on transient errors; missing objects mean the block is incomplete — restore or delete manually
    }
}

When it happens

Trigger: block.Download fails: missing objects in the block dir (incomplete block), GetObject permission/network errors, checksum/size mismatch, or local temp-dir write failures (disk full, permissions).

Common situations: Source block partially deleted or never fully uploaded; object store outage mid-download; insufficient disk space in the OS temp dir; IAM policy allowing List but not Get on the block's objects.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• download index file
• unable to create bucket
• unable to sync blocks
• write index header
• generate index header

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/fca3d7dbdbbd87ff. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/safe_delete.go:67

	if found {
		return errors.Errorf("%s dir seems to exists in backup bucket. Remove this block manually if you are sure it is safe to do", id)
	}

	// Create a tempdir to locally store TSDB block.
	tempdir, err := os.MkdirTemp("", fmt.Sprintf("safe-delete-%s", id.String()))
	if err != nil {
		return err
	}
	defer func() {
		if err := os.RemoveAll(tempdir); err != nil {
			level.Warn(ctx.Logger).Log("msg", "failed to delete dir", "dir", tempdir, "err", err)
		}
	}()

	// Download the TSDB block.
	dir := filepath.Join(tempdir, id.String())
	if err := block.Download(ctx, ctx.Logger, ctx.Bkt, id, dir); err != nil {
		return errors.Wrap(err, "download from source")
	}

	// Backup the block.
	if err := backupDownloaded(ctx, ctx.Logger, dir, ctx.BackupBkt, id); err != nil {
		return err
	}

	// Block uploaded, so we are ok to remove from src bucket.
	if ctx.DeleteDelay.Seconds() == 0 {
		level.Info(ctx.Logger).Log("msg", "Deleting block", "id", id.String())
		if err := block.Delete(ctx, ctx.Logger, ctx.Bkt, id); err != nil {
			return errors.Wrap(err, "delete from source")
		}
	}

	level.Info(ctx.Logger).Log("msg", "Marking block as deleted", "id", id.String())
	if err := block.MarkForDeletion(ctx, ctx.Logger, ctx.Bkt, id, "manual verify-repair", ctx.metrics.blocksMarkedForDeletion); err != nil {
		return errors.Wrap(err, "marking delete from source")

View on GitHub (pinned to 35b8b99117)