thanos-io/thanos · error

download index file

Error message

download index file %s

What it means

verifyIndex downloads the block's index file from the bucket with objstore.DownloadFile; failure is wrapped as "download index file %s" (object key "<ULID>/index"). This is the read path used when verifying (not repairing) a block's index health.

Solutions

  1. Check the wrapped cause: NoSuchKey means the block is incomplete in the bucket — inspect the block's meta.json/objects.
  2. Verify objstore config points to the intended bucket.
  3. Test GetObject access with the same credentials.
  4. Retry on transient network errors; enable client retries in the objstore config.
  5. If the block is genuinely incomplete, restore it from backup or delete/mark it for compaction.

Example fix

// before: incomplete block in bucket (missing index object)
// cause: s3 NoSuchKey <ulid>/index
// after: restore block from backup then verify again
$ thanos tools bucket verify --objstore.config-file=s3.yaml --id=<ulid>
Defensive patterns

Strategy: try-catch

Validate before calling

// check the index object exists before verification
ok, err := bkt.Exists(ctx, path.Join(id.String(), "index"))
if err != nil || !ok { return fmt.Errorf("block %s incomplete: no index object", id) }

Try / catch

if err := verify.VerifyIssue(ctx, ...); err != nil {
    if strings.Contains(err.Error(), "download index file") {
        // if wrapped error is NoSuchKey, mark block incomplete / restore from backup
    }
}

Prevention

When it happens

Trigger: DownloadFile fails because the index object does not exist in the bucket (missing block files), the object key is wrong, the object store is unreachable, credentials lack GetObject permission, or the network drops mid-transfer.

Common situations: Block partially uploaded/deleted in the bucket (missing index object -> NoSuchKey); wrong objstore config pointing at the wrong bucket/prefix; expired credentials; rate limiting from the object store.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/123b4ffe938a46f4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/index_issue.go:134

		return errors.Wrapf(err, "repaired block is invalid %s", resid)
	}

	level.Info(ctx.Logger).Log("msg", "uploading repaired block", "newID", resid)
	if err = block.Upload(ctx, ctx.Logger, ctx.Bkt, filepath.Join(dir, resid.String()), metadata.NoneFunc); err != nil {
		return errors.Wrapf(err, "upload of %s failed", resid)
	}

	level.Info(ctx.Logger).Log("msg", "safe deleting broken block", "id", id, "issue")
	if err := BackupAndDeleteDownloaded(ctx, filepath.Join(dir, id.String()), id); err != nil {
		return errors.Wrapf(err, "safe deleting old block %s failed", id)
	}

	return nil
}

func verifyIndex(ctx Context, id ulid.ULID, dir string, meta *metadata.Meta) (stats block.HealthStats, err error) {
	if err := objstore.DownloadFile(ctx, ctx.Logger, ctx.Bkt, path.Join(id.String(), block.IndexFilename), filepath.Join(dir, block.IndexFilename)); err != nil {
		return stats, errors.Wrapf(err, "download index file %s", path.Join(id.String(), block.IndexFilename))
	}

	stats, err = block.GatherIndexHealthStats(ctx, ctx.Logger, filepath.Join(dir, block.IndexFilename), meta.MinTime, meta.MaxTime)
	if err != nil {
		return stats, errors.Wrapf(err, "gather index issues %s", id)
	}

	level.Debug(ctx.Logger).Log("stats", fmt.Sprintf("%+v", stats), "id", id)

	return stats, stats.AnyErr()
}

View on GitHub (pinned to 35b8b99117)