What causes "upload of %s failed" in thanos-io/thanos?

Inspect the wrapped cause: for auth errors, refresh/rotate the object store credentials.

upload of failed

Error message

upload of %s failed

What it means

After the repaired block passes verification, repairIndex uploads it to the object store via block.Upload; a failure is wrapped as "upload of %s failed". The repaired block exists locally but was not persisted to the bucket.

Solutions

Inspect the wrapped cause: for auth errors, refresh/rotate the object store credentials.
Test bucket write access with the same credentials (e.g. 'thanos tools bucket ls' and a test put).
Re-run the repair; Upload is idempotent per block ID and will retry the transfer.
If 'already exists' style cause, verify the existing remote block is not corrupt; remove it manually before re-uploading.
Increase object store client timeout/retry settings in the objstore config.

Example fix

// before: verifier objstore config with a short-lived static key
{"type":"S3","config":{"access_key":"AKIA..."}}
// after: use instance role / credential refresh
{"type":"S3","config":{"role_arn":"arn:aws:iam::...:role/thanos-verifier"}}

Defensive patterns

Strategy: retry

Validate before calling

// pre-check bucket write access
_, err := bkt.Upload(ctx, ".probe/thanos-verifier-probe", strings.NewReader("ok"))

Try / catch

if err := verifier.VerifyRepair(...); err != nil {
    if strings.Contains(err.Error(), "upload of ") {
        // backoff-retry the whole repair; upload is safe to repeat per block ID
    }
}

Prevention

Use non-expiring credentials (instance roles/workload identity)
Enable retries and adequate timeouts in the objstore config
Test PutObject permission before long verification runs

When it happens

Trigger: block.Upload to ctx.Bkt fails: object store unreachable, credentials invalid/expired, bucket write permission missing, network timeout mid-multipart, or the destination block ID already exists in the bucket.

Common situations: Expired S3/GCS credentials; misconfigured objstore config in the verifier; bucket policy denying PutObject; proxy/firewall cutting long uploads; a previous partial upload leaving the resid prefix present.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6fe14c37058bb07a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/index_issue.go:121

		dir,
		id,
		metadata.BucketRepairSource,
		block.IgnoreCompleteOutsideChunk,
		block.IgnoreDuplicateOutsideChunk,
		block.IgnoreIssue347OutsideChunk,
	)
	if err != nil {
		return errors.Wrapf(err, "repair failed for block %s", id)
	}
	level.Info(ctx.Logger).Log("msg", "verifying repaired block", "id", id, "newID", resid)

	if err := block.VerifyIndex(ctx, ctx.Logger, filepath.Join(dir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {
		return errors.Wrapf(err, "repaired block is invalid %s", resid)
	}

	level.Info(ctx.Logger).Log("msg", "uploading repaired block", "newID", resid)
	if err = block.Upload(ctx, ctx.Logger, ctx.Bkt, filepath.Join(dir, resid.String()), metadata.NoneFunc); err != nil {
		return errors.Wrapf(err, "upload of %s failed", resid)
	}

	level.Info(ctx.Logger).Log("msg", "safe deleting broken block", "id", id, "issue")
	if err := BackupAndDeleteDownloaded(ctx, filepath.Join(dir, id.String()), id); err != nil {
		return errors.Wrapf(err, "safe deleting old block %s failed", id)
	}

	return nil
}

func verifyIndex(ctx Context, id ulid.ULID, dir string, meta *metadata.Meta) (stats block.HealthStats, err error) {
	if err := objstore.DownloadFile(ctx, ctx.Logger, ctx.Bkt, path.Join(id.String(), block.IndexFilename), filepath.Join(dir, block.IndexFilename)); err != nil {
		return stats, errors.Wrapf(err, "download index file %s", path.Join(id.String(), block.IndexFilename))
	}

	stats, err = block.GatherIndexHealthStats(ctx, ctx.Logger, filepath.Join(dir, block.IndexFilename), meta.MinTime, meta.MaxTime)
	if err != nil {
		return stats, errors.Wrapf(err, "gather index issues %s", id)

View on GitHub (pinned to 35b8b99117)