thanos-io/thanos · error

upload of failed

Error message

upload of %s failed

What it means

This error wraps a failure of block.Upload when the compactor, after successfully repairing and verifying a block, uploads the new block (resid) from tmpdir to the object store bucket (compact.go repairBucketBlock). It is wrapped with retry(...), so the compactor treats it as retryable. It means the repaired, valid block could not be persisted to storage.

Solutions

1. Check the inner error for the storage-specific cause (HTTP status, credential error); fix IAM/bucket policy to allow write on the bucket
2. Rely on the built-in retry: transient network errors are retried automatically; investigate only if it exhausts retries
3. Verify no external process (tmp cleaner, k8s gc) deletes tmpdir/<resid> during compaction; pin the compactor's tmpdir to durable storage
4. Check object storage service health/quotas (request rate, egress) if failures correlate with large uploads
5. Validate objstore configuration (thanos tools bucket verify) and credentials rotation settings

Example fix

// before: failing bucket config with missing write scope (S3 IAM)
// after: grant the compactor's role PutObject on the bucket prefix
// aws iam: add statement {"Effect":"Allow","Action":["s3:PutObject","s3:AbortMultipartUpload"],"Resource":"arn:aws:s3:::thanos-bucket/*"}

// preflight: ensure the compactor can write to the bucket before long repair jobs
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := bkt.Upload(ctx, "health-check", bytes.NewReader(nil)); err != nil {
    return fmt.Errorf("bucket not writable: %w", err)
}

err := retry(func() error { // thanos' own retry wrapper already applies here
    return block.Upload(ctx, logger, bkt, blockDir, metadata.NoneFunc)
})
if err != nil {
    logger.Error("upload exhausted retries; check object storage health and IAM", "err", err)
}

When it happens

Trigger: block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc) fails: object storage returns 5xx/403/timeout mid-upload, network interruption between compactor and bucket, bucket write permission missing, multipart upload aborted by the provider, or local tmpdir files were removed mid-upload (external cleanup job deleting tmp).

Common situations: S3/GCS/Azure credentials expired or lacking PutObject permission on the bucket; object storage rate limiting or throttling during large uploads; proxy/firewall cutting long connections; compactor's tmp cleanup (or k8s emptyDir eviction) racing with the upload; bucket versioning/retention policies rejecting writes.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• get meta file
• read meta file
• upload downsampled block
• upload
• upload file to bucket

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8282504ea900eb39. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1151

	meta, err := metadata.ReadFromDir(bdir)
	if err != nil {
		return errors.Wrapf(err, "read meta from %s", bdir)
	}

	resid, err := block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk)
	if err != nil {
		return errors.Wrapf(err, "repair failed for block %s", ie.id)
	}

	// Verify repaired id before uploading it.
	if err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {
		return errors.Wrapf(err, "repaired block is invalid %s", resid)
	}

	level.Info(logger).Log("msg", "uploading repaired block", "newID", resid)
	if err = block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc); err != nil {
		return retry(errors.Wrapf(err, "upload of %s failed", resid))
	}

	level.Info(logger).Log("msg", "deleting broken block", "id", ie.id)

	// Spawn a new context so we always mark a block for deletion in full on shutdown.
	delCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
	defer cancel()

	// TODO(bplotka): Issue with this will introduce overlap that will halt compactor. Automate that (fix duplicate overlaps caused by this).
	if err := block.MarkForDeletion(delCtx, logger, bkt, ie.id, "source of repaired block", blocksMarkedForDeletion); err != nil {
		return errors.Wrapf(err, "marking old block %s for deletion has failed", ie.id)
	}
	return nil
}

func (cg *Group) compact(ctx context.Context, dir string, planner Planner, comp Compactor, blockDeletableChecker BlockDeletableChecker, compactionLifecycleCallback CompactionLifecycleCallback, errChan chan error) (bool, []ulid.ULID, error) {
	cg.mtx.Lock()
	defer cg.mtx.Unlock()

View on GitHub (pinned to 35b8b99117)