thanos-io/thanos · error

upload

Error message

upload

What it means

Wraps block.UploadPromBlock failing when uploading the rewritten block as a Prometheus-style block to the object store. Upload writes all block files (index, chunks, meta.json) and verifies checksums; failures stem from object store errors, network issues, permission problems, or corrupted local files (hash mismatch on upload).

Solutions

1. Verify object store write permissions and credentials for the bucket, then retry the rewrite.
2. Retry after confirming network stability; the operation is safe to re-run since IDs are freshly generated.
3. If checksum errors persist, clear tmpDir and re-download/re-rewrite the source block.

Example fix

// before (stale credentials)
export AWS_SECRET_ACCESS_KEY=old
// after
export AWS_SECRET_ACCESS_KEY=$(aws get-new-key) # re-auth before long-running rewrite

// pre-check write access
if err := bkt.Upload(ctx, path.Join(testPrefix, "probe"), []byte("ok")); err != nil {
    return errors.Wrap(err, "bucket not writable")
}

if err := block.UploadPromBlock(ctx, logger, bkt, dir, hashFn); err != nil {
    if isRateLimit(err) { time.Sleep(30*time.Second); return retry() }
    return errors.Wrap(err, "upload")
}

When it happens

Trigger: tbc.promBlocks is set and the upload of the new block directory fails: wrong bucket/credentials, network interruption, object-store rate limiting, or a local file corruption detected during hashing.

Common situations: Expired cloud credentials during long rewrite runs; S3/GCS throttling when uploading many chunk files; bucket write permissions missing for the service account; partial local flush producing bad checksums.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• download
• upload of failed
• upload downsampled block
• upload file to bucket
• read meta.json for block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4a94e032f8215771. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/tools_bucket.go:1322

				if tbc.dryRun {
					level.Info(logger).Log("msg", "dry run finished. Changes should be printed to stderr", "Block ID", id)
					continue
				}

				level.Info(logger).Log("msg", "wrote new block after modifications; flushing", "source", id, "new", newID)
				meta.Stats, err = d.Flush()
				if err != nil {
					return errors.Wrap(err, "flush")
				}
				if err := meta.WriteToDir(logger, filepath.Join(tbc.tmpDir, newID.String())); err != nil {
					return err
				}

				level.Info(logger).Log("msg", "uploading new block", "source", id, "new", newID)
				if tbc.promBlocks {
					if err := block.UploadPromBlock(ctx, logger, insBkt, filepath.Join(tbc.tmpDir, newID.String()), metadata.HashFunc(*hashFunc)); err != nil {
						return errors.Wrap(err, "upload")
					}
				} else {
					if err := block.Upload(ctx, logger, insBkt, filepath.Join(tbc.tmpDir, newID.String()), metadata.HashFunc(*hashFunc)); err != nil {
						return errors.Wrap(err, "upload")
					}
				}
				level.Info(logger).Log("msg", "uploaded", "source", id, "new", newID)

				if !tbc.dryRun && tbc.deleteBlocks {
					if err := block.MarkForDeletion(ctx, logger, insBkt, id, "block rewritten", stubCounter); err != nil {
						level.Error(logger).Log("msg", "failed to mark block for deletion", "id", id.String(), "err", err)
					}
				}
			}
			level.Info(logger).Log("msg", "rewrite done", "IDs", strings.Join(tbc.blockIDs, ","))
			return nil
		}, func(err error) {
			cancel()

View on GitHub (pinned to 35b8b99117)