thanos-io/thanos · error

upload

Error message

upload %v

What it means

Wraps an error from s.upload(ctx, m) during Sync when out-of-order uploads are disabled — any block upload failure aborts the whole sync and returns the partial uploaded list. The underlying error (in [908]/[909] or bucket Put calls) is included with the failing block ULID.

Solutions

1. Look at the wrapped cause for the specific block; if the block files are corrupted, delete the local block and restore it or let Prometheus re-create it.
2. If the cause is object storage, fix credentials/network/throttling (see [904]) and re-run Sync — it is idempotent.
3. Consider enabling --shipper.allow-out-of-order-uploads so single-block failures don't block all other uploads; failed blocks are retried on the next sync.
4. Free disk space / fix data-dir permissions if hard-link creation failed.

Example fix

// before
// upload 01ARZ...: upload file /data/01ARZ.../chunks/000001: read /data/01ARZ.../chunks/000001: input/output error
// after (resilient config)
// thanos sidecar --data-dir=/data --bucket=thanos --shipper.allow-out-of-order-uploads
// # then repair/replace the corrupted block 01ARZ...

if err := validateBlockIntegrity(blockDir); err != nil {
    return fmt.Errorf("block %s corrupt, skip upload: %w", blockDir, err)
}

if _, err := shipper.Sync(ctx); err != nil {
    var partial map[ulid.ULID]struct{}
    if errors.As(err, &uploaded) {
        // uploaded is returned alongside the error; retry only the missing ULIDs
    }
}

When it happens

Trigger: Sync with allowOutOfOrderUploads=false where uploading a block fails: hard-link creation into the upload dir fails, a file read fails (corrupted block), or the object storage Put/multipart upload errors.

Common situations: Corrupted segment files in a block (Checksum mismatches); object store throttling or auth expiry mid-sync; disk full or read-only data dir preventing hard-link creation; network interruption during a large compacted-block upload.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• failed to sync / blocks
• upload downsampled block
• upload file to bucket
• invalid magic number
• read meta from

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e30b10e3260343a4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:429

		ok, err := s.bucket.Exists(ctx, path.Join(m.ULID.String(), block.MetaFilename))
		if err != nil {
			return uploaded, errors.Wrap(err, "check exists")
		}
		if ok {
			meta.Uploaded = append(meta.Uploaded, m.ULID)
			continue
		}

		// Skip overlap check if out of order uploads is enabled.
		if m.Compaction.Level > 1 && !s.allowOutOfOrderUploads {
			if err := checker.IsOverlapping(ctx, m.BlockMeta); err != nil {
				return uploaded, errors.Errorf("Found overlap or error during sync, cannot upload compacted block, details: %v", err)
			}
		}

		if err := s.upload(ctx, m); err != nil {
			if !s.allowOutOfOrderUploads {
				return uploaded, errors.Wrapf(err, "upload %v", m.ULID)
			}

			// No error returned, just log line. This is because we want other blocks to be uploaded even
			// though this one failed. It will be retried on second Sync iteration.
			level.Error(s.logger).Log("msg", "shipping failed", "block", m.ULID, "err", err)
			uploadErrs++
			continue
		}
		meta.Uploaded = append(meta.Uploaded, m.ULID)
		uploaded++
		s.metrics.uploads.Inc()
	}
	if err := WriteMetaFile(s.logger, s.metadataFilePath, meta); err != nil {
		level.Warn(s.logger).Log("msg", "updating meta file failed", "err", err)
	}

	failedExecution = false
	if uploadErrs > 0 || len(failedBlocks) > 0 {

View on GitHub (pinned to 35b8b99117)