thanos-io/thanos · error

upload file to bucket

Error message

upload file %s to bucket

What it means

MarkForDeletion failed uploading deletion-mark.json to the block prefix in the bucket. Until this upload succeeds, garbage collection cannot remove the block, so the marking operation reports failure to callers like compactor repair/clean.

Solutions

1. Check object storage connectivity and upload permissions
2. Inspect the wrapped error for the bucket-level cause
3. Retry marking once storage is healthy

Example fix

// before
if err := block.MarkForDeletion(ctx, logger, bkt, id, "manual"); err != nil { return err }
// after
err := block.MarkForDeletion(ctx, logger, bkt, id, "manual")
if err != nil {
    if ctx.Err() != nil { return ctx.Err() }
    logger.Warn("mark for deletion failed, retrying", "err", err)
    return retry(func() error { return block.MarkForDeletion(ctx, logger, bkt, id, "manual") })
}

ok, err := bkt.Exists(ctx, path.Join(id.String(), "deletion.mark"))
_ = ok // ensure bucket client is configured and reachable first

err := block.MarkForDeletion(ctx, logger, bkt, id, details)
if err != nil {
    if errors.Is(err, context.Canceled) { return err }
    return backoff.Retry(func() error { return block.MarkForDeletion(ctx, logger, bkt, id, details) })
}

When it happens

Trigger: bkt.Upload fails for <ULID>/deletion-mark.json due to network errors, missing bucket/permissions, throttling, or context cancellation during MarkForDeletion.

Common situations: Object store credentials lack put permission; bucket temporarily unavailable or rate-limited; network partition between the compactor and storage; context canceled by shutdown.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• upload of failed
• sync before first pass of downsampling
• sync before second pass of downsampling
• upload downsampled block
• upload

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/2b4e92018c349a23. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/block.go:190

		return errors.Wrapf(err, "check exists %s in bucket", deletionMarkFile)
	}
	if deletionMarkExists {
		level.Warn(logger).Log("msg", "requested to mark for deletion, but file already exists; this should not happen; investigate", "err", errors.Errorf("file %s already exists in bucket", deletionMarkFile))
		return nil
	}

	deletionMark, err := json.Marshal(metadata.DeletionMark{
		ID:           id,
		DeletionTime: time.Now().Unix(),
		Version:      metadata.DeletionMarkVersion1,
		Details:      details,
	})
	if err != nil {
		return errors.Wrap(err, "json encode deletion mark")
	}

	if err := bkt.Upload(ctx, deletionMarkFile, bytes.NewBuffer(deletionMark)); err != nil {
		return errors.Wrapf(err, "upload file %s to bucket", deletionMarkFile)
	}
	markedForDeletion.Inc()
	level.Info(logger).Log("msg", "block has been marked for deletion", "block", id)
	return nil
}

// Delete removes directory that is meant to be block directory.
// NOTE: Always prefer this method for deleting blocks.
//   - We have to delete block's files in the certain order (meta.json first and deletion-mark.json last)
//     to ensure we don't end up with malformed partial blocks. Thanos system handles well partial blocks
//     only if they don't have meta.json. If meta.json is present Thanos assumes valid block.
//   - This avoids deleting empty dir (whole bucket) by mistake.
func Delete(ctx context.Context, logger log.Logger, bkt objstore.Bucket, id ulid.ULID) error {
	metaFile := path.Join(id.String(), MetaFilename)
	deletionMarkFile := path.Join(id.String(), metadata.DeletionMarkFilename)

	// Delete block meta file.
	ok, err := bkt.Exists(ctx, metaFile)

View on GitHub (pinned to 35b8b99117)