thanos-io/thanos · error

get from origin bucket

Error message

get %v from origin bucket

What it means

ensureObjectReplicated failed to fetch an object from the origin bucket via rs.fromBkt.Get. If the source read fails, replication to the target bucket cannot proceed and the error is wrapped with the object name.

Solutions

1. Check the source bucket still contains the object (mc ls / gsutil ls)
2. Verify fromBkt credentials and permissions (read access)
3. Retry — the replication loop re-runs and skips if the source block is intentionally gone (ignore deletion races)
4. Check object storage logs/endpoint reachability and rate limits

// pre-check object presence in origin before replication
if _, err := fromBkt.Exists(ctx, objectName); err != nil {
    return fmt.Errorf("object %s not available in origin: %w", objectName, err)
}

_, err := rs.fromBkt.Get(ctx, objectName)
if err != nil {
    if errors.Is(err, context.Canceled) { return err }
    if isNotExist(err) {
        // source block already deleted: skip, don't retry
        return nil
    }
    return errors.Wrapf(err, "get %v from origin bucket", objectName)
}

When it happens

Trigger: Block replication calls ensureObjectReplicated for an object name; the object bucket Get fails with NotFound (object deleted/GC'd from source), permissions error, or backend connectivity error.

Common situations: Compactor removed the block between listing and fetching; wrong bucket/endpoint credentials; object storage outage or throttling; object filtered by relabel still referenced.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• compaction
• sync before first pass of downsampling
• sync before second pass of downsampling
• sync before retention
• retention failed

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/431aab8f57cab0bf. Report an issue: GitHub.

Appendix: source

Thrown at pkg/replicate/scheme.go:298

func (rs *replicationScheme) ensureObjectReplicated(ctx context.Context, objectName string) error {
	level.Debug(rs.logger).Log("msg", "ensuring object is replicated", "object", objectName)

	exists, err := rs.toBkt.Exists(ctx, objectName)
	if err != nil {
		return errors.Wrapf(err, "check if %v exists in target bucket", objectName)
	}

	// skip if already exists.
	if exists {
		level.Debug(rs.logger).Log("msg", "skipping object as already replicated", "object", objectName)
		return nil
	}

	level.Debug(rs.logger).Log("msg", "object not present in target bucket, replicating", "object", objectName)

	r, err := rs.fromBkt.Get(ctx, objectName)
	if err != nil {
		return errors.Wrapf(err, "get %v from origin bucket", objectName)
	}

	defer r.Close()

	if err = rs.toBkt.Upload(ctx, objectName, r); err != nil {
		return errors.Wrapf(err, "upload %v to target bucket", objectName)
	}

	level.Info(rs.logger).Log("msg", "object replicated", "object", objectName)
	rs.metrics.objectsReplicated.Inc()

	return nil
}

View on GitHub (pinned to 35b8b99117)