thanos-io/thanos · error

check exists

Error message

check exists

What it means

Wraps an error from s.bucket.Exists(ctx, <ulid>/meta.json) during Sync: the bucket existence check for a block's meta file failed. Sync checks the bucket before uploading each block to skip already-uploaded ones; a failed Exists call (network, auth, object store error) aborts the remaining sync with the partial 'uploaded' list.

Solutions

1. Inspect the wrapped cause — it names the backend (s3/gcs/azure) and the underlying error; fix that first.
2. Verify objstore.config credentials and bucket name/region with `thanos tools bucket verify` or a direct listing.
3. Check network connectivity/DNS and any proxy/firewall between the sidecar and the object store endpoint.
4. Rotate expired credentials and confirm IAM permissions (s3:HeadObject / storage.objects.get).
5. Retry — Sync is idempotent and safe to re-run; consider increasing provider timeouts if throttled.

Example fix

// before
// check exists: s3: HeadObject https://s3.eu-west-1.amazonaws.com/thanos-bucket/01ARZ.../meta.json: AccessDenied
// after (fix IAM policy)
// {"Effect":"Allow","Action":["s3:ListBucket","s3:GetObject","s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::thanos-bucket","arn:aws:s3:::thanos-bucket/*"]}

cfg, err := objstore.ClientFromConfig(bucketCfg)
if err != nil { return err }
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
if _, err := cfg.Iter(ctx, "", func(string) error { return nil }); err != nil {
    return fmt.Errorf("bucket preflight failed: %w", err)
}

if _, err := shipper.Sync(ctx); err != nil {
    if strings.Contains(err.Error(), "check exists") {
        // backend auth/network issue; back off and retry with jitter
        return retry.WithBackoff(ctx, syncFn)
    }
}

When it happens

Trigger: Any Sync invocation where the object storage backend returns an error for the Exists request — e.g. bucket does not exist, expired credentials, DNS/network failure, throttling, or misconfigured objstore endpoint.

Common situations: Wrong bucket name or region in objstore config; IAM/AccessKey revoked or clock skew breaking S3 signatures; GCS/S3 throttling (rate limits); network partition between sidecar and object storage.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• get range reader
• read postings range
• sync before first pass of downsampling
• sync before second pass of downsampling
• upload file to bucket

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6b6430f8bc02f837. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:413

		}

		if m.Stats.NumSamples == 0 {
			// Ignore empty blocks.
			level.Debug(s.logger).Log("msg", "ignoring empty block", "block", m.ULID)
			continue
		}

		// We only ship of the first compacted block level as normal flow.
		if m.Compaction.Level > 1 {
			if !s.uploadCompacted {
				continue
			}
		}

		// Check against bucket if the meta file for this block exists.
		ok, err := s.bucket.Exists(ctx, path.Join(m.ULID.String(), block.MetaFilename))
		if err != nil {
			return uploaded, errors.Wrap(err, "check exists")
		}
		if ok {
			meta.Uploaded = append(meta.Uploaded, m.ULID)
			continue
		}

		// Skip overlap check if out of order uploads is enabled.
		if m.Compaction.Level > 1 && !s.allowOutOfOrderUploads {
			if err := checker.IsOverlapping(ctx, m.BlockMeta); err != nil {
				return uploaded, errors.Errorf("Found overlap or error during sync, cannot upload compacted block, details: %v", err)
			}
		}

		if err := s.upload(ctx, m); err != nil {
			if !s.allowOutOfOrderUploads {
				return uploaded, errors.Wrapf(err, "upload %v", m.ULID)
			}

View on GitHub (pinned to 35b8b99117)