thanos-io/thanos · error

get meta file

Error message

get meta file: %v

What it means

loadMeta wraps any non-not-found error from fetching meta.json as errors.Wrapf(err, "get meta file: %v", metaFile). Unlike ErrorSyncMetaNotFound, this is a genuine storage failure (network, permissions, throttling, client errors) — the object still exists but the Get call failed. The original error is preserved as the cause, and the block metadata read fails.

Solutions

1. Inspect the wrapped cause error to identify the storage-layer failure (auth, throttling, network).
2. Retry the sync — transient network/throttle failures usually resolve; bucket fetchers retry on subsequent syncs.
3. Fix object-store credentials/permissions if the cause is access denied.
4. Enable request retries/rate limiting in the objstore client config (e.g. S3 max_retries, GCS backoff) for large buckets.

Example fix

// before: no retry/backoff configured
// s3:
//   endpoint: s3.amazonaws.com

// after: enable retries for transient Get failures
// s3:
//   endpoint: s3.amazonaws.com
//   max_retries: 5
// http_config:
//   backoff: ...

// pre-flight: verify credentials and bucket reachability before sync
if _, err := bkt.Iter(ctx, "", nil); err != nil {
    return fmt.Errorf("bucket not reachable/authorized: %w", err)
}

func isGetMetaFailure(err error) bool {
    return err != nil && !errors.Is(err, block.ErrorSyncMetaNotFound) &&
        strings.Contains(err.Error(), "get meta file:")
}

err := retry(ctx, 3, func() error {
    _, ferr := loadMetaFn(ctx)
    if ferr != nil && !errors.Is(ferr, block.ErrorSyncMetaNotFound) {
        return ferr // transient Get failure: retry with backoff
    }
    return nil
})

When it happens

Trigger: loadMeta: ReaderWithExpectedErrs(...).Get(ctx, metaFile) returns an error that is NOT an object-not-found error (IsObjNotFoundErr false); it is wrapped with 'get meta file: <path>'. Causes include S3/GCS API errors, network timeouts, IAM/permission denials, rate limiting.

Common situations: S3 throttling (SlowDown) during large bucket syncs; expired or missing object-store credentials; network partition between Thanos and the object store; misconfigured bucket endpoint or region; GCS per-request rate limits.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• %v
• read meta file
• upload of failed
• read meta.json for block
• check exists in bucket

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5ca29f80ea9a4346. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/fetcher.go:484

		if err == nil {
			return m, nil
		}

		if !errors.Is(err, os.ErrNotExist) {
			level.Warn(f.logger).Log("msg", "best effort read of the local meta.json failed; removing cached block dir", "dir", cachedBlockDir, "err", err)
			if err := os.RemoveAll(cachedBlockDir); err != nil {
				level.Warn(f.logger).Log("msg", "best effort remove of cached dir failed; ignoring", "dir", cachedBlockDir, "err", err)
			}
		}
	}

	r, err := f.bkt.ReaderWithExpectedErrs(f.bkt.IsObjNotFoundErr).Get(ctx, metaFile)
	if f.bkt.IsObjNotFoundErr(err) {
		// Meta.json was deleted between bkt.Exists and here.
		return nil, errors.Wrapf(ErrorSyncMetaNotFound, "%v", err)
	}
	if err != nil {
		return nil, errors.Wrapf(err, "get meta file: %v", metaFile)
	}

	defer runutil.CloseWithLogOnErr(f.logger, r, "close bkt meta get")

	metaContent, err := io.ReadAll(r)
	if err != nil {
		return nil, errors.Wrapf(err, "read meta file: %v", metaFile)
	}

	m := &metadata.Meta{}
	if err := json.Unmarshal(metaContent, m); err != nil {
		return nil, errors.Wrapf(ErrorSyncMetaCorrupted, "meta.json %v unmarshal: %v", metaFile, err)
	}

	if m.Version != metadata.TSDBVersion1 {
		return nil, errors.Errorf("unexpected meta file: %s version: %d", metaFile, m.Version)
	}

View on GitHub (pinned to 35b8b99117)