thanos-io/thanos · warning · ErrorSyncMetaNotFound

%v

Error message

%v

What it means

This is the opaque '%v'-formatted wrapper produced in loadMeta when the object-not-found error is wrapped by ErrorSyncMetaNotFound: errors.Wrapf(ErrorSyncMetaNotFound, "%v", err). The message is literally the stringified underlying bucket error (e.g. the S3/GCS not-found message) and carries ErrorSyncMetaNotFound as its cause. It means meta.json vanished between the Exists check and the Get call.

Solutions

1. Handle as expected: the fetcher counts it as noMetas and continues; no action needed for a single occurrence.
2. Increase compactor deletion-delay so blocks are not removed while syncers are in flight.
3. Identify the deleting actor (compactor, lifecycle policy, manual cleanup) if it recurs frequently.
4. Verify bucket client configuration (endpoint, prefix) is correct so not-found errors are not caused by looking in the wrong location.

func isSyncMetaNotFound(err error) bool {
    return errors.Is(err, block.ErrorSyncMetaNotFound)
}

_, err := fetcher.Fetch(ctx, metas)
if errors.Is(err, block.ErrorSyncMetaNotFound) {
    // '%v' wrapper: same sentinel, match with errors.Is, not string compare
    return // skip; block is gone
}

When it happens

Trigger: loadMeta: f.bkt.ReaderWithExpectedErrs(f.bkt.IsObjNotFoundErr).Get(ctx, metaFile) returns err where f.bkt.IsObjNotFoundErr(err) is true, so the error is re-wrapped with the '%v' format. Callers match it via errors.Is(err, ErrorSyncMetaNotFound).

Common situations: Concurrent compactor/GC deletion during block sync; lifecycle/expiration rules removing objects; another node cleaning up blocks; eventually-consistent stores returning stale Exists results.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• meta.json not found
• get meta file
• read meta file
• filter blocks marked for deletion
• upload of failed

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/44f8514d5f256bb4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/fetcher.go:481

	// Best effort load from local dir.
	if f.cacheDir != "" {
		m, err := metadata.ReadFromDir(cachedBlockDir)
		if err == nil {
			return m, nil
		}

		if !errors.Is(err, os.ErrNotExist) {
			level.Warn(f.logger).Log("msg", "best effort read of the local meta.json failed; removing cached block dir", "dir", cachedBlockDir, "err", err)
			if err := os.RemoveAll(cachedBlockDir); err != nil {
				level.Warn(f.logger).Log("msg", "best effort remove of cached dir failed; ignoring", "dir", cachedBlockDir, "err", err)
			}
		}
	}

	r, err := f.bkt.ReaderWithExpectedErrs(f.bkt.IsObjNotFoundErr).Get(ctx, metaFile)
	if f.bkt.IsObjNotFoundErr(err) {
		// Meta.json was deleted between bkt.Exists and here.
		return nil, errors.Wrapf(ErrorSyncMetaNotFound, "%v", err)
	}
	if err != nil {
		return nil, errors.Wrapf(err, "get meta file: %v", metaFile)
	}

	defer runutil.CloseWithLogOnErr(f.logger, r, "close bkt meta get")

	metaContent, err := io.ReadAll(r)
	if err != nil {
		return nil, errors.Wrapf(err, "read meta file: %v", metaFile)
	}

	m := &metadata.Meta{}
	if err := json.Unmarshal(metaContent, m); err != nil {
		return nil, errors.Wrapf(ErrorSyncMetaCorrupted, "meta.json %v unmarshal: %v", metaFile, err)
	}

	if m.Version != metadata.TSDBVersion1 {

View on GitHub (pinned to 35b8b99117)