What causes "meta.json bkt get for %s" in thanos-io/thanos?

Check object storage connectivity and permissions

meta.json bkt get for

Error message

meta.json bkt get for %s

What it means

Wraps bkt.Get in DownloadMeta when fetching a block's meta.json from object storage. It fires on bucket access errors (auth, network, missing object) while reading the block's metadata; partial uploads are not yet distinguished from network errors at this site.

Solutions

Check object storage connectivity and permissions
If not found, the block may be partial — let cleanup handle it
Retry after transient errors

Example fix

// before
meta, err := block.DownloadMeta(ctx, logger, bkt, id)
if err != nil { return err }
// after
meta, err := block.DownloadMeta(ctx, logger, bkt, id)
if err != nil {
    if bkt.IsObjNotFoundErr(errors.Unwrap(err)) {
        logger.Warn("block disappeared from bucket, skipping", "id", id)
        return nil
    }
    return err
}

Defensive patterns

Strategy: try-catch

Validate before calling

ok, err := bkt.Exists(ctx, path.Join(id.String(), block.MetaFilename))
if err != nil { return err }
if !ok { return fmt.Errorf("block %s not present in bucket", id) }

Try / catch

meta, err := block.DownloadMeta(ctx, logger, bkt, id)
if err != nil {
    if bkt.IsObjNotFoundErr(errors.Unwrap(err)) { return errSkipMissingBlock }
    return err
}

Prevention

Use bkt.IsObjNotFoundErr to distinguish missing blocks from real failures
Skip/re-list blocks that vanished during GC rather than failing hard
Confirm ULIDs come from a fresh bucket listing, not stale caches

When it happens

Trigger: Calling block.DownloadMeta for a ULID whose meta.json is not in the bucket (partial upload, GC'd block, wrong ULID) or when the object-store Get request fails (auth/network/throttling).

Common situations: Store gateway or tooling resolving a block ID that was garbage-collected; listing race during compaction; typo'd or stale ULID; object store credentials/bucket misconfiguration.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/c7f5f6a96ccce508. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/block.go:283

			return deleteDirRec(ctx, logger, bkt, name, keep)
		}
		if keep(name) {
			return nil
		}
		if err := bkt.Delete(ctx, name); err != nil {
			return err
		}
		level.Debug(logger).Log("msg", "deleted file", "file", name, "bucket", bkt.Name())
		return nil
	})
}

// DownloadMeta downloads only meta file from bucket by block ID.
// TODO(bwplotka): Differentiate between network error & partial upload.
func DownloadMeta(ctx context.Context, logger log.Logger, bkt objstore.Bucket, id ulid.ULID) (metadata.Meta, error) {
	rc, err := bkt.Get(ctx, path.Join(id.String(), MetaFilename))
	if err != nil {
		return metadata.Meta{}, errors.Wrapf(err, "meta.json bkt get for %s", id.String())
	}
	defer runutil.CloseWithLogOnErr(logger, rc, "download meta bucket client")

	var m metadata.Meta

	obj, err := io.ReadAll(rc)
	if err != nil {
		return metadata.Meta{}, errors.Wrapf(err, "read meta.json for block %s", id.String())
	}

	if err = json.Unmarshal(obj, &m); err != nil {
		return metadata.Meta{}, errors.Wrapf(err, "unmarshal meta.json for block %s", id.String())
	}

	return m, nil
}

func IsBlockMetaFile(path string) bool {

View on GitHub (pinned to 35b8b99117)