thanos-io/thanos · error

read meta.json for block

Error message

read meta.json for block %s

What it means

DownloadMeta in pkg/block/block.go fetches meta.json for a block from object storage and reads it fully. This error wraps any failure of io.ReadAll on the bucket reader — the HTTP/objstore read failed mid-stream (connection reset, timeout, cancellation). The ULID in the message identifies which block's meta.json could not be downloaded.

Solutions

1. Retry the download; this is usually a transient network failure.
2. Verify ctx is not cancelled/timed out before or during the download.
3. Check object storage connectivity, credentials, and network egress from the pod/host.
4. Check object store server-side logs for aborted requests.

Example fix

// before
obj, err := io.ReadAll(rc)
if err != nil {
	return metadata.Meta{}, errors.Wrapf(err, "read meta.json for block %s", id.String())
}
// after
obj, err := io.ReadAll(rc)
if err != nil {
	if ctx.Err() != nil {
		return metadata.Meta{}, errors.Wrapf(ctx.Err(), "context done while reading meta.json for block %s", id.String())
	}
	return metadata.Meta{}, errors.Wrapf(err, "read meta.json for block %s", id.String())
}

// caller side
if err := ctx.Err(); err != nil {
	return fmt.Errorf("context already cancelled before DownloadMeta(%s): %w", id, err)
}
if exists, _ := bkt.Exists(ctx, path.Join(id.String(), metadata.MetaFilename)); !exists {
	return fmt.Errorf("meta.json for block %s does not exist in bucket", id)
}

func hasCtx(ctx context.Context) bool { return ctx != nil && ctx.Err() == nil }

m, err := block.DownloadMeta(ctx, logger, bkt, id)
if err != nil {
	if ctx.Err() != nil { return err } // cancellation, do not retry
	// transient read failure: retry with backoff
	return retry(ctx, func() error { _, err = block.DownloadMeta(ctx, logger, bkt, id); return err })
}

When it happens

Trigger: bkt.Get(ctx, id/meta.json) succeeded in opening the reader but io.ReadAll(rc) returned an error — network interruption to object storage, context cancellation/deadline exceeded during transfer, or the object backend closing the connection prematurely.

Common situations: S3/GCS transient network blips; slow connection hitting the store-gateway or compactor request timeout; context cancelled by parent operation while downloading; misconfigured bucket causing stale/aborted HTTP connections.

Related errors

• check exists in bucket
• get meta file
• read meta file
• get TOC from object storage of
• get header from object storage of

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/d241a4b6f3bed2dd. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/block.go:291

		level.Debug(logger).Log("msg", "deleted file", "file", name, "bucket", bkt.Name())
		return nil
	})
}

// DownloadMeta downloads only meta file from bucket by block ID.
// TODO(bwplotka): Differentiate between network error & partial upload.
func DownloadMeta(ctx context.Context, logger log.Logger, bkt objstore.Bucket, id ulid.ULID) (metadata.Meta, error) {
	rc, err := bkt.Get(ctx, path.Join(id.String(), MetaFilename))
	if err != nil {
		return metadata.Meta{}, errors.Wrapf(err, "meta.json bkt get for %s", id.String())
	}
	defer runutil.CloseWithLogOnErr(logger, rc, "download meta bucket client")

	var m metadata.Meta

	obj, err := io.ReadAll(rc)
	if err != nil {
		return metadata.Meta{}, errors.Wrapf(err, "read meta.json for block %s", id.String())
	}

	if err = json.Unmarshal(obj, &m); err != nil {
		return metadata.Meta{}, errors.Wrapf(err, "unmarshal meta.json for block %s", id.String())
	}

	return m, nil
}

func IsBlockMetaFile(path string) bool {
	return filepath.Base(path) == MetaFilename
}

func IsBlockDir(path string) (id ulid.ULID, ok bool) {
	id, err := ulid.Parse(filepath.Base(path))
	return id, err == nil
}

View on GitHub (pinned to 35b8b99117)