thanos-io/thanos · error

read meta file

Error message

read meta file: %v

What it means

loadMeta wraps an io.ReadAll failure on the opened meta.json reader as errors.Wrapf(err, "read meta file: %v", metaFile). The Get succeeded but streaming the object body failed mid-read — typically a dropped connection or checksum/IO error while downloading. The error carries the file path and the underlying cause; the block's metadata cannot be loaded and the sync of that block fails.

Solutions

1. Retry the sync — this is usually transient; the fetcher will pick the block up on the next iteration.
2. Check network stability/proxy idle timeouts between Thanos and the object store; raise proxy read timeouts.
3. Verify the object in the bucket is intact (compare content-length/checksum); re-upload if truncated.
4. If persistent, inspect the objstore client logs for the underlying cause and adjust retry/backoff settings.

// pre-flight: confirm object is readable end-to-end
rc, err := bkt.Get(ctx, metaPath)
if err == nil {
    _, err = io.Copy(io.Discard, rc) // surfaces read errors early
    rc.Close()
}

func isReadMetaFailure(err error) bool {
    return err != nil && strings.Contains(err.Error(), "read meta file:")
}

m, err := loadMetaFn(ctx)
if err != nil && strings.Contains(err.Error(), "read meta file:") {
    // transient body-read failure: retry with exponential backoff
    return backoff.Retry(ctx, loadMetaFn, backoff.WithMaxRetries(3))
}

When it happens

Trigger: loadMeta: io.ReadAll(r) after a successful bkt Get returns err (connection reset during body read, S3 content-length mismatch, disk/proxy failure), wrapped with 'read meta file: <metaFile>'.

Common situations: Unstable network or proxy idle timeouts killing long reads; S3/GCS returning truncated bodies; object sizes altered mid-transfer; TLS termination issues; large-scale syncs amplifying transient failures.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• get meta file
• get header from object storage of
• upload of failed
• read meta.json for block
• check exists in bucket

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6da31ef9100dd697. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/fetcher.go:491

				level.Warn(f.logger).Log("msg", "best effort remove of cached dir failed; ignoring", "dir", cachedBlockDir, "err", err)
			}
		}
	}

	r, err := f.bkt.ReaderWithExpectedErrs(f.bkt.IsObjNotFoundErr).Get(ctx, metaFile)
	if f.bkt.IsObjNotFoundErr(err) {
		// Meta.json was deleted between bkt.Exists and here.
		return nil, errors.Wrapf(ErrorSyncMetaNotFound, "%v", err)
	}
	if err != nil {
		return nil, errors.Wrapf(err, "get meta file: %v", metaFile)
	}

	defer runutil.CloseWithLogOnErr(f.logger, r, "close bkt meta get")

	metaContent, err := io.ReadAll(r)
	if err != nil {
		return nil, errors.Wrapf(err, "read meta file: %v", metaFile)
	}

	m := &metadata.Meta{}
	if err := json.Unmarshal(metaContent, m); err != nil {
		return nil, errors.Wrapf(ErrorSyncMetaCorrupted, "meta.json %v unmarshal: %v", metaFile, err)
	}

	if m.Version != metadata.TSDBVersion1 {
		return nil, errors.Errorf("unexpected meta file: %s version: %d", metaFile, m.Version)
	}

	// Best effort cache in local dir.
	if f.cacheDir != "" {
		if err := os.MkdirAll(cachedBlockDir, os.ModePerm); err != nil {
			level.Warn(f.logger).Log("msg", "best effort mkdir of the meta.json block dir failed; ignoring", "dir", cachedBlockDir, "err", err)
		}

		if err := m.WriteToDir(f.logger, cachedBlockDir); err != nil {

View on GitHub (pinned to 35b8b99117)