thanos-io/thanos · error

failed to read

Error message

failed to read %s

What it means

ReadMetaFile reads the shipper's bookkeeping file thanos.shipper.json from the given path via os.ReadFile. If the file cannot be read (missing, permission denied, I/O error), the error is wrapped with "failed to read <path>". The shipper stores the list of already-uploaded block ULIDs in this file, so callers use it to compute which blocks still need uploading.

Solutions

1. If the data dir is new, treat this as initialization: on first Sync the shipper writes the meta file — ensure the process can create files in the data dir.
2. Check permissions on thanos.shipper.json and its directory; chown/chmod so the shipper UID can read it.
3. Restore the file if it was deleted (or accept re-upload of blocks — safe, as uploads are idempotent — by letting the shipper recreate it).
4. Verify the mount is not read-only and the path passed to the shipper is correct (the error includes the full path).

Example fix

// before: assuming meta file always exists
meta, _ := shipper.ReadMetaFile(path)
// after: tolerate a missing file on first run
meta, err := shipper.ReadMetaFile(path)
if os.IsNotExist(errors.Cause(err)) {
	meta = &shipper.Meta{Version: shipper.MetaVersion1}
}

metaPath := filepath.Join(dataDir, "thanos.shipper.json")
if _, err := os.Stat(metaPath); os.IsNotExist(err) {
	level.Info(logger).Log("msg", "no shipper meta yet, first run")
}

meta, err := shipper.ReadMetaFile(metaPath)
if err != nil {
	if os.IsNotExist(errors.Cause(err)) {
		meta = &shipper.Meta{Version: shipper.MetaVersion1}
	} else {
		return errors.Wrap(err, "read shipper meta")
	}
}

When it happens

Trigger: Calling UploadedBlocks(), Sync(), or AreAllBlocksUploaded() (and the anonymous iterator wrapper) when <data-dir>/thanos.shipper.json does not exist yet or is unreadable.

Common situations: First run against a fresh data directory where the meta file was never written; file deleted by an operator or cleanup job; permission change on the data dir; read-only mount after an incident recovery; running the process as a different UID than the file owner.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• read meta of
• could not sync metas
• write meta
• unable to access path
• unable to open tbc directory

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e98d3466a2cd396e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:653

		runutil.CloseWithLogOnErr(logger, f, "write meta file close")
		return err
	}

	// Force the kernel to persist the file on disk to avoid data loss if the host crashes.
	if err := f.Sync(); err != nil {
		return err
	}
	if err := f.Close(); err != nil {
		return err
	}
	return renameFile(logger, tmp, path)
}

// ReadMetaFile reads the given meta from <dir>/thanos.shipper.json.
func ReadMetaFile(path string) (*Meta, error) {
	b, err := os.ReadFile(path)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to read %s", path)
	}

	var m Meta
	if err := json.Unmarshal(b, &m); err != nil {
		return nil, errors.Wrapf(err, "failed to parse %s as JSON: %q", path, string(b))
	}
	if m.Version != MetaVersion1 {
		return nil, errors.Errorf("unexpected meta file version %d", m.Version)
	}

	return &m, nil
}

func renameFile(logger log.Logger, from, to string) error {
	if err := os.RemoveAll(to); err != nil {
		return err
	}
	if err := os.Rename(from, to); err != nil {

View on GitHub (pinned to 35b8b99117)