thanos-io/thanos · error

file: ; err

Error message

file: %s; err: %v

What it means

ReadMarker in pkg/block/metadata/markers.go reads a marker file (no-compact-mark, no-downsample-mark, deletion-mark) from a block metadata directory and unmarshals its JSON into the marker struct. This error is raised when the marker file exists but its bytes are not valid JSON for the marker type, wrapping ErrorUnmarshalMarker with the file path and the underlying json error. M3DB throws it so operators know a metadata marker file is corrupt or was written by an incompatible format.

Solutions

1. Inspect the marker file's contents (cat the path shown in the error) to confirm whether the JSON is valid.
2. Delete the corrupted marker file if the operation it guards (e.g. deletion or compaction skip) can be safely re-applied, then re-run the operation.
3. Restore the marker file from a backup or replicate it from a healthy node in the cluster.
4. If this recurs, check disk health and fsync behavior on the node; a failing disk can corrupt metadata writes.

Example fix

// before: blindly retrying ReadMarker on a corrupt file
err := metadata.ReadMarker(filePath, marker) // keeps failing

// after: validate then recreate the marker
if err := metadata.ReadMarker(filePath, marker); err != nil {
    if errors.Is(err, metadata.ErrorUnmarshalMarker) {
        os.Remove(filePath) // drop corrupt marker, re-create via WriteMarker
    }
}

data, err := os.ReadFile(markerFile)
if err == nil && !json.Valid(data) {
    // treat as corrupt marker before calling ReadMarker
}

func isMarkerJSONValid(path string) bool {
    b, err := os.ReadFile(path)
    return err == nil && json.Valid(b)
}

if err := metadata.ReadMarker(path, marker); err != nil {
    if errors.Is(err, metadata.ErrorUnmarshalMarker) {
        // corrupt marker: quarantine/delete file and recreate
        os.Remove(path)
    }
    return err
}

When it happens

Trigger: Calling ReadMarker on a marker file whose contents cannot be json.Unmarshal-ed into the expected marker struct: truncated file (crash during WriteMarker), manually edited marker file with invalid JSON, empty file, or a file written with a different schema/field types than the marker struct expects.

Common situations: Disk full or node crash left a partially written no-compact-mark or deletion-mark file; an operator hand-edited the marker; backup/restore copied a corrupted marker; a version mismatch introduced fields the struct cannot decode (e.g. string where int expected).

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• read meta
• meta.json corrupted
• unexpected no-compact-mark file version
• unexpected no-downsample-mark file version
• could not sync metas

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/56c33bf92b7fce72. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/metadata/markers.go:136

// ReadMarker reads the given mark file from <dir>/<marker filename>.json in bucket.
func ReadMarker(ctx context.Context, logger log.Logger, bkt objstore.InstrumentedBucketReader, dir string, marker Marker) error {
	markerFile := path.Join(dir, marker.markerFilename())
	r, err := bkt.ReaderWithExpectedErrs(bkt.IsObjNotFoundErr).Get(ctx, markerFile)
	if err != nil {
		if bkt.IsObjNotFoundErr(err) {
			return ErrorMarkerNotFound
		}
		return errors.Wrapf(err, "get file: %s", markerFile)
	}
	defer runutil.CloseWithLogOnErr(logger, r, "close bkt marker reader")

	metaContent, err := io.ReadAll(r)
	if err != nil {
		return errors.Wrapf(err, "read file: %s", markerFile)
	}

	if err := json.Unmarshal(metaContent, marker); err != nil {
		return errors.Wrapf(ErrorUnmarshalMarker, "file: %s; err: %v", markerFile, err.Error())
	}
	switch marker.markerFilename() {
	case NoCompactMarkFilename:
		if version := marker.(*NoCompactMark).Version; version != NoCompactMarkVersion1 {
			return errors.Errorf("unexpected no-compact-mark file version %d, expected %d", version, NoCompactMarkVersion1)
		}
	case NoDownsampleMarkFilename:
		if version := marker.(*NoDownsampleMark).Version; version != NoDownsampleMarkVersion1 {
			return errors.Errorf("unexpected no-downsample-mark file version %d, expected %d", version, NoDownsampleMarkVersion1)
		}
	case DeletionMarkFilename:
		if version := marker.(*DeletionMark).Version; version != DeletionMarkVersion1 {
			return errors.Errorf("unexpected deletion-mark file version %d, expected %d", version, DeletionMarkVersion1)
		}
	}
	return nil
}

View on GitHub (pinned to 35b8b99117)