What causes "unmarshal marker JSON" in thanos-io/thanos?

Treat as a transient/corruption signal and skip the block, as the fetcher does (with a warning); re-attempt later.

unmarshal marker JSON

Error message

unmarshal marker JSON

What it means

ErrorUnmarshalMarker is the sentinel error returned by metadata.ReadMarker when the marker JSON file exists but cannot be unmarshalled into the requested marker struct. Per its doc comment, this typically happens because the marker was only partially uploaded to the object storage, or the file is not valid JSON.

Solutions

Treat as a transient/corruption signal and skip the block, as the fetcher does (with a warning); re-attempt later.
If it recurs for the same block, delete and re-upload the marker file in the object storage.
Inspect the marker file content in the bucket to confirm whether it is truncated or invalid JSON, and rewrite it.

Example fix

// before
if err := metadata.ReadMarker(ctx, logger, bkt, id.String(), m); err != nil {
	return err
}
// after
if err := metadata.ReadMarker(ctx, logger, bkt, id.String(), m); err != nil {
	if errors.Cause(err) == metadata.ErrorUnmarshalMarker {
		level.Warn(logger).Log("msg", "partial marker file", "block", id, "err", err)
		continue
	}
	return err
}

Defensive patterns

Strategy: retry

Try / catch

if errors.Cause(err) == metadata.ErrorUnmarshalMarker {
	level.Warn(logger).Log("msg", "partial marker file; retry later or delete it", "block", id, "err", err)
	return nil // or retry with backoff
}

Prevention

Use atomic uploads (upload to temp key then copy/rename) when writing marker files.
Monitor repeated unmarshal failures for the same block — it signals a permanently corrupt marker to delete.
Avoid manually editing marker JSON in the bucket; regenerate it with tooling that validates JSON.

When it happens

Trigger: ReadMarker reads deletion-mark.json (or another marker) whose bytes are truncated/corrupt — e.g. an interrupted or partially-uploaded marker file — so json.Unmarshal fails and ReadMarker wraps this sentinel with the file name and underlying message.

Common situations: Object storage multipart uploads interrupted mid-write; lifecycle/proxy mangling marker content; manually edited marker files; network truncation when downloading the marker.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f82b447b6c272d9a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/metadata/markers.go:44

	NoCompactMarkFilename = "no-compact-mark.json"
	// NoDownsampleMarkFilename is the known json filenanme for optional file storing details about why block has to be excluded from downsampling.
	// If such file is present in block dir, it means the block has to be excluded from downsampling.
	NoDownsampleMarkFilename = "no-downsample-mark.json"
	// DeletionMarkVersion1 is the version of deletion-mark file supported by Thanos.
	DeletionMarkVersion1 = 1
	// NoCompactMarkVersion1 is the version of no-compact-mark file supported by Thanos.
	NoCompactMarkVersion1 = 1
	// NoDownsampleVersion1 is the version of no-downsample-mark file supported by Thanos.
	NoDownsampleMarkVersion1 = 1
)

var (
	// ErrorMarkerNotFound is the error when marker file is not found.
	ErrorMarkerNotFound = errors.New("marker not found")
	// ErrorUnmarshalMarker is the error when unmarshalling marker JSON file.
	// This error can occur because marker has been partially uploaded to block storage
	// or the marker file is not a valid json file.
	ErrorUnmarshalMarker = errors.New("unmarshal marker JSON")
)

type Marker interface {
	markerFilename() string
}

// DeletionMark stores block id and when block was marked for deletion.
type DeletionMark struct {
	// ID of the tsdb block.
	ID ulid.ULID `json:"id"`
	// Version of the file.
	Version int `json:"version"`
	// Details is a human readable string giving details of reason.
	Details string `json:"details,omitempty"`

	// DeletionTime is a unix timestamp of when the block was marked to be deleted.
	DeletionTime int64 `json:"deletion_time"`
}

View on GitHub (pinned to 35b8b99117)