thanos-io/thanos · info

marker not found

Error message

marker not found

What it means

ErrorMarkerNotFound is the sentinel error returned by metadata.ReadMarker when the marker file (e.g. deletion-mark.json or no-downsample-mark.json) does not exist in the object storage bucket. It is exported so callers can compare with errors.Cause / errors.Is and treat absence as a normal, expected outcome rather than a failure.

Solutions

1. Treat it as a non-error: skip the block (as pkg/block/fetcher.go does with 'continue') instead of failing the operation.
2. Verify the bucket/endpoint configuration points at the storage where the markers actually live.
3. If the marker is expected to exist, re-check its upload (partial upload lifecycle) or regenerate the marker file.

Example fix

// before
if err := metadata.ReadMarker(ctx, logger, bkt, id.String(), m); err != nil {
	return err
}
// after
if err := metadata.ReadMarker(ctx, logger, bkt, id.String(), m); err != nil {
	if errors.Cause(err) == metadata.ErrorMarkerNotFound {
		continue // marker absent: nothing to process
	}
	return err
}

err := metadata.ReadMarker(ctx, logger, bkt, id.String(), mark)
if err != nil {
	if errors.Cause(err) == metadata.ErrorMarkerNotFound {
		// marker absent: normal case, skip block
		return nil
	}
	return err
}

When it happens

Trigger: metadata.ReadMarker is called for a block whose marker file was never written, was deleted, or the block itself does not exist in the bucket; bkt.Get returns an object-not-found error which ReadMarker maps to this sentinel.

Common situations: Fetcher scanning all blocks for deletion marks where most blocks legitimately have no deletion-mark.json; eventual consistency lag right after block upload; wrong bucket/endpoint configuration so expected markers are 'missing'; blocks removed by compaction.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• meta.json bkt get for
• range not found
• unmarshal marker JSON
• create meta fetcher
• create syncer

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e68376ab840d1ff9. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/metadata/markers.go:40

	// If such file is present in block dir, it means the block is meant to be deleted after certain delay.
	DeletionMarkFilename = "deletion-mark.json"
	// NoCompactMarkFilename is the known json filename for optional file storing details about why block has to be excluded from compaction.
	// If such file is present in block dir, it means the block has to excluded from compaction (both vertical and horizontal) or rewrite (e.g deletions).
	NoCompactMarkFilename = "no-compact-mark.json"
	// NoDownsampleMarkFilename is the known json filenanme for optional file storing details about why block has to be excluded from downsampling.
	// If such file is present in block dir, it means the block has to be excluded from downsampling.
	NoDownsampleMarkFilename = "no-downsample-mark.json"
	// DeletionMarkVersion1 is the version of deletion-mark file supported by Thanos.
	DeletionMarkVersion1 = 1
	// NoCompactMarkVersion1 is the version of no-compact-mark file supported by Thanos.
	NoCompactMarkVersion1 = 1
	// NoDownsampleVersion1 is the version of no-downsample-mark file supported by Thanos.
	NoDownsampleMarkVersion1 = 1
)

var (
	// ErrorMarkerNotFound is the error when marker file is not found.
	ErrorMarkerNotFound = errors.New("marker not found")
	// ErrorUnmarshalMarker is the error when unmarshalling marker JSON file.
	// This error can occur because marker has been partially uploaded to block storage
	// or the marker file is not a valid json file.
	ErrorUnmarshalMarker = errors.New("unmarshal marker JSON")
)

type Marker interface {
	markerFilename() string
}

// DeletionMark stores block id and when block was marked for deletion.
type DeletionMark struct {
	// ID of the tsdb block.
	ID ulid.ULID `json:"id"`
	// Version of the file.
	Version int `json:"version"`
	// Details is a human readable string giving details of reason.
	Details string `json:"details,omitempty"`

View on GitHub (pinned to 35b8b99117)