jaegertracing/jaeger · critical

internal data consistency issue

Error message

internal data consistency issue

What it means

ErrInternalConsistencyError signals that the badger store's internal data violated an invariant. It is returned by badger TraceReader.GetTrace when the primary-key lookup for a single trace ID yields more than one distinct trace — which should be impossible — indicating corrupted or duplicated index/PK entries. It is an internal-condition error, not something a caller's request parameters cause.

Source

Thrown at internal/storage/v1/badger/spanstore/reader.go:49

	ErrStartTimeMinGreaterThanMax = errors.New("min start time is above max")

	// ErrDurationMinGreaterThanMax occurs when duration min is above duration max
	ErrDurationMinGreaterThanMax = errors.New("min duration is above max")

	// ErrMalformedRequestObject occurs when a request object is nil
	ErrMalformedRequestObject = errors.New("malformed request object")

	// ErrStartAndEndTimeNotSet occurs when start time and end time are not set
	ErrStartAndEndTimeNotSet = errors.New("start and end time must be set")

	// ErrUnableToFindTraceIDAggregation occurs when an aggregation query for TraceIDs fail.
	ErrUnableToFindTraceIDAggregation = errors.New("could not find aggregation of traceIDs")

	// ErrNotSupported during development, don't support every option - yet
	ErrNotSupported = errors.New("this query parameter is not supported yet")

	// ErrInternalConsistencyError indicates internal data consistency issue
	ErrInternalConsistencyError = errors.New("internal data consistency issue")
)

const (
	defaultNumTraces = 100
	sizeOfTraceID    = 16
	encodingTypeBits = 0x0F
)

// TraceReader reads traces from the local badger store
type TraceReader struct {
	store *badger.DB
	cache *CacheStore
}

// executionPlan is internal structure to track the index filtering
type executionPlan struct {
	startTimeMin []byte
	startTimeMax []byte

View on GitHub (pinned to 806f444784)

Solutions

1. Stop writers and run badger's integrity/verification tooling (badger info; iterate keys for the trace ID prefix) to inspect duplicates.
2. Restore the badger data directory from a known-good backup, or drop and rebuild the store if traces are expendable.
3. Ensure only one process opens the badger directory and that the Jaeger version writing the data matches the reader's encoding expectations.
4. If reproducible, file an issue with the trace ID and store state; treat the storage backend as unhealthy and fail requests.

func isInternalConsistencyFailure(err error) bool {
	return errors.Is(err, spanstore.ErrInternalConsistencyError)
}

trace, err := reader.GetTrace(ctx, q)
if errors.Is(err, spanstore.ErrInternalConsistencyError) {
	log.Error("badger store inconsistent: multiple traces for one ID", "traceID", q.TraceID)
	alertOnDataCorruption(ctx)
	return nil, httpError(500, "storage backend unhealthy") // do not retry blindly
}
if err != nil { return nil, err }

When it happens

Trigger: GetTrace(ctx, GetTraceParameters{TraceID: id}) where getTraces returns len(traces) > 1 for one ID: duplicate primary-key entries from a corrupted database file, a crash mid-write without proper sync, or manual key manipulation/sidecar writes racing the reader.

Common situations: Running badger storage on a disk that suffered partial corruption; reopening a badger directory concurrently from two processes; mixing JSON- and proto-encoded writes from mismatched Jaeger versions against the same data directory; killed process leaving inconsistent index cache entries.

Related errors

• service name must be set
• min start time is above max
• min duration is above max
• malformed request object
• start and end time must be set

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/5dfbab2427ffbea5. Report an issue: GitHub.