{"record":{"id":"5dfbab2427ffbea5","repo":"jaegertracing/jaeger","slug":"internal-data-consistency-issue","errorCode":null,"errorMessage":"internal data consistency issue","messagePattern":"internal data consistency issue","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"internal/storage/v1/badger/spanstore/reader.go","lineNumber":49,"sourceCode":"\tErrStartTimeMinGreaterThanMax = errors.New(\"min start time is above max\")\n\n\t// ErrDurationMinGreaterThanMax occurs when duration min is above duration max\n\tErrDurationMinGreaterThanMax = errors.New(\"min duration is above max\")\n\n\t// ErrMalformedRequestObject occurs when a request object is nil\n\tErrMalformedRequestObject = errors.New(\"malformed request object\")\n\n\t// ErrStartAndEndTimeNotSet occurs when start time and end time are not set\n\tErrStartAndEndTimeNotSet = errors.New(\"start and end time must be set\")\n\n\t// ErrUnableToFindTraceIDAggregation occurs when an aggregation query for TraceIDs fail.\n\tErrUnableToFindTraceIDAggregation = errors.New(\"could not find aggregation of traceIDs\")\n\n\t// ErrNotSupported during development, don't support every option - yet\n\tErrNotSupported = errors.New(\"this query parameter is not supported yet\")\n\n\t// ErrInternalConsistencyError indicates internal data consistency issue\n\tErrInternalConsistencyError = errors.New(\"internal data consistency issue\")\n)\n\nconst (\n\tdefaultNumTraces = 100\n\tsizeOfTraceID    = 16\n\tencodingTypeBits = 0x0F\n)\n\n// TraceReader reads traces from the local badger store\ntype TraceReader struct {\n\tstore *badger.DB\n\tcache *CacheStore\n}\n\n// executionPlan is internal structure to track the index filtering\ntype executionPlan struct {\n\tstartTimeMin []byte\n\tstartTimeMax []byte","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v1/badger/spanstore/reader.go#L31-L67","documentation":"ErrInternalConsistencyError signals that the badger store's internal data violated an invariant. It is returned by badger TraceReader.GetTrace when the primary-key lookup for a single trace ID yields more than one distinct trace — which should be impossible — indicating corrupted or duplicated index/PK entries. It is an internal-condition error, not something a caller's request parameters cause.","triggerScenarios":"GetTrace(ctx, GetTraceParameters{TraceID: id}) where getTraces returns len(traces) > 1 for one ID: duplicate primary-key entries from a corrupted database file, a crash mid-write without proper sync, or manual key manipulation/sidecar writes racing the reader.","commonSituations":"Running badger storage on a disk that suffered partial corruption; reopening a badger directory concurrently from two processes; mixing JSON- and proto-encoded writes from mismatched Jaeger versions against the same data directory; killed process leaving inconsistent index cache entries.","solutions":["Stop writers and run badger's integrity/verification tooling (badger info; iterate keys for the trace ID prefix) to inspect duplicates.","Restore the badger data directory from a known-good backup, or drop and rebuild the store if traces are expendable.","Ensure only one process opens the badger directory and that the Jaeger version writing the data matches the reader's encoding expectations.","If reproducible, file an issue with the trace ID and store state; treat the storage backend as unhealthy and fail requests."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":"func isInternalConsistencyFailure(err error) bool {\n\treturn errors.Is(err, spanstore.ErrInternalConsistencyError)\n}","tryCatchPattern":"trace, err := reader.GetTrace(ctx, q)\nif errors.Is(err, spanstore.ErrInternalConsistencyError) {\n\tlog.Error(\"badger store inconsistent: multiple traces for one ID\", \"traceID\", q.TraceID)\n\talertOnDataCorruption(ctx)\n\treturn nil, httpError(500, \"storage backend unhealthy\") // do not retry blindly\n}\nif err != nil { return nil, err }","preventionTips":["Ensure only one process opens the badger data directory at a time.","Keep writer and reader on the same Jaeger version and encoding (JSON vs proto) for a data directory.","Back up the badger directory and monitor disk health where the store lives.","Alert on ErrInternalConsistencyError — it indicates corruption requiring repair/restore, not a retry loop.","Run badger's GC/verification maintenance on a schedule."],"tags":["go","badger","data-corruption","consistency"],"backgroundTag":"data-corruption-detected","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}