{"record":{"id":"d71deddbd29f85c8","repo":"jaegertracing/jaeger","slug":"unmarshalling-documents-failed-w","errorCode":null,"errorMessage":"unmarshalling documents failed: %w","messagePattern":"unmarshalling documents failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/v1/elasticsearch/samplingstore/storage.go","lineNumber":98,"sourceCode":"\t// context.Background: sampling writes are not request-scoped, and the async\n\t// indexer this uses ignores the context anyway (see BulkIndexer.WriteBatch).\n\treturn s.batchWriter.WriteBatch(context.Background(), items)\n}\n\nfunc (s *SamplingStore) GetThroughput(start, end time.Time) ([]*model.Throughput, error) {\n\tctx := context.Background()\n\treadIndices := s.rotation.ReadTargets(start, end)\n\tsearchResult, err := s.searcher.Search(ctx, readIndices, esclient.SearchRequest{\n\t\tSize:  s.maxDocCount,\n\t\tQuery: buildTSQuery(start, end),\n\t})\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to search for throughputs: %w\", err)\n\t}\n\toutput := make([]dbmodel.TimeThroughput, len(searchResult.Hits.Hits))\n\tfor i, hit := range searchResult.Hits.Hits {\n\t\tif err := json.Unmarshal(hit.Source, &output[i]); err != nil {\n\t\t\treturn nil, fmt.Errorf(\"unmarshalling documents failed: %w\", err)\n\t\t}\n\t}\n\toutThroughputs := make([]dbmodel.Throughput, len(output))\n\tfor i, out := range output {\n\t\toutThroughputs[i] = out.Throughput\n\t}\n\treturn dbmodel.ToThroughputs(outThroughputs), nil\n}\n\nfunc (s *SamplingStore) InsertProbabilitiesAndQPS(hostname string,\n\tprobabilities model.ServiceOperationProbabilities,\n\tqps model.ServiceOperationQPS,\n) error {\n\tts := s.now()\n\twriteIndexName := s.rotation.WriteTarget(ts)\n\tval := dbmodel.ProbabilitiesAndQPS{\n\t\tHostname:      hostname,\n\t\tProbabilities: probabilities,","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v1/elasticsearch/samplingstore/storage.go#L80-L116","documentation":"After a successful GetThroughput search, each hit's _source JSON is unmarshalled into dbmodel.TimeThroughput; if a stored document does not match the expected schema, json.Unmarshal fails and the error is wrapped with 'unmarshalling documents failed'. This indicates corrupt or incompatible data at rest, not a transport problem.","triggerScenarios":"Calling GetThroughput when at least one hit document's _source lacks the expected fields/types (e.g. missing 'throughput' array, wrong field types) — typically documents written by an incompatible Jaeger version or by external tooling.","commonSituations":"Upgrading/downgrading Jaeger across a dbmodel change for sampling documents; manually reindexed or externally written documents in the throughput indices; partially written documents from an earlier failed ingestion pipeline.","solutions":["Inspect the offending hit's _source in Elasticsearch and compare with dbmodel.TimeThroughput's JSON tags.","Delete/re-ingest the incompatible documents (re-run the sampling/probability ingestion that populates them).","Align Jaeger versions: documents written by a much older/newer schema may need migration or index recreation.","If indices are corrupt beyond repair, drop the throughput indices and let the collector rewrite them."],"exampleFix":null,"handlingStrategy":"type-guard","validationCode":"var doc struct {\n    Throughput []dbmodel.Throughput `json:\"throughput\"`\n}\nif len(hit.Source) == 0 || json.Unmarshal(hit.Source, &doc) != nil || doc.Throughput == nil {\n    // skip or quarantine this document before passing to storage layer\n}","typeGuard":"func validThroughputDoc(src []byte) bool {\n    var d struct {\n        Throughput []json.RawMessage `json:\"throughput\"`\n    }\n    return json.Unmarshal(src, &d) == nil && len(d.Throughput) > 0\n}","tryCatchPattern":"tp, err := store.GetThroughput(ctx, start, end)\nif err != nil {\n    if strings.Contains(err.Error(), \"unmarshalling documents failed\") {\n        // inspect/reindex the offending documents in the throughput indices\n    }\n    return err\n}","preventionTips":["Write sampling documents only through the matching Jaeger collector version.","Avoid hand-editing or external writes to throughput indices.","Pin Jaeger versions across collector and query so dbmodel stays compatible.","Validate one stored document's schema after schema upgrades."],"tags":["elasticsearch","sampling","json","deserialization"],"backgroundTag":"json-unmarshal-failed","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}