{"record":{"id":"0f92fc170ab68c18","repo":"jaegertracing/jaeger","slug":"unmarshalling-elasticsearch-documents-failed","errorCode":null,"errorMessage":"unmarshalling ElasticSearch documents failed","messagePattern":"unmarshalling ElasticSearch documents failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/v2/elasticsearch/depstore/storage.go","lineNumber":88,"sourceCode":"\t}})\n}\n\n// GetDependencies returns all interservice dependencies\nfunc (s *DependencyStore) GetDependencies(ctx context.Context, endTs time.Time, lookback time.Duration) ([]dbmodel.DependencyLink, error) {\n\treadIndices := s.rotation.ReadTargets(endTs.Add(-lookback), endTs)\n\tsearchResult, err := s.searcher.Search(ctx, readIndices, esclient.SearchRequest{\n\t\tSize:  s.maxDocCount,\n\t\tQuery: buildTSQuery(endTs, lookback),\n\t})\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to search for dependencies: %w\", err)\n\t}\n\n\tvar retDependencies []dbmodel.DependencyLink\n\tfor _, hit := range searchResult.Hits.Hits {\n\t\tvar tToD dbmodel.TimeDependencies\n\t\tif err := json.Unmarshal(hit.Source, &tToD); err != nil {\n\t\t\treturn nil, errors.New(\"unmarshalling ElasticSearch documents failed\")\n\t\t}\n\t\tretDependencies = append(retDependencies, tToD.Dependencies...)\n\t}\n\treturn retDependencies, nil\n}\n\nfunc buildTSQuery(endTs time.Time, lookback time.Duration) query.Query {\n\treturn query.NewRangeQuery(\"timestamp\").Gte(endTs.Add(-lookback)).Lte(endTs)\n}\n","sourceCodeStart":70,"sourceCodeEnd":98,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v2/elasticsearch/depstore/storage.go#L70-L98","documentation":"GetDependencies in the Elasticsearch dependency store returns this error when a search hit's _source document cannot be JSON-unmarshalled into dbmodel.TimeDependencies. It replaces the underlying unmarshal error with a generic message, so the raw cause is not propagated.","triggerScenarios":"A stored dependency document's source bytes are not valid JSON for dbmodel.TimeDependencies (schema drift between Jaeger versions, index created by v1 with different mapping, corrupted documents, or hand-inserted documents).","commonSituations":"Upgrading Jaeger v1 indexes read by a v2 reader; manual reindexing that changed field names; partial writes or documents from another tool sharing the index.","solutions":["Inspect the affected Elasticsearch documents in the jaeger-dependencies index and fix or delete malformed ones","Rebuild the dependencies index from trace data using the Spark/remote dependency job","Check for schema mismatches between the writer that produced the documents and this reader (dbmodel.TimeDependencies shape)","Log hit.Source for the failing hit to see the actual malformed payload, since this error hides the underlying cause"],"exampleFix":"// before\nif err := json.Unmarshal(hit.Source, &tToD); err != nil {\n    return nil, errors.New(\"unmarshalling ElasticSearch documents failed\")\n}\n// after (caller-side diagnosis)\ndeps, err := reader.GetDependencies(ctx, query)\nif err != nil && strings.Contains(err.Error(), \"unmarshalling ElasticSearch documents failed\") {\n    // inspect/reindex the dependency index; the underlying JSON is malformed\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"func looksLikeTimeDependencies(src []byte) bool {\n    var probe struct { Dependencies []dbmodel.DependencyLink `json:\"dependencies\"` }\n    return json.Unmarshal(src, &probe) == nil\n}","tryCatchPattern":"deps, err := reader.GetDependencies(ctx, query)\nif err != nil && strings.Contains(err.Error(), \"unmarshalling ElasticSearch documents failed\") {\n    // query/reindex the jaeger-dependencies index; some hit._source is not valid TimeDependencies JSON\n}","preventionTips":["Keep writer and reader Jaeger schema versions aligned","Monitor/rebuild the dependencies index after upgrades","Log hit.Source at write/reindex time to catch malformed documents early"],"tags":["elasticsearch","json","deserialization","jaeger"],"backgroundTag":"json-unmarshal-failed","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}