jaegertracing/jaeger · error

unmarshalling ElasticSearch documents failed

Error message

unmarshalling ElasticSearch documents failed

What it means

GetDependencies in the Elasticsearch dependency store returns this error when a search hit's _source document cannot be JSON-unmarshalled into dbmodel.TimeDependencies. It replaces the underlying unmarshal error with a generic message, so the raw cause is not propagated.

Source

Thrown at internal/storage/v2/elasticsearch/depstore/storage.go:88

	}})
}

// GetDependencies returns all interservice dependencies
func (s *DependencyStore) GetDependencies(ctx context.Context, endTs time.Time, lookback time.Duration) ([]dbmodel.DependencyLink, error) {
	readIndices := s.rotation.ReadTargets(endTs.Add(-lookback), endTs)
	searchResult, err := s.searcher.Search(ctx, readIndices, esclient.SearchRequest{
		Size:  s.maxDocCount,
		Query: buildTSQuery(endTs, lookback),
	})
	if err != nil {
		return nil, fmt.Errorf("failed to search for dependencies: %w", err)
	}

	var retDependencies []dbmodel.DependencyLink
	for _, hit := range searchResult.Hits.Hits {
		var tToD dbmodel.TimeDependencies
		if err := json.Unmarshal(hit.Source, &tToD); err != nil {
			return nil, errors.New("unmarshalling ElasticSearch documents failed")
		}
		retDependencies = append(retDependencies, tToD.Dependencies...)
	}
	return retDependencies, nil
}

func buildTSQuery(endTs time.Time, lookback time.Duration) query.Query {
	return query.NewRangeQuery("timestamp").Gte(endTs.Add(-lookback)).Lte(endTs)
}

View on GitHub (pinned to 806f444784)

Solutions

1. Inspect the affected Elasticsearch documents in the jaeger-dependencies index and fix or delete malformed ones
2. Rebuild the dependencies index from trace data using the Spark/remote dependency job
3. Check for schema mismatches between the writer that produced the documents and this reader (dbmodel.TimeDependencies shape)
4. Log hit.Source for the failing hit to see the actual malformed payload, since this error hides the underlying cause

Example fix

// before
if err := json.Unmarshal(hit.Source, &tToD); err != nil {
    return nil, errors.New("unmarshalling ElasticSearch documents failed")
}
// after (caller-side diagnosis)
deps, err := reader.GetDependencies(ctx, query)
if err != nil && strings.Contains(err.Error(), "unmarshalling ElasticSearch documents failed") {
    // inspect/reindex the dependency index; the underlying JSON is malformed
}

func looksLikeTimeDependencies(src []byte) bool {
    var probe struct { Dependencies []dbmodel.DependencyLink `json:"dependencies"` }
    return json.Unmarshal(src, &probe) == nil
}

deps, err := reader.GetDependencies(ctx, query)
if err != nil && strings.Contains(err.Error(), "unmarshalling ElasticSearch documents failed") {
    // query/reindex the jaeger-dependencies index; some hit._source is not valid TimeDependencies JSON
}

When it happens

Trigger: A stored dependency document's source bytes are not valid JSON for dbmodel.TimeDependencies (schema drift between Jaeger versions, index created by v1 with different mapping, corrupted documents, or hand-inserted documents).

Common situations: Upgrading Jaeger v1 indexes read by a v2 reader; manual reindexing that changed field names; partial writes or documents from another tool sharing the index.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• unmarshalling documents failed: %w
• failed to parse bulk response: %w
• file must begin with '['
• max spans count reached
• empty configuration

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/0f92fc170ab68c18. Report an issue: GitHub.