jaegertracing/jaeger · error

unmarshalling documents failed: %w

Error message

unmarshalling documents failed: %w

What it means

After a successful GetThroughput search, each hit's _source JSON is unmarshalled into dbmodel.TimeThroughput; if a stored document does not match the expected schema, json.Unmarshal fails and the error is wrapped with 'unmarshalling documents failed'. This indicates corrupt or incompatible data at rest, not a transport problem.

Source

Thrown at internal/storage/v1/elasticsearch/samplingstore/storage.go:98

	// context.Background: sampling writes are not request-scoped, and the async
	// indexer this uses ignores the context anyway (see BulkIndexer.WriteBatch).
	return s.batchWriter.WriteBatch(context.Background(), items)
}

func (s *SamplingStore) GetThroughput(start, end time.Time) ([]*model.Throughput, error) {
	ctx := context.Background()
	readIndices := s.rotation.ReadTargets(start, end)
	searchResult, err := s.searcher.Search(ctx, readIndices, esclient.SearchRequest{
		Size:  s.maxDocCount,
		Query: buildTSQuery(start, end),
	})
	if err != nil {
		return nil, fmt.Errorf("failed to search for throughputs: %w", err)
	}
	output := make([]dbmodel.TimeThroughput, len(searchResult.Hits.Hits))
	for i, hit := range searchResult.Hits.Hits {
		if err := json.Unmarshal(hit.Source, &output[i]); err != nil {
			return nil, fmt.Errorf("unmarshalling documents failed: %w", err)
		}
	}
	outThroughputs := make([]dbmodel.Throughput, len(output))
	for i, out := range output {
		outThroughputs[i] = out.Throughput
	}
	return dbmodel.ToThroughputs(outThroughputs), nil
}

func (s *SamplingStore) InsertProbabilitiesAndQPS(hostname string,
	probabilities model.ServiceOperationProbabilities,
	qps model.ServiceOperationQPS,
) error {
	ts := s.now()
	writeIndexName := s.rotation.WriteTarget(ts)
	val := dbmodel.ProbabilitiesAndQPS{
		Hostname:      hostname,
		Probabilities: probabilities,

View on GitHub (pinned to 806f444784)

Solutions

1. Inspect the offending hit's _source in Elasticsearch and compare with dbmodel.TimeThroughput's JSON tags.
2. Delete/re-ingest the incompatible documents (re-run the sampling/probability ingestion that populates them).
3. Align Jaeger versions: documents written by a much older/newer schema may need migration or index recreation.
4. If indices are corrupt beyond repair, drop the throughput indices and let the collector rewrite them.

var doc struct {
    Throughput []dbmodel.Throughput `json:"throughput"`
}
if len(hit.Source) == 0 || json.Unmarshal(hit.Source, &doc) != nil || doc.Throughput == nil {
    // skip or quarantine this document before passing to storage layer
}

func validThroughputDoc(src []byte) bool {
    var d struct {
        Throughput []json.RawMessage `json:"throughput"`
    }
    return json.Unmarshal(src, &d) == nil && len(d.Throughput) > 0
}

tp, err := store.GetThroughput(ctx, start, end)
if err != nil {
    if strings.Contains(err.Error(), "unmarshalling documents failed") {
        // inspect/reindex the offending documents in the throughput indices
    }
    return err
}

When it happens

Trigger: Calling GetThroughput when at least one hit document's _source lacks the expected fields/types (e.g. missing 'throughput' array, wrong field types) — typically documents written by an incompatible Jaeger version or by external tooling.

Common situations: Upgrading/downgrading Jaeger across a dbmodel change for sampling documents; manually reindexed or externally written documents in the throughput indices; partially written documents from an earlier failed ingestion pipeline.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• unmarshalling ElasticSearch documents failed
• failed to find latest index
• failed to parse bulk response: %w
• failed to search for throughputs: %w
• failed to get latest index: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/d71deddbd29f85c8. Report an issue: GitHub.