jaegertracing/jaeger · error

error reading throughput from storage: %w

Error message

error reading throughput from storage: %w

What it means

GetThroughput queries Cassandra for throughput strings and wraps any error returned when the gocql iterator is closed (iter.Close() flushes paging and surfaces deferred read errors). The %w wrap means the underlying Cassandra driver error (timeout, coordinator failure, schema disagreement) is preserved for errors.Is/As inspection. It indicates the read did not complete cleanly, not that data was absent.

Source

Thrown at internal/storage/v1/cassandra/samplingstore/storage.go:87

}

// InsertThroughput implements samplingstore.Writer#InsertThroughput.
func (s *SamplingStore) InsertThroughput(throughput []*model.Throughput) error {
	throughputStr := throughputToString(throughput)
	query := s.session.Query(insertThroughput, generateRandomBucket(), gocql.TimeUUID(), throughputStr)
	return s.metrics.operationThroughput.Exec(query, s.logger)
}

// GetThroughput implements samplingstore.Reader#GetThroughput.
func (s *SamplingStore) GetThroughput(start, end time.Time) ([]*model.Throughput, error) {
	iter := s.session.Query(getThroughput, gocql.UUIDFromTime(start), gocql.UUIDFromTime(end)).Iter()
	var throughput []*model.Throughput
	var throughputStr string
	for iter.Scan(&throughputStr) {
		throughput = append(throughput, s.stringToThroughput(throughputStr)...)
	}
	if err := iter.Close(); err != nil {
		err = fmt.Errorf("error reading throughput from storage: %w", err)
		return nil, err
	}
	return throughput, nil
}

// InsertProbabilitiesAndQPS implements samplingstore.Writer#InsertProbabilitiesAndQPS.
func (s *SamplingStore) InsertProbabilitiesAndQPS(
	hostname string,
	probabilities model.ServiceOperationProbabilities,
	qps model.ServiceOperationQPS,
) error {
	probabilitiesAndQPSStr := probabilitiesAndQPSToString(probabilities, qps)
	query := s.session.Query(insertProbabilities, constBucket, gocql.TimeUUID(), hostname, probabilitiesAndQPSStr)
	return s.metrics.probabilities.Exec(query, s.logger)
}

// GetLatestProbabilities implements samplingstore.Reader#GetLatestProbabilities.
func (s *SamplingStore) GetLatestProbabilities() (model.ServiceOperationProbabilities, error) {

View on GitHub (pinned to 806f444784)

Solutions

1. Unwrap with errors.As to inspect the underlying gocql error (e.g. *gocql.RequestError) and check the Cassandra cluster/node health at the reported error time.
2. Increase read timeout / consistency tuning in the gocql cluster config if timeouts recur under load.
3. Retry GetThroughput with backoff for transient node failures; run nodetool repair if tombstone-heavy partitions cause read timeouts.
4. Verify all nodes agree on schema (nodetool describecluster) after recent migrations.

Example fix

// before
throughput, err := store.GetThroughput("svc", "op")
if err != nil {
    return err // opaque wrapped message
}
// after
throughput, err := store.GetThroughput("svc", "op")
if err != nil {
    var reqErr *gocql.RequestError
    if errors.As(err, &reqErr) && reqErr.Code() == gocql.ErrCodeTimeout {
        return retryWithBackoff()
    }
    return err
}

// Go
throughput, err := store.GetThroughput(service, operation)
if err != nil {
    var reqErr interface{ Code() int }
    if errors.As(err, &reqErr) {
        // transient driver-level failure: retry with backoff
    }
    return fmt.Errorf("throughput unavailable: %w", err)
}

When it happens

Trigger: Calling SamplingStore.GetThroughput(service, operation) when the Cassandra cluster returns an error during result paging, e.g. connection dropped mid-scan, read timeout on the throughput table, or schema disagreement after a migration.

Common situations: Cassandra node restarts or network flakiness during query paging; read timeouts under heavy load (tombstone-heavy partitions); rolling schema migrations where coordinator nodes disagree on the table definition.

Related errors

• error reading probabilities from storage: %w
• failed to Exec query '%s': %w
• invalid version
• failed to acquire resource lock due to cassandra error: %w
• unknown column for position: %q

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/e2eeaf6b0250798e. Report an issue: GitHub.