jaegertracing/jaeger · error

error reading probabilities from storage: %w

Error message

error reading probabilities from storage: %w

What it means

GetLatestProbabilities reads the latest sampling probabilities string and wraps the error from closing the gocql iterator, preserving the driver error via %w. It signals the query against the sampling probabilities table failed at the driver/cluster level (timeout, unavailable coordinator, broken connection). Note the code scans only one row, so a multi-row table silently yields only the first.

Source

Thrown at internal/storage/v1/cassandra/samplingstore/storage.go:110

// InsertProbabilitiesAndQPS implements samplingstore.Writer#InsertProbabilitiesAndQPS.
func (s *SamplingStore) InsertProbabilitiesAndQPS(
	hostname string,
	probabilities model.ServiceOperationProbabilities,
	qps model.ServiceOperationQPS,
) error {
	probabilitiesAndQPSStr := probabilitiesAndQPSToString(probabilities, qps)
	query := s.session.Query(insertProbabilities, constBucket, gocql.TimeUUID(), hostname, probabilitiesAndQPSStr)
	return s.metrics.probabilities.Exec(query, s.logger)
}

// GetLatestProbabilities implements samplingstore.Reader#GetLatestProbabilities.
func (s *SamplingStore) GetLatestProbabilities() (model.ServiceOperationProbabilities, error) {
	iter := s.session.Query(getLatestProbabilities).Iter()
	var probabilitiesStr string
	iter.Scan(&probabilitiesStr)
	if err := iter.Close(); err != nil {
		err = fmt.Errorf("error reading probabilities from storage: %w", err)
		return nil, err
	}
	return s.stringToProbabilities(probabilitiesStr), nil
}

// This is random enough for storage purposes
func generateRandomBucket() int64 {
	return time.Now().UnixNano() % 10
}

func probabilitiesAndQPSToString(probabilities model.ServiceOperationProbabilities, qps model.ServiceOperationQPS) string {
	var buf bytes.Buffer
	writer := csv.NewWriter(&buf)
	for svc, opProbabilities := range probabilities {
		for op, probability := range opProbabilities {
			opQPS := 0.0
			if _, ok := qps[svc]; ok {
				opQPS = qps[svc][op]

View on GitHub (pinned to 806f444784)

Solutions

1. Unwrap with errors.As/errors.Is to identify the gocql driver error and check Cassandra node availability and logs for that timestamp.
2. Keep the gocql session alive for the process lifetime; do not close it while queries may still be iterating.
3. Add retry with backoff for transient unavailability; tune read timeouts/consistency if timeouts recur.
4. Confirm schema agreement across nodes after any DDL change to the probabilities table.

Example fix

// before
if err != nil {
    log.Fatalf("probabilities read failed: %v", err)
}
// after
if err != nil {
    if errors.Is(err, gocql.ErrUnavailable) {
        return retryGetLatestProbabilities(3)
    }
    return fmt.Errorf("probabilities read failed: %w", err)
}

// Go
probs, err := store.GetLatestProbabilities()
if err != nil {
    if errors.Is(err, gocql.ErrUnavailable) || errors.Is(err, gocql.ErrTimeout) {
        return retryWithBackoff(func() (model.ServiceOperationProbabilities, error) {
            return store.GetLatestProbabilities()
        })
    }
    return nil, err
}

When it happens

Trigger: Calling SamplingStore.GetLatestProbabilities() when iter.Close() reports a deferred query error: node unavailable mid-page, read timeout, or the client connection was closed before iteration finished.

Common situations: Short-lived processes closing the session before the query fully drains; Cassandra consistency/availability problems; cluster under pressure causing coordinator read timeouts on the probabilities table.

Related errors

• error reading throughput from storage: %w
• failed to Exec query '%s': %w
• invalid version
• failed to acquire resource lock due to cassandra error: %w
• unknown column for position: %q

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/4268615600787bf7. Report an issue: GitHub.