{"record":{"id":"4268615600787bf7","repo":"jaegertracing/jaeger","slug":"error-reading-probabilities-from-storage-w","errorCode":null,"errorMessage":"error reading probabilities from storage: %w","messagePattern":"error reading probabilities from storage: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/v1/cassandra/samplingstore/storage.go","lineNumber":110,"sourceCode":"\n// InsertProbabilitiesAndQPS implements samplingstore.Writer#InsertProbabilitiesAndQPS.\nfunc (s *SamplingStore) InsertProbabilitiesAndQPS(\n\thostname string,\n\tprobabilities model.ServiceOperationProbabilities,\n\tqps model.ServiceOperationQPS,\n) error {\n\tprobabilitiesAndQPSStr := probabilitiesAndQPSToString(probabilities, qps)\n\tquery := s.session.Query(insertProbabilities, constBucket, gocql.TimeUUID(), hostname, probabilitiesAndQPSStr)\n\treturn s.metrics.probabilities.Exec(query, s.logger)\n}\n\n// GetLatestProbabilities implements samplingstore.Reader#GetLatestProbabilities.\nfunc (s *SamplingStore) GetLatestProbabilities() (model.ServiceOperationProbabilities, error) {\n\titer := s.session.Query(getLatestProbabilities).Iter()\n\tvar probabilitiesStr string\n\titer.Scan(&probabilitiesStr)\n\tif err := iter.Close(); err != nil {\n\t\terr = fmt.Errorf(\"error reading probabilities from storage: %w\", err)\n\t\treturn nil, err\n\t}\n\treturn s.stringToProbabilities(probabilitiesStr), nil\n}\n\n// This is random enough for storage purposes\nfunc generateRandomBucket() int64 {\n\treturn time.Now().UnixNano() % 10\n}\n\nfunc probabilitiesAndQPSToString(probabilities model.ServiceOperationProbabilities, qps model.ServiceOperationQPS) string {\n\tvar buf bytes.Buffer\n\twriter := csv.NewWriter(&buf)\n\tfor svc, opProbabilities := range probabilities {\n\t\tfor op, probability := range opProbabilities {\n\t\t\topQPS := 0.0\n\t\t\tif _, ok := qps[svc]; ok {\n\t\t\t\topQPS = qps[svc][op]","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v1/cassandra/samplingstore/storage.go#L92-L128","documentation":"GetLatestProbabilities reads the latest sampling probabilities string and wraps the error from closing the gocql iterator, preserving the driver error via %w. It signals the query against the sampling probabilities table failed at the driver/cluster level (timeout, unavailable coordinator, broken connection). Note the code scans only one row, so a multi-row table silently yields only the first.","triggerScenarios":"Calling SamplingStore.GetLatestProbabilities() when iter.Close() reports a deferred query error: node unavailable mid-page, read timeout, or the client connection was closed before iteration finished.","commonSituations":"Short-lived processes closing the session before the query fully drains; Cassandra consistency/availability problems; cluster under pressure causing coordinator read timeouts on the probabilities table.","solutions":["Unwrap with errors.As/errors.Is to identify the gocql driver error and check Cassandra node availability and logs for that timestamp.","Keep the gocql session alive for the process lifetime; do not close it while queries may still be iterating.","Add retry with backoff for transient unavailability; tune read timeouts/consistency if timeouts recur.","Confirm schema agreement across nodes after any DDL change to the probabilities table."],"exampleFix":"// before\nif err != nil {\n    log.Fatalf(\"probabilities read failed: %v\", err)\n}\n// after\nif err != nil {\n    if errors.Is(err, gocql.ErrUnavailable) {\n        return retryGetLatestProbabilities(3)\n    }\n    return fmt.Errorf(\"probabilities read failed: %w\", err)\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Go\nprobs, err := store.GetLatestProbabilities()\nif err != nil {\n    if errors.Is(err, gocql.ErrUnavailable) || errors.Is(err, gocql.ErrTimeout) {\n        return retryWithBackoff(func() (model.ServiceOperationProbabilities, error) {\n            return store.GetLatestProbabilities()\n        })\n    }\n    return nil, err\n}","preventionTips":["Do not close the Cassandra session while queries may still iterate.","Add retry-with-backoff around sampling storage reads.","Alert on Cassandra coordinator timeouts and unavailability.","Verify schema agreement after every migration."],"tags":["cassandra","storage","gocql","error-wrapping"],"backgroundTag":"cassandra-read-failure","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}