{"record":{"id":"e2eeaf6b0250798e","repo":"jaegertracing/jaeger","slug":"error-reading-throughput-from-storage-w","errorCode":null,"errorMessage":"error reading throughput from storage: %w","messagePattern":"error reading throughput from storage: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/v1/cassandra/samplingstore/storage.go","lineNumber":87,"sourceCode":"}\n\n// InsertThroughput implements samplingstore.Writer#InsertThroughput.\nfunc (s *SamplingStore) InsertThroughput(throughput []*model.Throughput) error {\n\tthroughputStr := throughputToString(throughput)\n\tquery := s.session.Query(insertThroughput, generateRandomBucket(), gocql.TimeUUID(), throughputStr)\n\treturn s.metrics.operationThroughput.Exec(query, s.logger)\n}\n\n// GetThroughput implements samplingstore.Reader#GetThroughput.\nfunc (s *SamplingStore) GetThroughput(start, end time.Time) ([]*model.Throughput, error) {\n\titer := s.session.Query(getThroughput, gocql.UUIDFromTime(start), gocql.UUIDFromTime(end)).Iter()\n\tvar throughput []*model.Throughput\n\tvar throughputStr string\n\tfor iter.Scan(&throughputStr) {\n\t\tthroughput = append(throughput, s.stringToThroughput(throughputStr)...)\n\t}\n\tif err := iter.Close(); err != nil {\n\t\terr = fmt.Errorf(\"error reading throughput from storage: %w\", err)\n\t\treturn nil, err\n\t}\n\treturn throughput, nil\n}\n\n// InsertProbabilitiesAndQPS implements samplingstore.Writer#InsertProbabilitiesAndQPS.\nfunc (s *SamplingStore) InsertProbabilitiesAndQPS(\n\thostname string,\n\tprobabilities model.ServiceOperationProbabilities,\n\tqps model.ServiceOperationQPS,\n) error {\n\tprobabilitiesAndQPSStr := probabilitiesAndQPSToString(probabilities, qps)\n\tquery := s.session.Query(insertProbabilities, constBucket, gocql.TimeUUID(), hostname, probabilitiesAndQPSStr)\n\treturn s.metrics.probabilities.Exec(query, s.logger)\n}\n\n// GetLatestProbabilities implements samplingstore.Reader#GetLatestProbabilities.\nfunc (s *SamplingStore) GetLatestProbabilities() (model.ServiceOperationProbabilities, error) {","sourceCodeStart":69,"sourceCodeEnd":105,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v1/cassandra/samplingstore/storage.go#L69-L105","documentation":"GetThroughput queries Cassandra for throughput strings and wraps any error returned when the gocql iterator is closed (iter.Close() flushes paging and surfaces deferred read errors). The %w wrap means the underlying Cassandra driver error (timeout, coordinator failure, schema disagreement) is preserved for errors.Is/As inspection. It indicates the read did not complete cleanly, not that data was absent.","triggerScenarios":"Calling SamplingStore.GetThroughput(service, operation) when the Cassandra cluster returns an error during result paging, e.g. connection dropped mid-scan, read timeout on the throughput table, or schema disagreement after a migration.","commonSituations":"Cassandra node restarts or network flakiness during query paging; read timeouts under heavy load (tombstone-heavy partitions); rolling schema migrations where coordinator nodes disagree on the table definition.","solutions":["Unwrap with errors.As to inspect the underlying gocql error (e.g. *gocql.RequestError) and check the Cassandra cluster/node health at the reported error time.","Increase read timeout / consistency tuning in the gocql cluster config if timeouts recur under load.","Retry GetThroughput with backoff for transient node failures; run nodetool repair if tombstone-heavy partitions cause read timeouts.","Verify all nodes agree on schema (nodetool describecluster) after recent migrations."],"exampleFix":"// before\nthroughput, err := store.GetThroughput(\"svc\", \"op\")\nif err != nil {\n    return err // opaque wrapped message\n}\n// after\nthroughput, err := store.GetThroughput(\"svc\", \"op\")\nif err != nil {\n    var reqErr *gocql.RequestError\n    if errors.As(err, &reqErr) && reqErr.Code() == gocql.ErrCodeTimeout {\n        return retryWithBackoff()\n    }\n    return err\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Go\nthroughput, err := store.GetThroughput(service, operation)\nif err != nil {\n    var reqErr interface{ Code() int }\n    if errors.As(err, &reqErr) {\n        // transient driver-level failure: retry with backoff\n    }\n    return fmt.Errorf(\"throughput unavailable: %w\", err)\n}","preventionTips":["Keep the gocql session long-lived and healthy (ping/health checks).","Tune read timeouts and consistency for the throughput table workload.","Monitor Cassandra node availability and schema agreement.","Run periodic repairs to avoid tombstone-driven read timeouts."],"tags":["cassandra","storage","gocql","error-wrapping"],"backgroundTag":"cassandra-read-failure","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}