{"record":{"id":"6b085c39eb116272","repo":"risingwavelabs/risingwave","slug":"too-many-inflight-time-travel-queries-max-infligh","errorCode":null,"errorMessage":"too many inflight time travel queries, max_inflight_time_travel_query={}","messagePattern":"too many inflight time travel queries, max_inflight_time_travel_query=(.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/meta/src/hummock/manager/time_travel.rs","lineNumber":546,"sourceCode":"        let count = hummock_sstable_info::Entity::find()\n            .count(&self.env.meta_store_ref().conn)\n            .await?;\n        Ok(count)\n    }\n\n    /// Attempt to locate the version corresponding to `query_epoch`.\n    ///\n    /// The version is retrieved from `hummock_epoch_to_version`, selecting the entry with the largest epoch that's lte `query_epoch`.\n    ///\n    /// The resulted version is complete, i.e. with correct `SstableInfo`.\n    pub async fn epoch_to_version(\n        &self,\n        query_epoch: HummockEpoch,\n        table_id: TableId,\n    ) -> Result<HummockVersion> {\n        let sql_store = self.env.meta_store_ref();\n        let _permit = self.inflight_time_travel_query.try_acquire().map_err(|_| {\n            anyhow!(format!(\n                \"too many inflight time travel queries, max_inflight_time_travel_query={}\",\n                self.env.opts.max_inflight_time_travel_query\n            ))\n        })?;\n        let epoch_to_version = hummock_epoch_to_version::Entity::find()\n            .filter(\n                Condition::any()\n                    .add(\n                        hummock_epoch_to_version::Column::TableId\n                            .eq(i64::from(table_id.as_raw_id())),\n                    )\n                    // for backward compatibility\n                    .add(hummock_epoch_to_version::Column::TableId.eq(0)),\n            )\n            .filter(\n                hummock_epoch_to_version::Column::Epoch\n                    .lte(risingwave_meta_model::Epoch::try_from(query_epoch).unwrap()),\n            )","sourceCodeStart":528,"sourceCodeEnd":564,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/hummock/manager/time_travel.rs#L528-L564","documentation":"epoch_to_version bounds concurrent time-travel point-in-time queries with a semaphore sized by max_inflight_time_travel_query. When the semaphore has no free permits, acquiring fails and this error is returned instead of queuing, protecting the meta node from overload. It is a backpressure signal, not a data problem.","triggerScenarios":"Issuing many concurrent time-travel queries (e.g. many flashback / time-travel reads of a table at historical epochs) such that inflight queries exceed max_inflight_time_travel_query; permits leak-free but long-running queries hold them.","commonSituations":"Burst of time-travel queries from monitoring or user flashback requests; max_inflight_time_travel_query left at a low default while workload concurrency is high; slow meta store making queries hold permits longer.","solutions":["Increase max_inflight_time_travel_query in the meta node config to match expected concurrent time-travel load.","Retry the query with backoff after in-flight queries complete — the error is transient.","Rate-limit time-travel query submission on the client side.","Tune meta store performance (DB connections, hardware) so queries release permits faster."],"exampleFix":"// before\n[meta]\nmax_inflight_time_travel_query = 10\n\n// after\n[meta]\nmax_inflight_time_travel_query = 100","handlingStrategy":"retry","validationCode":"// client-side throttle: cap concurrent time-travel queries\nlet sem = Arc::new(Semaphore::new(max_inflight_time_travel_query as usize));\nlet _permit = sem.acquire().await.unwrap(); // queue locally before RPC","typeGuard":null,"tryCatchPattern":"// treat acquire-rejection as transient backpressure\nmatch meta.epoch_to_version(epoch, table_id).await {\n    Err(e) if e.to_string().contains(\"too many inflight time travel queries\") => {\n        tokio::time::sleep(backoff).await; // retry with exponential backoff\n    }\n    other => other,\n}","preventionTips":["Size max_inflight_time_travel_query above peak concurrent time-travel demand.","Rate-limit or queue time-travel queries at the client/application layer.","Alert on sustained semaphore saturation to know when to raise the limit.","Keep the meta store fast (indexes, connection pool) so permits are released quickly."],"tags":["time-travel","hummock","rate-limit","concurrency","risingwave-meta"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}