too many inflight time travel queries, max_inflight_time_tra

Error message

too many inflight time travel queries, max_inflight_time_travel_query={}

What it means

epoch_to_version bounds concurrent time-travel point-in-time queries with a semaphore sized by max_inflight_time_travel_query. When the semaphore has no free permits, acquiring fails and this error is returned instead of queuing, protecting the meta node from overload. It is a backpressure signal, not a data problem.

Source

Thrown at src/meta/src/hummock/manager/time_travel.rs:546

        let count = hummock_sstable_info::Entity::find()
            .count(&self.env.meta_store_ref().conn)
            .await?;
        Ok(count)
    }

    /// Attempt to locate the version corresponding to `query_epoch`.
    ///
    /// The version is retrieved from `hummock_epoch_to_version`, selecting the entry with the largest epoch that's lte `query_epoch`.
    ///
    /// The resulted version is complete, i.e. with correct `SstableInfo`.
    pub async fn epoch_to_version(
        &self,
        query_epoch: HummockEpoch,
        table_id: TableId,
    ) -> Result<HummockVersion> {
        let sql_store = self.env.meta_store_ref();
        let _permit = self.inflight_time_travel_query.try_acquire().map_err(|_| {
            anyhow!(format!(
                "too many inflight time travel queries, max_inflight_time_travel_query={}",
                self.env.opts.max_inflight_time_travel_query
            ))
        })?;
        let epoch_to_version = hummock_epoch_to_version::Entity::find()
            .filter(
                Condition::any()
                    .add(
                        hummock_epoch_to_version::Column::TableId
                            .eq(i64::from(table_id.as_raw_id())),
                    )
                    // for backward compatibility
                    .add(hummock_epoch_to_version::Column::TableId.eq(0)),
            )
            .filter(
                hummock_epoch_to_version::Column::Epoch
                    .lte(risingwave_meta_model::Epoch::try_from(query_epoch).unwrap()),
            )

View on GitHub (pinned to 6469eb736d)

Solutions

Increase max_inflight_time_travel_query in the meta node config to match expected concurrent time-travel load.
Retry the query with backoff after in-flight queries complete — the error is transient.
Rate-limit time-travel query submission on the client side.
Tune meta store performance (DB connections, hardware) so queries release permits faster.

Example fix

// before
[meta]
max_inflight_time_travel_query = 10

// after
[meta]
max_inflight_time_travel_query = 100

Defensive patterns

Strategy: retry

Validate before calling

// client-side throttle: cap concurrent time-travel queries
let sem = Arc::new(Semaphore::new(max_inflight_time_travel_query as usize));
let _permit = sem.acquire().await.unwrap(); // queue locally before RPC

Try / catch

// treat acquire-rejection as transient backpressure
match meta.epoch_to_version(epoch, table_id).await {
    Err(e) if e.to_string().contains("too many inflight time travel queries") => {
        tokio::time::sleep(backoff).await; // retry with exponential backoff
    }
    other => other,
}

Prevention

Size max_inflight_time_travel_query above peak concurrent time-travel demand.
Rate-limit or queue time-travel queries at the client/application layer.
Alert on sustained semaphore saturation to know when to raise the limit.
Keep the meta store fast (indexes, connection pool) so permits are released quickly.

When it happens

Trigger: Issuing many concurrent time-travel queries (e.g. many flashback / time-travel reads of a table at historical epochs) such that inflight queries exceed max_inflight_time_travel_query; permits leak-free but long-running queries hold them.

Common situations: Burst of time-travel queries from monitoring or user flashback requests; max_inflight_time_travel_query left at a low default while workload concurrency is high; slow meta store making queries hold permits longer.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/6b085c39eb116272. Report an issue: GitHub.