influxdata/influxdb · error · TableIndexCacheError

Failed to update table index: join task failed

Error message

Failed to update table index: join task failed

What it means

TableIndexCacheError::UpdateTaskFailed, thrown when the tokio task spawned to update a cached TableIndex panics or is cancelled before finishing (tokio::task::JoinError). The cache refreshes indexes in background tasks so reads are not blocked; a failed join means the refresh never completed and the cache holds stale data.

Solutions

1. Check the inner JoinError: is_cancelled() vs is_panic(); for panics inspect the panic payload and the task's logs/backtrace to find the root cause
2. Fix the panic source in the update logic (often an unwrap/expect on missing data or a store object in unexpected shape)
3. Avoid aborting index update tasks during shutdown; drain them before stopping the runtime
4. Retrigger the index update or invalidate the cache entry so the next access reloads from the object store

Example fix

// before: propagate JoinError directly
handle.await??;
// after: distinguish cancel vs panic and retry
match handle.await {
    Ok(Ok(idx)) => insert(idx),
    Ok(Err(e)) => return Err(e.into()),
    Err(join_err) if join_err.is_cancelled() => warn!("index update cancelled"),
    Err(join_err) => error!(panic=?join_err.into_panic(), "index update task panicked"),
}

// before spawning, ensure the runtime is shutting down cleanly
if tokio::runtime::Handle::try_current()
    .map(|h| h.shutdown_timeout(Duration::ZERO).is_ok())
    .is_err() { /* drain tasks before exit */ }

fn join_ok<T>(r: Result<Result<T, TableIndexCacheError>, tokio::task::JoinError>)
    -> Option<Result<T, TableIndexCacheError>> {
    match r {
        Ok(res) => Some(res),
        Err(j) => {
            tracing::error!(cancelled = j.is_cancelled(), "index update task died");
            None
        }
    }
}

match handle.await {
    Err(join_err) if join_err.is_panic() => {
        let payload = join_err.into_panic();
        error!("index update panicked; invalidating cache entry");
        cache.invalidate(table_id);
    }
    Err(_) => info!("index update cancelled (shutdown)"),
    Ok(res) => res?,
}

When it happens

Trigger: Calling the cache's update path that spawns a task via tokio::spawn / JoinSet and then awaits .join()/.join_next(): the spawned future panics (e.g. bug in TableIndex update, OOM via abort), or the task is aborted on shutdown/timeout.

Common situations: A panic inside index update code (unwrap on missing snapshot, arithmetic overflow); runtime shutdown or task abort during graceful stop; watchdog timeouts cancelling slow index updates on very large tables.

Related errors

• Failed to update table index: join task failed
• join error
• Join error, please report
• request trigger execution cancelled
• Table snapshot persistence task failed

AI-assisted analysis of influxdata/influxdb@06200ef96b (2026-09-19). Data as JSON: /api/errors/2a09fc58ac314a78. Report an issue: GitHub.

Appendix: source

Thrown at influxdb3_write/src/table_index_cache.rs:75

    #[error("Unexpected error")]
    Unexpected(#[source] anyhow::Error),

    #[error("Table snapshot persistence task failed")]
    TableSnapshotPersistenceTaskFailed(#[source] tokio::task::JoinError),

    #[error("Failed to list table indices from object store")]
    ListIndices(#[source] object_store::Error),

    #[error("Failed to parse table index path from object store path")]
    TableIndexPath(#[source] crate::paths::PathError),

    #[error("Failed to list table index snapshots from object store")]
    ListSnapshots(#[source] object_store::Error),

    #[error("Failed to parse table index snapshot path from object store path")]
    TableIndexSnapshotPath(#[source] crate::paths::PathError),

    #[error("Failed to update table index: join task failed")]
    UpdateTaskFailed(#[source] tokio::task::JoinError),

    #[error("Failed to list object store metadata")]
    ListMetasError(#[source] object_store::Error),

    #[error("Failed to load table index conversion marker bytes from object store")]
    LoadConversionMarkerBytesError(#[source] object_store::Error),

    #[error("Failed to serialize initial table index conversion marker")]
    SerializeConversionMarkerError(#[source] serde_json::Error),

    #[error("Failed to put initial table index conversion marker to object store")]
    PutConversionMarkerError(#[source] object_store::Error),

    #[error("Failed to parse snapshot info file path")]
    ParseSnapshotPathError(#[source] anyhow::Error),

    #[error("Failed to split persisted snapshot into table snapshots")]

View on GitHub (pinned to 06200ef96b)