{"record":{"id":"2a09fc58ac314a78","repo":"influxdata/influxdb","slug":"failed-to-update-table-index-join-task-failed-cache","errorCode":null,"errorMessage":"Failed to update table index: join task failed","messagePattern":"Failed to update table index: join task failed","errorType":"exception","errorClass":"TableIndexCacheError","httpStatus":null,"severity":"error","filePath":"influxdb3_write/src/table_index_cache.rs","lineNumber":75,"sourceCode":"    #[error(\"Unexpected error\")]\n    Unexpected(#[source] anyhow::Error),\n\n    #[error(\"Table snapshot persistence task failed\")]\n    TableSnapshotPersistenceTaskFailed(#[source] tokio::task::JoinError),\n\n    #[error(\"Failed to list table indices from object store\")]\n    ListIndices(#[source] object_store::Error),\n\n    #[error(\"Failed to parse table index path from object store path\")]\n    TableIndexPath(#[source] crate::paths::PathError),\n\n    #[error(\"Failed to list table index snapshots from object store\")]\n    ListSnapshots(#[source] object_store::Error),\n\n    #[error(\"Failed to parse table index snapshot path from object store path\")]\n    TableIndexSnapshotPath(#[source] crate::paths::PathError),\n\n    #[error(\"Failed to update table index: join task failed\")]\n    UpdateTaskFailed(#[source] tokio::task::JoinError),\n\n    #[error(\"Failed to list object store metadata\")]\n    ListMetasError(#[source] object_store::Error),\n\n    #[error(\"Failed to load table index conversion marker bytes from object store\")]\n    LoadConversionMarkerBytesError(#[source] object_store::Error),\n\n    #[error(\"Failed to serialize initial table index conversion marker\")]\n    SerializeConversionMarkerError(#[source] serde_json::Error),\n\n    #[error(\"Failed to put initial table index conversion marker to object store\")]\n    PutConversionMarkerError(#[source] object_store::Error),\n\n    #[error(\"Failed to parse snapshot info file path\")]\n    ParseSnapshotPathError(#[source] anyhow::Error),\n\n    #[error(\"Failed to split persisted snapshot into table snapshots\")]","sourceCodeStart":57,"sourceCodeEnd":93,"githubUrl":"https://github.com/influxdata/influxdb/blob/06200ef96ba82c5f6727e5038a83af8e722c6875/influxdb3_write/src/table_index_cache.rs#L57-L93","documentation":"TableIndexCacheError::UpdateTaskFailed, thrown when the tokio task spawned to update a cached TableIndex panics or is cancelled before finishing (tokio::task::JoinError). The cache refreshes indexes in background tasks so reads are not blocked; a failed join means the refresh never completed and the cache holds stale data.","triggerScenarios":"Calling the cache's update path that spawns a task via tokio::spawn / JoinSet and then awaits .join()/.join_next(): the spawned future panics (e.g. bug in TableIndex update, OOM via abort), or the task is aborted on shutdown/timeout.","commonSituations":"A panic inside index update code (unwrap on missing snapshot, arithmetic overflow); runtime shutdown or task abort during graceful stop; watchdog timeouts cancelling slow index updates on very large tables.","solutions":["Check the inner JoinError: is_cancelled() vs is_panic(); for panics inspect the panic payload and the task's logs/backtrace to find the root cause","Fix the panic source in the update logic (often an unwrap/expect on missing data or a store object in unexpected shape)","Avoid aborting index update tasks during shutdown; drain them before stopping the runtime","Retrigger the index update or invalidate the cache entry so the next access reloads from the object store"],"exampleFix":"// before: propagate JoinError directly\nhandle.await??;\n// after: distinguish cancel vs panic and retry\nmatch handle.await {\n    Ok(Ok(idx)) => insert(idx),\n    Ok(Err(e)) => return Err(e.into()),\n    Err(join_err) if join_err.is_cancelled() => warn!(\"index update cancelled\"),\n    Err(join_err) => error!(panic=?join_err.into_panic(), \"index update task panicked\"),\n}","handlingStrategy":"try-catch","validationCode":"// before spawning, ensure the runtime is shutting down cleanly\nif tokio::runtime::Handle::try_current()\n    .map(|h| h.shutdown_timeout(Duration::ZERO).is_ok())\n    .is_err() { /* drain tasks before exit */ }","typeGuard":"fn join_ok<T>(r: Result<Result<T, TableIndexCacheError>, tokio::task::JoinError>)\n    -> Option<Result<T, TableIndexCacheError>> {\n    match r {\n        Ok(res) => Some(res),\n        Err(j) => {\n            tracing::error!(cancelled = j.is_cancelled(), \"index update task died\");\n            None\n        }\n    }\n}","tryCatchPattern":"match handle.await {\n    Err(join_err) if join_err.is_panic() => {\n        let payload = join_err.into_panic();\n        error!(\"index update panicked; invalidating cache entry\");\n        cache.invalidate(table_id);\n    }\n    Err(_) => info!(\"index update cancelled (shutdown)\"),\n    Ok(res) => res?,\n}","preventionTips":["Avoid unwrap/expect in index update tasks; return Result instead","Don't abort index-update tasks on shutdown — drain with a timeout","Add panic hooks/logging (tracing panic handler) around spawned tasks","Watch memory pressure: OOM kills manifest as cancelled tasks"],"tags":["tokio","async","task-panic","influxdb3"],"backgroundTag":"thread-interrupted","analyzedSha":"06200ef96ba82c5f6727e5038a83af8e722c6875","analyzedAt":"2026-09-19T12:55:30.003Z","contentChangedAt":"2026-09-19T12:55:30.003Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}