{"record":{"id":"0580253dc33bc98e","repo":"influxdata/influxdb","slug":"object-store-operation-failed-table-index-cache","errorCode":null,"errorMessage":"Object store operation failed","messagePattern":"Object store operation failed","errorType":"exception","errorClass":"TableIndexCacheError","httpStatus":null,"severity":"critical","filePath":"influxdb3_write/src/table_index_cache.rs","lineNumber":42,"sourceCode":"};\n\nuse influxdb3_id::{DbId, TableId, TableIndexId};\nuse influxdb3_wal::SnapshotSequenceNumber;\n\nuse crate::{\n    ParquetFile,\n    paths::{\n        SnapshotInfoFilePath, TableIndexConversionCompletedPath, TableIndexPath,\n        TableIndexSnapshotPath,\n    },\n    table_index::{CoreTableIndex, IndexMetadata, TableIndex, TableIndexSnapshot},\n};\n\nuse thiserror::Error;\n\n#[derive(Debug, Error)]\npub enum TableIndexCacheError {\n    #[error(\"Object store operation failed\")]\n    ObjectStore(#[source] object_store::Error),\n\n    #[error(\"JSON serialization/deserialization failed\")]\n    Json(#[source] serde_json::Error),\n\n    #[error(\"TableIndex operation failed\")]\n    TableIndex(#[source] crate::table_index::TableIndexError),\n\n    #[error(\"Object meta is missing filename\")]\n    MissingFilename,\n\n    #[error(\"Failed to parse snapshot sequence number from filename\")]\n    InvalidSnapshotSequenceNumber,\n\n    #[error(\"Unexpected error\")]\n    Unexpected(#[source] anyhow::Error),\n\n    #[error(\"Table snapshot persistence task failed\")]","sourceCodeStart":24,"sourceCodeEnd":60,"githubUrl":"https://github.com/influxdata/influxdb/blob/06200ef96ba82c5f6727e5038a83af8e722c6875/influxdb3_write/src/table_index_cache.rs#L24-L60","documentation":"TableIndexCacheError::ObjectStore wraps an object_store::Error raised while the TableIndexCache performs object store I/O outside its more specific variants (list/get/put/delete helpers). It indicates the backing store failed an operation while loading, updating, or persisting table index data through the LRU cache.","triggerScenarios":"Calls through TableIndexCache (get_or_load, update_from_object_store, initialize, purge paths) hit a RetryableObjectStore operation that returns a non-NotFound error after default retries.","commonSituations":"Transient S3/GCS outages, expired cloud credentials mid-run, hitting request-rate limits during concurrent cache warm-up, or a bucket that was deleted or renamed while the server was running.","solutions":["Read the wrapped #[source] object_store::Error to identify the concrete failure (auth, throttling, NotFound) and fix that first.","Verify object store credentials are valid and not expired (check IAM session/token lifetime).","Confirm the configured bucket/container still exists and is reachable from the node.","For rate limiting, lower TableIndexCacheConfig::concurrency_limit or enable/raise retry backoff.","If transient, re-run TableIndexCache::initialize or restart the node once the store is healthy."],"exampleFix":"// before: default concurrency overwhelming the store\nTableIndexCacheConfig::default()\n// after: reduced concurrency\nTableIndexCacheConfig { concurrency_limit: 5, ..Default::default() }","handlingStrategy":"retry","validationCode":"// cheap liveness probe of the object store before cache-heavy operations\nlet _ = object_store.head(&probe_path).await.or_else(|e| match e {\n    object_store::Error::NotFound { .. } => Ok(()),\n    e => Err(e),\n})?;","typeGuard":"fn is_store_unavailable(e: &TableIndexCacheError) -> bool {\n    matches!(e, TableIndexCacheError::ObjectStore(_))\n}","tryCatchPattern":"match cache.get_or_load(&table_id).await {\n    Err(TableIndexCacheError::ObjectStore(e)) if retryable(&e) => {\n        tokio::time::sleep(backoff).await;\n        cache.get_or_load(&table_id).await?\n    }\n    r => r?,\n}","preventionTips":["Right-size TableIndexCacheConfig::concurrency_limit for your store's rate limits","Use short-lived credential renewal (instance profiles) to avoid mid-run expiry","Monitor bucket availability and error rates","Pin bucket/container names in config and validate them at startup"],"tags":["object-store","storage","rust","cache"],"backgroundTag":"network-request-failed","analyzedSha":"06200ef96ba82c5f6727e5038a83af8e722c6875","analyzedAt":"2026-09-19T12:55:30.003Z","contentChangedAt":"2026-09-19T12:55:30.003Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}