{"record":{"id":"6619b4d45f0fd5d7","repo":"influxdata/influxdb","slug":"object-store-operation-failed","errorCode":null,"errorMessage":"Object store operation failed","messagePattern":"Object store operation failed","errorType":"exception","errorClass":"TableIndexError","httpStatus":null,"severity":"critical","filePath":"influxdb3_write/src/table_index.rs","lineNumber":76,"sourceCode":"    #[error(\"Failed to parse table index snapshot path from object store path\")]\n    TableIndexSnapshotPath(#[source] crate::paths::PathError),\n\n    #[error(\"Failed to update table index: join task failed\")]\n    UpdateTaskFailed(#[source] tokio::task::JoinError),\n\n    #[error(\"Object meta is missing filename\")]\n    MissingFilename,\n\n    #[error(\"Failed to parse snapshot sequence number from filename\")]\n    InvalidSnapshotSequenceNumber,\n\n    #[error(\"Table snapshot persistence task failed\")]\n    TableSnapshotPersistenceTaskFailed(#[source] tokio::task::JoinError),\n\n    #[error(\"Unexpected error\")]\n    Unexpected(#[source] anyhow::Error),\n\n    #[error(\"Object store operation failed\")]\n    ObjectStore(#[source] object_store::Error),\n\n    #[error(\"JSON serialization/deserialization failed\")]\n    Json(#[source] serde_json::Error),\n}\n\npub type Result<T> = std::result::Result<T, TableIndexError>;\n\n/// An partial, incremental index into a given database/table. Should not be used to make decisions\n/// regarding gen1 file retention -- instead, all snapshots should be aggregated into a full\n/// CoreTableIndex.\n///\n/// TableIndexSnapshots are expected to be created during the normal snapshotting process alongside\n/// the full process-wide database + table index.\n#[derive(Debug, Serialize, Deserialize, Eq, PartialEq, Clone)]\npub struct TableIndexSnapshot {\n    #[serde(flatten)]\n    pub id: TableIndexId,","sourceCodeStart":58,"sourceCodeEnd":94,"githubUrl":"https://github.com/influxdata/influxdb/blob/06200ef96ba82c5f6727e5038a83af8e722c6875/influxdb3_write/src/table_index.rs#L58-L94","documentation":"TableIndexError::ObjectStore wraps an object_store::Error raised while reading or writing table index snapshot files. In influxdb3_write it is produced by TableIndexSnapshot::persist (put_with_default_retries) and TableIndexSnapshot::load_split_persist (get_with_default_retries and body reads). It means the underlying object store (S3, GCS, Azure, local file) rejected or failed the I/O operation, not that the index data itself is wrong.","triggerScenarios":"TableIndexSnapshot::persist fails its put_with_default_retries call; TableIndexSnapshot::load_split_persist fails get_with_default_retries or reading the response bytes during PersistedSnapshot -> TableIndexSnapshot conversion at startup; the object store returns any non-NotFound error (auth, throttling, network, permissions).","commonSituations":"Misconfigured object store credentials or endpoint, S3 throttling (slow down / 503) after retries are exhausted, network partitions between the InfluxDB 3 node and the bucket, missing bucket/container, or read-only IAM policies on the snapshot prefix.","solutions":["Inspect the #[source] object_store::Error for the concrete cause (NotFound, credentials, throttling) and fix that root issue first.","Verify the object store URL, bucket/container name, region, and credentials supplied to the InfluxDB 3 server configuration.","Check IAM/bucket policy grants read+write on the database/table index prefix used by this node.","Confirm network connectivity and DNS to the object store endpoint; for persistent throttling raise retry limits or reduce concurrency.","If NotFound during conversion, the referenced PersistedSnapshot may have been deleted; verify snapshot retention/cleanup settings."],"exampleFix":"// before: object store URL wrong region\nINFLUXDB3_OBJECT_STORE=s3 INFLUXDB3_BUCKET=my-bucket\n// after: correct region + explicit endpoint\nINFLUXDB3_OBJECT_STORE=s3 INFLUXDB3_BUCKET=my-bucket INFLUXDB3_AWS_DEFAULT_REGION=us-east-1","handlingStrategy":"retry","validationCode":"// verify store reachable before conversion\nobject_store.head(&probe_path).await.map_err(|e| {\n    if matches!(e, object_store::Error::NotFound { .. }) { Ok(()) } else { Err(e) }\n})?;","typeGuard":"fn is_transient_store_err(e: &object_store::Error) -> bool {\n    matches!(e, object_store::Error::Generic { .. }) // classify throttling/network errors for retry\n}","tryCatchPattern":"match TableIndexSnapshot::persist(store, ).await {\n    Err(TableIndexError::ObjectStore(e)) if is_retryable(&e) => backoff_retry().await,\n    Err(TableIndexError::ObjectStore(e)) => error!(\"store misconfigured: {e}\"),\n    other => other?,\n}","preventionTips":["Pre-flight check credentials and bucket reachability at server startup","Use a RetryableObjectStore with capped exponential backoff for all index I/O","Alert on object store 5xx/throttling metrics before index writes start failing","Keep IAM policies granting read+write on the index prefix"],"tags":["object-store","storage","rust","io"],"backgroundTag":"network-request-failed","analyzedSha":"06200ef96ba82c5f6727e5038a83af8e722c6875","analyzedAt":"2026-09-19T12:55:30.003Z","contentChangedAt":"2026-09-19T12:55:30.003Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}