influxdata/influxdb · critical · TableIndexError

Object store operation failed

Error message

Object store operation failed

What it means

TableIndexError::ObjectStore wraps an object_store::Error raised while reading or writing table index snapshot files. In influxdb3_write it is produced by TableIndexSnapshot::persist (put_with_default_retries) and TableIndexSnapshot::load_split_persist (get_with_default_retries and body reads). It means the underlying object store (S3, GCS, Azure, local file) rejected or failed the I/O operation, not that the index data itself is wrong.

Solutions

1. Inspect the #[source] object_store::Error for the concrete cause (NotFound, credentials, throttling) and fix that root issue first.
2. Verify the object store URL, bucket/container name, region, and credentials supplied to the InfluxDB 3 server configuration.
3. Check IAM/bucket policy grants read+write on the database/table index prefix used by this node.
4. Confirm network connectivity and DNS to the object store endpoint; for persistent throttling raise retry limits or reduce concurrency.
5. If NotFound during conversion, the referenced PersistedSnapshot may have been deleted; verify snapshot retention/cleanup settings.

Example fix

// before: object store URL wrong region
INFLUXDB3_OBJECT_STORE=s3 INFLUXDB3_BUCKET=my-bucket
// after: correct region + explicit endpoint
INFLUXDB3_OBJECT_STORE=s3 INFLUXDB3_BUCKET=my-bucket INFLUXDB3_AWS_DEFAULT_REGION=us-east-1

// verify store reachable before conversion
object_store.head(&probe_path).await.map_err(|e| {
    if matches!(e, object_store::Error::NotFound { .. }) { Ok(()) } else { Err(e) }
})?;

fn is_transient_store_err(e: &object_store::Error) -> bool {
    matches!(e, object_store::Error::Generic { .. }) // classify throttling/network errors for retry
}

match TableIndexSnapshot::persist(store, ).await {
    Err(TableIndexError::ObjectStore(e)) if is_retryable(&e) => backoff_retry().await,
    Err(TableIndexError::ObjectStore(e)) => error!("store misconfigured: {e}"),
    other => other?,
}

When it happens

Trigger: TableIndexSnapshot::persist fails its put_with_default_retries call; TableIndexSnapshot::load_split_persist fails get_with_default_retries or reading the response bytes during PersistedSnapshot -> TableIndexSnapshot conversion at startup; the object store returns any non-NotFound error (auth, throttling, network, permissions).

Common situations: Misconfigured object store credentials or endpoint, S3 throttling (slow down / 503) after retries are exhausted, network partitions between the InfluxDB 3 node and the bucket, missing bucket/container, or read-only IAM policies on the snapshot prefix.

Related errors

• Failed to load table index from object store
• object store error
• object store error
• object_store error
• Object store operation failed

AI-assisted analysis of influxdata/influxdb@06200ef96b (2026-09-19). Data as JSON: /api/errors/6619b4d45f0fd5d7. Report an issue: GitHub.

Appendix: source

Thrown at influxdb3_write/src/table_index.rs:76

    #[error("Failed to parse table index snapshot path from object store path")]
    TableIndexSnapshotPath(#[source] crate::paths::PathError),

    #[error("Failed to update table index: join task failed")]
    UpdateTaskFailed(#[source] tokio::task::JoinError),

    #[error("Object meta is missing filename")]
    MissingFilename,

    #[error("Failed to parse snapshot sequence number from filename")]
    InvalidSnapshotSequenceNumber,

    #[error("Table snapshot persistence task failed")]
    TableSnapshotPersistenceTaskFailed(#[source] tokio::task::JoinError),

    #[error("Unexpected error")]
    Unexpected(#[source] anyhow::Error),

    #[error("Object store operation failed")]
    ObjectStore(#[source] object_store::Error),

    #[error("JSON serialization/deserialization failed")]
    Json(#[source] serde_json::Error),
}

pub type Result<T> = std::result::Result<T, TableIndexError>;

/// An partial, incremental index into a given database/table. Should not be used to make decisions
/// regarding gen1 file retention -- instead, all snapshots should be aggregated into a full
/// CoreTableIndex.
///
/// TableIndexSnapshots are expected to be created during the normal snapshotting process alongside
/// the full process-wide database + table index.
#[derive(Debug, Serialize, Deserialize, Eq, PartialEq, Clone)]
pub struct TableIndexSnapshot {
    #[serde(flatten)]
    pub id: TableIndexId,

View on GitHub (pinned to 06200ef96b)