meilisearch/meilisearch · error · anyhow::Error

internal

internal

Error message

Failed to read web identity token file: {e}

What it means

Meilisearch's experimental S3 snapshot feature tries to read the OIDC web identity token file (path from --experimental-s3-web-identity-token-file / MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE) before calling STS AssumeRoleWithWebIdentity. tokio::fs::read_to_string failed, so the file does not exist at the given path, is not readable by the meilisearch process, or is not valid UTF-8. The error surfaces as a failed snapshot task because credential extraction runs inside the S3 snapshot upload path (process_snapshot_to_s3).

Source

Thrown at crates/index-scheduler/src/scheduler/enterprise_edition/s3.rs:24

use meilisearch_types::heed::Env;
use meilisearch_types::index_uid::AnyIndex;
use meilisearch_types::milli;
use meilisearch_types::milli::progress::Progress;
use meilisearch_types::tasks::{Status, Task};

use crate::{Error, IndexScheduler, Result};

impl IndexScheduler {
    async fn assume_role_with_web_identity(
        role_arn: &str,
        web_identity_token_file: &std::path::Path,
        ip_policy: http_client::policy::IpPolicy,
    ) -> anyhow::Result<StsCredentials> {
        use std::env::VarError;

        let token = tokio::fs::read_to_string(web_identity_token_file)
            .await
            .map_err(|e| anyhow::anyhow!("Failed to read web identity token file: {e}"))?;

        let duration: u32 =
            match std::env::var("MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_DURATION_SECONDS") {
                Ok(s) => s.parse()?,
                Err(VarError::NotPresent) => 3600,
                Err(VarError::NotUnicode(e)) => {
                    anyhow::bail!("Invalid duration: {e:?}")
                }
            };

        let form_data = [
            ("Action", "AssumeRoleWithWebIdentity"),
            ("Version", "2011-06-15"),
            ("RoleArn", role_arn),
            ("RoleSessionName", "meilisearch-snapshot-session"),
            ("WebIdentityToken", &token),
            ("DurationSeconds", &duration.to_string()),
        ];

View on GitHub (pinned to 577f7af289)

Solutions

1. Verify the path exists and is readable by the meilisearch user: ls -l and sudo -u meilisearch head -1 <token-file>
2. Use an absolute path for MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE; relative paths resolve against the process working directory, not the data dir
3. In containers, confirm the token volume is mounted (Kubernetes: check the projected serviceAccountToken volume in the pod spec)
4. Check the file is valid UTF-8 text (file <token-file>, it should be plain JWT text, not binary)
5. Alternatively drop web-identity auth and pass --experimental-s3-access-key-id / --experimental-s3-secret-access-key instead

Example fix

# before
MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE=token.txt
# after (absolute path to the projected service account token)
MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

// Rust caller-side check before enabling S3 web-identity snapshots
fn token_file_ok(p: &std::path::Path) -> bool {
    p.is_absolute()
        && std::fs::metadata(p).map(|m| m.is_file() && m.permissions().readonly() == false).unwrap_or(false)
        && std::fs::read_to_string(p).map(|s| !s.trim().is_empty() && s.split('.').count() == 3).unwrap_or(false)
}
assert!(token_file_ok(std::path::Path::new("/var/run/secrets/eks.amazonaws.com/serviceaccount/token")));

// When orchestrating snapshot tasks, inspect the task error chain
match task_error {
    e if e.to_string().contains("Failed to read web identity token file") => {
        // fix the MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE path or mount, then re-enqueue the snapshot task
    }
    other => tracing::error!(%other, "snapshot failed"),
}

When it happens

Trigger: A snapshot-to-S3 task is enqueued with s3_role_arn + s3_web_identity_token_file set (and no static access/secret keys), and the token file path is wrong, relative to a different working directory, not mounted into the container, has wrong permissions, or contains binary/non-UTF-8 bytes.

Common situations: Kubernetes/EKS setups where the projected service-account token lives at /var/run/secrets/eks.amazonaws.com/serviceaccount/token but the path is misspelled or the volume is not mounted in the meilisearch pod; Docker deployments with a path not volume-mounted; paths written relative to the data dir instead of the process cwd; file owned by root with 0600 while meilisearch runs as another user.

Related errors

• cannot open ocsp file
• cannot read oscp file
• {} route(s) missing summary
• {} route(s) missing description
• {} path issue(s) found

AI-assisted analysis of meilisearch/meilisearch@577f7af289 (2026-08-16). Data as JSON: /api/errors/f5c62a63bd552da5. Report an issue: GitHub.