{"record":{"id":"f5c62a63bd552da5","repo":"meilisearch/meilisearch","slug":"internal","errorCode":"internal","errorMessage":"Failed to read web identity token file: {e}","messagePattern":"Failed to read web identity token file: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/index-scheduler/src/scheduler/enterprise_edition/s3.rs","lineNumber":24,"sourceCode":"use meilisearch_types::heed::Env;\nuse meilisearch_types::index_uid::AnyIndex;\nuse meilisearch_types::milli;\nuse meilisearch_types::milli::progress::Progress;\nuse meilisearch_types::tasks::{Status, Task};\n\nuse crate::{Error, IndexScheduler, Result};\n\nimpl IndexScheduler {\n    async fn assume_role_with_web_identity(\n        role_arn: &str,\n        web_identity_token_file: &std::path::Path,\n        ip_policy: http_client::policy::IpPolicy,\n    ) -> anyhow::Result<StsCredentials> {\n        use std::env::VarError;\n\n        let token = tokio::fs::read_to_string(web_identity_token_file)\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Failed to read web identity token file: {e}\"))?;\n\n        let duration: u32 =\n            match std::env::var(\"MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_DURATION_SECONDS\") {\n                Ok(s) => s.parse()?,\n                Err(VarError::NotPresent) => 3600,\n                Err(VarError::NotUnicode(e)) => {\n                    anyhow::bail!(\"Invalid duration: {e:?}\")\n                }\n            };\n\n        let form_data = [\n            (\"Action\", \"AssumeRoleWithWebIdentity\"),\n            (\"Version\", \"2011-06-15\"),\n            (\"RoleArn\", role_arn),\n            (\"RoleSessionName\", \"meilisearch-snapshot-session\"),\n            (\"WebIdentityToken\", &token),\n            (\"DurationSeconds\", &duration.to_string()),\n        ];","sourceCodeStart":6,"sourceCodeEnd":42,"githubUrl":"https://github.com/meilisearch/meilisearch/blob/577f7af28942b71782eab1e59f44ad8296ce0a92/crates/index-scheduler/src/scheduler/enterprise_edition/s3.rs#L6-L42","documentation":"Meilisearch's experimental S3 snapshot feature tries to read the OIDC web identity token file (path from --experimental-s3-web-identity-token-file / MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE) before calling STS AssumeRoleWithWebIdentity. tokio::fs::read_to_string failed, so the file does not exist at the given path, is not readable by the meilisearch process, or is not valid UTF-8. The error surfaces as a failed snapshot task because credential extraction runs inside the S3 snapshot upload path (process_snapshot_to_s3).","triggerScenarios":"A snapshot-to-S3 task is enqueued with s3_role_arn + s3_web_identity_token_file set (and no static access/secret keys), and the token file path is wrong, relative to a different working directory, not mounted into the container, has wrong permissions, or contains binary/non-UTF-8 bytes.","commonSituations":"Kubernetes/EKS setups where the projected service-account token lives at /var/run/secrets/eks.amazonaws.com/serviceaccount/token but the path is misspelled or the volume is not mounted in the meilisearch pod; Docker deployments with a path not volume-mounted; paths written relative to the data dir instead of the process cwd; file owned by root with 0600 while meilisearch runs as another user.","solutions":["Verify the path exists and is readable by the meilisearch user: ls -l and sudo -u meilisearch head -1 <token-file>","Use an absolute path for MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE; relative paths resolve against the process working directory, not the data dir","In containers, confirm the token volume is mounted (Kubernetes: check the projected serviceAccountToken volume in the pod spec)","Check the file is valid UTF-8 text (file <token-file>, it should be plain JWT text, not binary)","Alternatively drop web-identity auth and pass --experimental-s3-access-key-id / --experimental-s3-secret-access-key instead"],"exampleFix":"# before\nMEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE=token.txt\n# after (absolute path to the projected service account token)\nMEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token","handlingStrategy":"validation","validationCode":"// Rust caller-side check before enabling S3 web-identity snapshots\nfn token_file_ok(p: &std::path::Path) -> bool {\n    p.is_absolute()\n        && std::fs::metadata(p).map(|m| m.is_file() && m.permissions().readonly() == false).unwrap_or(false)\n        && std::fs::read_to_string(p).map(|s| !s.trim().is_empty() && s.split('.').count() == 3).unwrap_or(false)\n}\nassert!(token_file_ok(std::path::Path::new(\"/var/run/secrets/eks.amazonaws.com/serviceaccount/token\")));","typeGuard":null,"tryCatchPattern":"// When orchestrating snapshot tasks, inspect the task error chain\nmatch task_error {\n    e if e.to_string().contains(\"Failed to read web identity token file\") => {\n        // fix the MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE path or mount, then re-enqueue the snapshot task\n    }\n    other => tracing::error!(%other, \"snapshot failed\"),\n}","preventionTips":["Always use absolute paths for MEILI_EXPERIMENTAL_S3_WEB_IDENTITY_TOKEN_FILE","Mount the projected service-account token volume read-only in the same pod as meilisearch","Add a startup/readiness probe script that validates existence and readability of the token file before enabling S3 snapshots"],"tags":["rust","meilisearch","s3","aws","sts","iam","filesystem","oidc","snapshots"],"backgroundTag":null,"analyzedSha":"577f7af28942b71782eab1e59f44ad8296ce0a92","analyzedAt":"2026-08-16T08:38:42.826Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}