{"record":{"id":"dd0aea14397d6939","repo":"windmill-labs/windmill","slug":"failed-to-execute-replace-ephemeral-command","errorCode":null,"errorMessage":"failed to execute  replace_ephemeral command: {}","messagePattern":"failed to execute  replace_ephemeral command: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"backend/windmill-worker/src/python_executor.rs","lineNumber":184,"sourceCode":"        .recursive(true)\n        .create(&format!(\"{job_dir}/dependencies\"))\n        .await\n        .expect(\"could not create dependencies dir\");\n}\n\n#[inline(always)]\npub fn handle_ephemeral_token(x: String) -> String {\n    #[cfg(feature = \"enterprise\")]\n    {\n        if let Some(full_cmd) = EPHEMERAL_TOKEN_CMD.as_ref() {\n            let mut splitted = full_cmd.split(\" \");\n            let cmd = splitted.next().unwrap();\n            let args = splitted.collect::<Vec<&str>>();\n            let output = std::process::Command::new(cmd)\n                .args(args)\n                .output()\n                .map(|x| String::from_utf8(x.stdout).unwrap())\n                .unwrap_or_else(|e| panic!(\"failed to execute  replace_ephemeral command: {}\", e));\n            let r = x.replace(\"EPHEMERAL_TOKEN\", &output.trim());\n            tracing::debug!(\"replaced ephemeral token: '{}'\", r);\n            return r;\n        }\n    }\n    x\n}\n\n/// Removes lockfile/requirements entries matching the worker's `pip_local_dependencies`\n/// regexes. Those packages are already provided locally (e.g. via `additional_python_paths`),\n/// so installing them again duplicates files and triggers expensive `postinstall` copies on\n/// every job. `#`-prefixed comment lines (e.g. the `# py:` lockfile header) are always kept.\n/// Returns `(kept_lines, ignored_lines)`.\nfn filter_pip_local_dependencies(lines: Vec<String>) -> (Vec<String>, Vec<String>) {\n    let Some(pip_local_dependencies) = WORKER_CONFIG.load().pip_local_dependencies.clone() else {\n        return (lines, vec![]);\n    };\n","sourceCodeStart":166,"sourceCodeEnd":202,"githubUrl":"https://github.com/windmill-labs/windmill/blob/e474e8803ce2ff5c2df09a58dab51d45f5c922ca/backend/windmill-worker/src/python_executor.rs#L166-L202","documentation":"Windmill supports ephemeral Python tokens whose command output replaces an `EPHEMERAL_TOKEN` placeholder in the executor command line (used by e.g. Databricks/ephemeral token providers). `handle_ephemeral_token` splits the configured command and runs it with `std::process::Command`; if the command cannot be spawned (or wait fails), it panics with the wrapped OS error. Note the message contains a double space after 'execute' — search accordingly.","triggerScenarios":"Configuring an ephemeral token command whose binary does not exist or is not on PATH (ErrNotFound), lacks execute permission (EACCES), or fails at spawn/wait for any other reason while processing a Python job that contains the EPHEMERAL_TOKEN placeholder.","commonSituations":"Typo in the command name in the ephemeral token env/config; image without the CLI used to fetch short-lived tokens (e.g. cloud provider credential helper); non-root container lacking permission to execute a protected binary.","solutions":["Verify the configured command's binary exists and is on PATH inside the worker container (`which <cmd>`)","Check the embedded OS error: NotFound → install the binary; PermissionDenied → chmod +x / fix ownership","Test the command manually in the worker container with the same env (it must print the token to stdout)","Ensure the command does not require TTY/interactive auth; use a non-interactive credential source"],"exampleFix":"// before: command not present in the worker image\nEPHEMERAL_TOKEN_CMD=\"databricks-cli create-token\"\n// after: install/point to an existing binary\nEPHEMERAL_TOKEN_CMD=\"/usr/local/bin/get-databricks-token\"  # ensure installed & executable","handlingStrategy":"validation","validationCode":"import subprocess, shutil, os\ncmd = os.environ.get('EPHEMERAL_TOKEN_CMD')\nif cmd:\n    binary = cmd.split()[0]\n    if not shutil.which(binary):\n        raise SystemExit(f'{binary} not on PATH in worker image')\n    out = subprocess.run(cmd, shell=True, capture_output=True, text=True)\n    if out.returncode != 0 or not out.stdout.strip():\n        raise SystemExit(f'token command failed: {out.stderr}')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Ensure the token-fetch binary is installed and executable in the worker image","Use non-interactive credential providers (no TTY/auth prompts)","Test the command inside the actual container with the same env vars","Check that the command output is clean single-line stdout (no extra logging)"],"tags":["rust","python","subprocess","token","worker"],"backgroundTag":"command-not-found","analyzedSha":"e474e8803ce2ff5c2df09a58dab51d45f5c922ca","analyzedAt":"2026-09-03T12:38:19.024Z","contentChangedAt":"2026-09-03T12:38:19.024Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}