What causes "failed to execute replace_ephemeral command: {}" in windmill-labs/windmill?

Verify the configured command's binary exists and is on PATH inside the worker container (`which `)

failed to execute replace_ephemeral command: {}

Error message

failed to execute  replace_ephemeral command: {}

What it means

Windmill supports ephemeral Python tokens whose command output replaces an `EPHEMERAL_TOKEN` placeholder in the executor command line (used by e.g. Databricks/ephemeral token providers). `handle_ephemeral_token` splits the configured command and runs it with `std::process::Command`; if the command cannot be spawned (or wait fails), it panics with the wrapped OS error. Note the message contains a double space after 'execute' — search accordingly.

Source

Thrown at backend/windmill-worker/src/python_executor.rs:184

        .recursive(true)
        .create(&format!("{job_dir}/dependencies"))
        .await
        .expect("could not create dependencies dir");
}

#[inline(always)]
pub fn handle_ephemeral_token(x: String) -> String {
    #[cfg(feature = "enterprise")]
    {
        if let Some(full_cmd) = EPHEMERAL_TOKEN_CMD.as_ref() {
            let mut splitted = full_cmd.split(" ");
            let cmd = splitted.next().unwrap();
            let args = splitted.collect::<Vec<&str>>();
            let output = std::process::Command::new(cmd)
                .args(args)
                .output()
                .map(|x| String::from_utf8(x.stdout).unwrap())
                .unwrap_or_else(|e| panic!("failed to execute  replace_ephemeral command: {}", e));
            let r = x.replace("EPHEMERAL_TOKEN", &output.trim());
            tracing::debug!("replaced ephemeral token: '{}'", r);
            return r;
        }
    }
    x
}

/// Removes lockfile/requirements entries matching the worker's `pip_local_dependencies`
/// regexes. Those packages are already provided locally (e.g. via `additional_python_paths`),
/// so installing them again duplicates files and triggers expensive `postinstall` copies on
/// every job. `#`-prefixed comment lines (e.g. the `# py:` lockfile header) are always kept.
/// Returns `(kept_lines, ignored_lines)`.
fn filter_pip_local_dependencies(lines: Vec<String>) -> (Vec<String>, Vec<String>) {
    let Some(pip_local_dependencies) = WORKER_CONFIG.load().pip_local_dependencies.clone() else {
        return (lines, vec![]);
    };

View on GitHub (pinned to e474e8803c)

Solutions

Verify the configured command's binary exists and is on PATH inside the worker container (`which <cmd>`)
Check the embedded OS error: NotFound → install the binary; PermissionDenied → chmod +x / fix ownership
Test the command manually in the worker container with the same env (it must print the token to stdout)
Ensure the command does not require TTY/interactive auth; use a non-interactive credential source

Example fix

// before: command not present in the worker image
EPHEMERAL_TOKEN_CMD="databricks-cli create-token"
// after: install/point to an existing binary
EPHEMERAL_TOKEN_CMD="/usr/local/bin/get-databricks-token"  # ensure installed & executable

Defensive patterns

Strategy: validation

Validate before calling

import subprocess, shutil, os
cmd = os.environ.get('EPHEMERAL_TOKEN_CMD')
if cmd:
    binary = cmd.split()[0]
    if not shutil.which(binary):
        raise SystemExit(f'{binary} not on PATH in worker image')
    out = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    if out.returncode != 0 or not out.stdout.strip():
        raise SystemExit(f'token command failed: {out.stderr}')

Prevention

Ensure the token-fetch binary is installed and executable in the worker image
Use non-interactive credential providers (no TTY/auth prompts)
Test the command inside the actual container with the same env vars
Check that the command output is clean single-line stdout (no extra logging)

When it happens

Trigger: Configuring an ephemeral token command whose binary does not exist or is not on PATH (ErrNotFound), lacks execute permission (EACCES), or fails at spawn/wait for any other reason while processing a Python job that contains the EPHEMERAL_TOKEN placeholder.

Common situations: Typo in the command name in the ephemeral token env/config; image without the CLI used to fetch short-lived tokens (e.g. cloud provider credential helper); non-root container lacking permission to execute a protected binary.

Related errors

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/dd0aea14397d6939. Report an issue: GitHub.