apache/kafka · error · Exception

Recursive fail invocation

Error message

Recursive fail invocation

What it means

Guard inside the Kafka release-script runtime (release/runtime.py) against re-entrant calls to fail(). fail() sets a module-global `failing` flag, runs every registered fail hook, prints 'FAILURE: <msg>', then sys.exit(1). If fail() is invoked again while `failing` is already True, it raises 'Recursive fail invocation' to prevent an infinite fail-hook loop instead of silently spiraling.

Source

Thrown at release/runtime.py:52

fail_hooks = []
failing = False


def append_fail_hook(name, hook_fn):
    """
    Register a fail hook function, to run in case fail() is called.
    """
    fail_hooks.append((name, hook_fn))


def fail(msg = ""):
    """
    Terminate execution with the given message,
    after running any registered hooks.
    """
    global failing
    if failing:
        raise Exception(f"Recursive fail invocation")
    failing = True

    for name, func in fail_hooks:
        try:
            func()
        except Exception as e:
            print(f"Exception caught in fail hook {name}: {e}")

    print(f"FAILURE: {msg}")
    sys.exit(1)


def prompt(msg):
    """
    Prompt user for input with the given message.
    This removes leading and trailing spaces.
    """
    text = input(msg)

View on GitHub (pinned to 996fb4585a)

Solutions

1. Audit every hook registered with append_fail_hook and remove any call to fail(), cmd(), or confirm_or_fail() from inside hooks - hooks must only do non-failing cleanup.
2. If a hook must run a subprocess, use execute() wrapped in try/except, or call cmd() with allow_failure=True so a failure returns False instead of routing back through fail().
3. Have hooks swallow their own errors locally (the loop at runtime.py:56-59 already catches them) and keep them idempotent and side-effect-light.
4. Reproduce by printing fail_hooks at startup to confirm which hook re-enters fail(), then refactor that specific hook.
5. If you genuinely need nested failure signaling, surface the inner problem via print/logging rather than a second fail() call.

Example fix

# before - hook re-enters fail() via cmd(), causing 'Recursive fail invocation'
import runtime

def cleanup_hook():
    # cmd() -> fail() on failure => recursive fail() while `failing` is True
    runtime.cmd('stop dev broker', 'bin/kafka-server-stop.sh')

runtime.append_fail_hook('cleanup', cleanup_hook)

# after - hook is non-failing: allow_failure=True + local try/except
def cleanup_hook():
    try:
        # allow_failure prevents cmd() from calling fail() on non-zero exit
        runtime.cmd('stop dev broker', 'bin/kafka-server-stop.sh', allow_failure=True)
    except Exception as e:
        # hook owns its own errors; never call fail() here
        print(f"cleanup best-effort failed: {e}")

runtime.append_fail_hook('cleanup', cleanup_hook)

# Run once at startup, BEFORE any fail() can fire, to statically prove no
# registered hook reaches fail()/cmd()/confirm_or_fail().
import inspect
import release.runtime as runtime

FORBIDDEN = ("runtime.fail(", "runtime.cmd(", "runtime.confirm_or_fail(",
             ".fail(", " cmd(", " confirm_or_fail(")

def validate_fail_hooks():
    for name, fn in runtime.fail_hooks:
        try:
            src = inspect.getsource(fn)
        except (OSError, TypeError):
            continue
        offenders = [tok for tok in FORBIDDEN if tok in src]
        if offenders:
            raise AssertionError(
                f"fail hook '{name}' may re-enter fail() via {offenders}; "
                f"refactor to non-failing cleanup."
            )

validate_fail_hooks()  # raises before the release run if a hook is unsafe

# If you must wrap fail() itself, catch only this specific recursion signal
# and fall back to a hard exit - never swallow and continue.
import sys
import release.runtime as runtime

def safe_fail(msg=""):
    try:
        runtime.fail(msg)
    except Exception as e:
        if "Recursive fail invocation" in str(e):
            print("FAILURE (recursive): " + msg, file=sys.stderr)
            sys.exit(2)
        raise  # any other exception propagates

When it happens

Trigger: A function registered via append_fail_hook(name, hook_fn) executes a code path that calls fail() again - most commonly the hook calls cmd(...), which on a subprocess failure after the user declines retry reaches runtime.py:146 fail(''); or the hook calls confirm_or_fail()/fail() directly. Because failing is already True from the outer fail() call, the inner call hits runtime.py:51-52 and raises.

Common situations: A custom cleanup hook shells out via cmd() to tear down a process and that subprocess exits non-zero; a hook logs to a remote system using a helper that itself calls fail() on network errors; a hook reuses confirm_or_fail() to ask the user something and the user answers 'n'; refactor accidentally routes hook error handling back through fail() instead of printing/logging.

Related errors

• Release {version} is not complete since there are unresolve

AI-assisted analysis of apache/kafka@996fb4585a (2026-08-11). Data as JSON: /api/errors/45bf2b4b26879600. Report an issue: GitHub.