{"record":{"id":"45bf2b4b26879600","repo":"apache/kafka","slug":"recursive-fail-invocation","errorCode":null,"errorMessage":"Recursive fail invocation","messagePattern":"Recursive fail invocation","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"release/runtime.py","lineNumber":52,"sourceCode":"fail_hooks = []\nfailing = False\n\n\ndef append_fail_hook(name, hook_fn):\n    \"\"\"\n    Register a fail hook function, to run in case fail() is called.\n    \"\"\"\n    fail_hooks.append((name, hook_fn))\n\n\ndef fail(msg = \"\"):\n    \"\"\"\n    Terminate execution with the given message,\n    after running any registered hooks.\n    \"\"\"\n    global failing\n    if failing:\n        raise Exception(f\"Recursive fail invocation\")\n    failing = True\n\n    for name, func in fail_hooks:\n        try:\n            func()\n        except Exception as e:\n            print(f\"Exception caught in fail hook {name}: {e}\")\n\n    print(f\"FAILURE: {msg}\")\n    sys.exit(1)\n\n\ndef prompt(msg):\n    \"\"\"\n    Prompt user for input with the given message.\n    This removes leading and trailing spaces.\n    \"\"\"\n    text = input(msg)","sourceCodeStart":34,"sourceCodeEnd":70,"githubUrl":"https://github.com/apache/kafka/blob/996fb4585aa1bcc8980b0e1b8d6b168b986cd979/release/runtime.py#L34-L70","documentation":"Guard inside the Kafka release-script runtime (release/runtime.py) against re-entrant calls to fail(). fail() sets a module-global `failing` flag, runs every registered fail hook, prints 'FAILURE: <msg>', then sys.exit(1). If fail() is invoked again while `failing` is already True, it raises 'Recursive fail invocation' to prevent an infinite fail-hook loop instead of silently spiraling.","triggerScenarios":"A function registered via append_fail_hook(name, hook_fn) executes a code path that calls fail() again - most commonly the hook calls cmd(...), which on a subprocess failure after the user declines retry reaches runtime.py:146 fail(''); or the hook calls confirm_or_fail()/fail() directly. Because failing is already True from the outer fail() call, the inner call hits runtime.py:51-52 and raises.","commonSituations":"A custom cleanup hook shells out via cmd() to tear down a process and that subprocess exits non-zero; a hook logs to a remote system using a helper that itself calls fail() on network errors; a hook reuses confirm_or_fail() to ask the user something and the user answers 'n'; refactor accidentally routes hook error handling back through fail() instead of printing/logging.","solutions":["Audit every hook registered with append_fail_hook and remove any call to fail(), cmd(), or confirm_or_fail() from inside hooks - hooks must only do non-failing cleanup.","If a hook must run a subprocess, use execute() wrapped in try/except, or call cmd() with allow_failure=True so a failure returns False instead of routing back through fail().","Have hooks swallow their own errors locally (the loop at runtime.py:56-59 already catches them) and keep them idempotent and side-effect-light.","Reproduce by printing fail_hooks at startup to confirm which hook re-enters fail(), then refactor that specific hook.","If you genuinely need nested failure signaling, surface the inner problem via print/logging rather than a second fail() call."],"exampleFix":"# before - hook re-enters fail() via cmd(), causing 'Recursive fail invocation'\nimport runtime\n\ndef cleanup_hook():\n    # cmd() -> fail() on failure => recursive fail() while `failing` is True\n    runtime.cmd('stop dev broker', 'bin/kafka-server-stop.sh')\n\nruntime.append_fail_hook('cleanup', cleanup_hook)\n\n# after - hook is non-failing: allow_failure=True + local try/except\ndef cleanup_hook():\n    try:\n        # allow_failure prevents cmd() from calling fail() on non-zero exit\n        runtime.cmd('stop dev broker', 'bin/kafka-server-stop.sh', allow_failure=True)\n    except Exception as e:\n        # hook owns its own errors; never call fail() here\n        print(f\"cleanup best-effort failed: {e}\")\n\nruntime.append_fail_hook('cleanup', cleanup_hook)","handlingStrategy":"validation","validationCode":"# Run once at startup, BEFORE any fail() can fire, to statically prove no\n# registered hook reaches fail()/cmd()/confirm_or_fail().\nimport inspect\nimport release.runtime as runtime\n\nFORBIDDEN = (\"runtime.fail(\", \"runtime.cmd(\", \"runtime.confirm_or_fail(\",\n             \".fail(\", \" cmd(\", \" confirm_or_fail(\")\n\ndef validate_fail_hooks():\n    for name, fn in runtime.fail_hooks:\n        try:\n            src = inspect.getsource(fn)\n        except (OSError, TypeError):\n            continue\n        offenders = [tok for tok in FORBIDDEN if tok in src]\n        if offenders:\n            raise AssertionError(\n                f\"fail hook '{name}' may re-enter fail() via {offenders}; \"\n                f\"refactor to non-failing cleanup.\"\n            )\n\nvalidate_fail_hooks()  # raises before the release run if a hook is unsafe","typeGuard":null,"tryCatchPattern":"# If you must wrap fail() itself, catch only this specific recursion signal\n# and fall back to a hard exit - never swallow and continue.\nimport sys\nimport release.runtime as runtime\n\ndef safe_fail(msg=\"\"):\n    try:\n        runtime.fail(msg)\n    except Exception as e:\n        if \"Recursive fail invocation\" in str(e):\n            print(\"FAILURE (recursive): \" + msg, file=sys.stderr)\n            sys.exit(2)\n        raise  # any other exception propagates","preventionTips":["Never call fail(), cmd(), or confirm_or_fail() from inside a hook registered with append_fail_hook.","When a hook needs to run a subprocess, use cmd(..., allow_failure=True) or execute() inside try/except.","Keep fail hooks idempotent and cheap - they run during the failure path when state is already broken.","Add the validate_fail_hooks() startup check in CI/release scripts so an unsafe hook is caught before a real release.","Remember runtime.py:56-59 already swallows exceptions raised by hooks, so log inside the hook rather than re-failing."],"tags":["release","runtime","release-script","recursion","fail-hook","control-flow","apache-kafka"],"backgroundTag":null,"analyzedSha":"996fb4585aa1bcc8980b0e1b8d6b168b986cd979","analyzedAt":"2026-08-11T22:03:28.655Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}