nathanmarz/storm · critical · RuntimeException

Failed a non-existent or already acked/failed id

Error message

Failed a non-existent or already acked/failed id: ${id}

What it means

ShellBolt runs a shell subprocess and keeps a map (_inputs) of pending tuple ids awaiting ack/fail from the shell process. When the shell sends a 'fail' action for an id that is not in that map — because it was already acked, already failed, or never emitted — Storm throws this RuntimeException, which kills the executor thread and typically the worker.

Solutions

1. Fix the shell script so each tuple id is acked or failed exactly once (track state, never double-report).
2. Ensure the script only fails ids it actually received via the input stream (echo ids received in __init/next).
3. On error paths, choose a single terminal action (ack OR fail) per tuple and return immediately after.
4. Upgrade Storm if you suspect a race between ack and fail being sent; keep ack/fail messages ordered.
5. Log every ack/fail id in the script during debugging to find the duplicate or bogus id.

Example fix

# before (script may fail after ack)
try:
    process(line)
    ack(msg['id'])
except Exception:
    fail(msg['id'])

# after (track terminal state)
done = set()
def safe_fail(tid):
    if tid in done:
        return
    done.add(tid)
    fail(tid)

pending = set()
# on receiving a tuple from Storm:
pending.add(msg_id)
# before sending fail:
if msg_id in pending:
    send_fail(msg_id)
    pending.discard(msg_id)

def is_pending(msg_id):
    return msg_id in pending

When it happens

Trigger: The multilang shell process sends a fail action referencing a tuple id that was already acked or failed, or an id that was never emitted to it (e.g. the shell script fabricates ids or double-reports the same tuple).

Common situations: Non-idempotent ack/fail logic in custom shell spout/bolt scripts (python/node/ruby), multilang scripts that ack and then fail the same tuple on error, or scripts emitting their own made-up ids after a crash/restart of their internal state.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Anchored onto after ack/fail
• Error when launching multilang subprocess
• Error when launching multilang subprocess
• Error during multilang processing
• Acked a non-existent or already acked/failed id

AI-assisted analysis of nathanmarz/storm@cdb116e942 (2026-09-12). Data as JSON: /api/errors/44469b3e05a776c8. Report an issue: GitHub.

Appendix: source

Thrown at storm-core/src/jvm/backtype/storm/task/ShellBolt.java:202

        _running = false;
        _process.destroy();
        _inputs.clear();
    }

    private void handleAck(Map action) {
        String id = (String) action.get("id");
        Tuple acked = _inputs.remove(id);
        if(acked==null) {
            throw new RuntimeException("Acked a non-existent or already acked/failed id: " + id);
        }
        _collector.ack(acked);
    }

    private void handleFail(Map action) {
        String id = (String) action.get("id");
        Tuple failed = _inputs.remove(id);
        if(failed==null) {
            throw new RuntimeException("Failed a non-existent or already acked/failed id: " + id);
        }
        _collector.fail(failed);
    }

    private void handleError(Map action) {
        String msg = (String) action.get("msg");
        _collector.reportError(new Exception("Shell Process Exception: " + msg));
    }

    private void handleEmit(Map action) throws InterruptedException {
        String stream = (String) action.get("stream");
        if(stream==null) stream = Utils.DEFAULT_STREAM_ID;
        Long task = (Long) action.get("task");
        List<Object> tuple = (List) action.get("tuple");
        List<Tuple> anchors = new ArrayList<Tuple>();
        Object anchorObj = action.get("anchors");
        if(anchorObj!=null) {
            if(anchorObj instanceof String) {

View on GitHub (pinned to cdb116e942)