{"record":{"id":"1ae20c168ccbbd0e","repo":"github/copilot-sdk","slug":"llm-inference-request-was-cancelled-by-the-runtime","errorCode":null,"errorMessage":"LLM inference request was cancelled by the runtime","messagePattern":"LLM inference request was cancelled by the runtime","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"java/sdk/src/main/java/com/github/copilot/LlmInferenceExchange.java","lineNumber":232,"sourceCode":"        join(api().httpResponseChunk(params));\n    }\n\n    void errorResponse(String message, String code) throws IOException {\n        synchronized (lock) {\n            if (finished) {\n                return;\n            }\n            finished = true;\n        }\n        var error = new LlmInferenceHttpResponseChunkError(message, code);\n        var params = new LlmInferenceHttpResponseChunkParams(requestId, \"\", null, Boolean.TRUE, error);\n        join(api().httpResponseChunk(params));\n    }\n\n    private void writeChunk(String data, boolean binary) throws IOException {\n        synchronized (lock) {\n            if (cancelled) {\n                throw new IOException(\"LLM inference request was cancelled by the runtime\");\n            }\n            if (!started) {\n                throw new IOException(\"LLM inference response writeResponse() called before startResponse()\");\n            }\n            if (finished) {\n                throw new IOException(\n                        \"LLM inference response writeResponse() called after endResponse()/errorResponse()\");\n            }\n        }\n        var params = new LlmInferenceHttpResponseChunkParams(requestId, data, binary ? Boolean.TRUE : null,\n                Boolean.FALSE, null);\n        join(api().httpResponseChunk(params));\n    }\n\n    private ServerLlmInferenceApi api() throws IOException {\n        ServerLlmInferenceApi api = rpcSupplier.get();\n        if (api == null) {\n            throw new IOException(\"LLM inference response used after RPC connection closed\");","sourceCodeStart":214,"sourceCodeEnd":250,"githubUrl":"https://github.com/github/copilot-sdk/blob/cd8cf15dc3f9e762615790aaed0a771a0f392755/java/sdk/src/main/java/com/github/copilot/LlmInferenceExchange.java#L214-L250","documentation":"This IOException is thrown by LlmInferenceExchange.writeChunk when a chunk of the LLM streaming response is about to be written but the exchange has already been cancelled. The library enforces the lifecycle contract of the exchange: once cancel() has been observed, no further response chunks may be sent to the server. It prevents writing data for a request the runtime has abandoned.","triggerScenarios":"writeResponseText or writeResponseBinary is called after the exchange was cancelled (cancelled flag set under lock), typically because the client disconnected, the request timed out, or cancel() was invoked concurrently while streaming chunks.","commonSituations":"Streaming token-by-token responses to a client that disconnects mid-generation; long inference requests that hit a timeout while the server-side code keeps writing chunks; racing cancel() with in-flight write loops without checking cancellation between chunks.","solutions":["Check the cancelled state (or catch IOException) in your streaming write loop and stop writing as soon as cancellation is signalled","Ensure startResponse, writeResponse*, and endResponse calls happen on one thread or are synchronized so cancel cannot interleave mid-stream","Wrap per-chunk writes in try-catch for IOException and treat 'cancelled by the runtime' as a normal abort, cleaning up without retrying","Investigate why cancellation fired: client disconnects, timeouts, or explicit cancel() calls in the runtime"],"exampleFix":"// before\nfor (String chunk : chunks) {\n    exchange.writeResponseText(chunk);\n}\n// after\ntry {\n    for (String chunk : chunks) {\n        exchange.writeResponseText(chunk);\n    }\n} catch (IOException e) {\n    if (String.valueOf(e.getMessage()).contains(\"cancelled by the runtime\")) {\n        return; // client cancelled; stop streaming\n    }\n    throw e;\n}","handlingStrategy":"try-catch","validationCode":"// no pre-call check possible; cancellation is concurrent under lock\n// optionally: if (exchange.isCancelled()) return;","typeGuard":null,"tryCatchPattern":"try {\n    exchange.writeResponseText(chunk);\n} catch (IOException e) {\n    if (String.valueOf(e.getMessage()).contains(\"cancelled by the runtime\")) return;\n    throw e;\n}","preventionTips":["Treat cancellation as a normal streaming outcome and abort quietly","Check cancelled state between chunks in long loops","Never reuse an exchange after cancellation","Keep start/write/end calls sequenced on one thread"],"tags":["io","streaming","lifecycle","cancellation"],"backgroundTag":"invalid-state-transition","analyzedSha":"cd8cf15dc3f9e762615790aaed0a771a0f392755","analyzedAt":"2026-09-09T18:32:31.973Z","contentChangedAt":"2026-09-09T18:32:31.973Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}