github/copilot-sdk · warning · IOException

LLM inference request was cancelled by the runtime

Error message

LLM inference request was cancelled by the runtime

What it means

This IOException is thrown by LlmInferenceExchange.writeChunk when a chunk of the LLM streaming response is about to be written but the exchange has already been cancelled. The library enforces the lifecycle contract of the exchange: once cancel() has been observed, no further response chunks may be sent to the server. It prevents writing data for a request the runtime has abandoned.

Solutions

1. Check the cancelled state (or catch IOException) in your streaming write loop and stop writing as soon as cancellation is signalled
2. Ensure startResponse, writeResponse*, and endResponse calls happen on one thread or are synchronized so cancel cannot interleave mid-stream
3. Wrap per-chunk writes in try-catch for IOException and treat 'cancelled by the runtime' as a normal abort, cleaning up without retrying
4. Investigate why cancellation fired: client disconnects, timeouts, or explicit cancel() calls in the runtime

Example fix

// before
for (String chunk : chunks) {
    exchange.writeResponseText(chunk);
}
// after
try {
    for (String chunk : chunks) {
        exchange.writeResponseText(chunk);
    }
} catch (IOException e) {
    if (String.valueOf(e.getMessage()).contains("cancelled by the runtime")) {
        return; // client cancelled; stop streaming
    }
    throw e;
}

// no pre-call check possible; cancellation is concurrent under lock
// optionally: if (exchange.isCancelled()) return;

try {
    exchange.writeResponseText(chunk);
} catch (IOException e) {
    if (String.valueOf(e.getMessage()).contains("cancelled by the runtime")) return;
    throw e;
}

When it happens

Trigger: writeResponseText or writeResponseBinary is called after the exchange was cancelled (cancelled flag set under lock), typically because the client disconnected, the request timed out, or cancel() was invoked concurrently while streaming chunks.

Common situations: Streaming token-by-token responses to a client that disconnects mid-generation; long inference requests that hit a timeout while the server-side code keeps writing chunks; racing cancel() with in-flight write loops without checking cancellation between chunks.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Request cancelled by runtime
• Copilot request was cancelled by the runtime.
• Request cancelled by runtime
• LLM inference response writeResponse() called before…
• LLM inference response writeResponse() called after…

AI-assisted analysis of github/copilot-sdk@cd8cf15dc3 (2026-09-09). Data as JSON: /api/errors/1ae20c168ccbbd0e. Report an issue: GitHub.

Appendix: source

Thrown at java/sdk/src/main/java/com/github/copilot/LlmInferenceExchange.java:232

        join(api().httpResponseChunk(params));
    }

    void errorResponse(String message, String code) throws IOException {
        synchronized (lock) {
            if (finished) {
                return;
            }
            finished = true;
        }
        var error = new LlmInferenceHttpResponseChunkError(message, code);
        var params = new LlmInferenceHttpResponseChunkParams(requestId, "", null, Boolean.TRUE, error);
        join(api().httpResponseChunk(params));
    }

    private void writeChunk(String data, boolean binary) throws IOException {
        synchronized (lock) {
            if (cancelled) {
                throw new IOException("LLM inference request was cancelled by the runtime");
            }
            if (!started) {
                throw new IOException("LLM inference response writeResponse() called before startResponse()");
            }
            if (finished) {
                throw new IOException(
                        "LLM inference response writeResponse() called after endResponse()/errorResponse()");
            }
        }
        var params = new LlmInferenceHttpResponseChunkParams(requestId, data, binary ? Boolean.TRUE : null,
                Boolean.FALSE, null);
        join(api().httpResponseChunk(params));
    }

    private ServerLlmInferenceApi api() throws IOException {
        ServerLlmInferenceApi api = rpcSupplier.get();
        if (api == null) {
            throw new IOException("LLM inference response used after RPC connection closed");

View on GitHub (pinned to cd8cf15dc3)