github/copilot-sdk · critical · IOException

LLM inference response used after RPC connection closed

Error message

LLM inference response used after RPC connection closed

What it means

This IOException comes from the private api() accessor of LlmInferenceExchange: the RPC connection to the server has been closed, so rpcSupplier.get() returns null and no further LLM inference API calls can be made. Any exchange operation (startResponse, writeChunk, endResponse, errorResponse) fails because the transport backing the response is gone.

Solutions

1. Ensure the RPC connection stays open for the full lifetime of every exchange; only close it after endResponse/errorResponse
2. Check rpcSupplier/connection state before writing and abort the response gracefully when null
3. Catch this IOException in streaming code and abort generation instead of retrying writes
4. Investigate why the connection closed early: timeouts, upstream disconnects, or premature client.close()

Example fix

// before
rpcClient.close();
exchange.writeResponseText(chunk);
// after
exchange.writeResponseText(chunk);
exchange.endResponse();
rpcClient.close();

// cannot query rpc state publicly; keep your own flag
if (rpcConnectionClosed) return;

try {
    exchange.writeResponseText(chunk);
} catch (IOException e) {
    if (String.valueOf(e.getMessage()).contains("after RPC connection closed")) {
        abortGeneration();
        return;
    }
    throw e;
}

When it happens

Trigger: Calling startResponse, writeResponseText/Binary, endResponse, or errorResponse after the underlying RPC connection (rpcSupplier) was torn down — server shutdown, client disconnect, or transport close before the response completed.

Common situations: Long-running inference whose stream outlives the connection (client cancelled, proxy timeout); app shutting down while a generation is still streaming; error handling that closes the RPC client in a finally block before draining the exchange.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

• Connection failures: ECONNREFUSED, ECONNRESET, and friends — why connections get refused, reset, or dropped.

Related errors

• Client not connected; call start() first
• LLM inference request was cancelled by the runtime
• LLM inference response writeResponse() called before…
• LLM inference response writeResponse() called after…
• The in-process runtime connection is closed.

AI-assisted analysis of github/copilot-sdk@cd8cf15dc3 (2026-09-09). Data as JSON: /api/errors/c06a6b2f2903d085. Report an issue: GitHub.

Appendix: source

Thrown at java/sdk/src/main/java/com/github/copilot/LlmInferenceExchange.java:250

                throw new IOException("LLM inference request was cancelled by the runtime");
            }
            if (!started) {
                throw new IOException("LLM inference response writeResponse() called before startResponse()");
            }
            if (finished) {
                throw new IOException(
                        "LLM inference response writeResponse() called after endResponse()/errorResponse()");
            }
        }
        var params = new LlmInferenceHttpResponseChunkParams(requestId, data, binary ? Boolean.TRUE : null,
                Boolean.FALSE, null);
        join(api().httpResponseChunk(params));
    }

    private ServerLlmInferenceApi api() throws IOException {
        ServerLlmInferenceApi api = rpcSupplier.get();
        if (api == null) {
            throw new IOException("LLM inference response used after RPC connection closed");
        }
        return api;
    }

    private static <T> T join(CompletableFuture<T> future) throws IOException {
        try {
            return future.join();
        } catch (CompletionException | CancellationException e) {
            Throwable cause = e.getCause() != null ? e.getCause() : e;
            throw new IOException(cause.getMessage(), cause);
        }
    }
}

View on GitHub (pinned to cd8cf15dc3)