What causes "Proxy backend timed out: {}" in floci-io/floci?

Raise floci.services.bedrock-runtime.proxy.request-timeout-seconds (FLOCI_SERVICES_BEDROCK_RUNTIME_PROXY_REQUEST_TIMEOUT_SECONDS)

floci-io/floci · error · AwsException

ModelTimeoutException

ModelTimeoutException

Error message

Proxy backend timed out: {}

What it means

The upstream proxy backend (OpenAI-compatible /chat/completions endpoint) did not answer within floci.services.bedrock-runtime.proxy.request-timeout-seconds. Java's HttpClient raised HttpTimeoutException; Floci maps it to ModelTimeoutException with HTTP 408.

Source

Thrown at src/main/java/io/github/hectorvent/floci/services/bedrockruntime/backend/ProxyBackend.java:91

        proxyConfig.apiKey()
                .filter(key -> !key.isBlank())
                .ifPresent(key -> builder.header("Authorization", "Bearer " + key));

        String requestBody;
        try {
            requestBody = objectMapper.writeValueAsString(openAiRequest);
        } catch (Exception e) {
            throw new AwsException("InternalServerException", "Failed to serialize proxy request: " + e.getMessage(), 500);
        }
        builder.POST(HttpRequest.BodyPublishers.ofString(requestBody));

        long start = System.nanoTime();
        HttpResponse<String> response;
        try {
            response = httpClient.send(builder.build(), HttpResponse.BodyHandlers.ofString());
        } catch (HttpTimeoutException e) {
            LOG.warnv("Bedrock proxy backend timed out: modelId={0}, url={1}, error={2}", modelId, uri, e.getMessage());
            throw new AwsException("ModelTimeoutException", "Proxy backend timed out: " + e.getMessage(), 408);
        } catch (Exception e) {
            LOG.warnv("Bedrock proxy backend call failed: modelId={0}, url={1}, error={2}", modelId, uri, e.getMessage());
            throw new AwsException("ModelErrorException", "Failed to reach proxy backend: " + e.getMessage(), 424);
        }
        long latencyMs = (System.nanoTime() - start) / 1_000_000;

        if (response.statusCode() >= 300) {
            LOG.warnv("Bedrock proxy backend returned HTTP {0}: {1}", response.statusCode(), response.body());
            throw new AwsException("ModelErrorException",
                    "Proxy backend returned HTTP " + response.statusCode() + ": " + truncate(response.body(), 512),
                    424);
        }

        JsonNode openAiResponse;
        try {
            openAiResponse = objectMapper.readTree(response.body());
        } catch (Exception e) {
            throw new AwsException("ModelErrorException", "Proxy backend returned malformed JSON: " + e.getMessage(), 424);

View on GitHub (pinned to 62ff490619)

Solutions

Raise floci.services.bedrock-runtime.proxy.request-timeout-seconds (FLOCI_SERVICES_BEDROCK_RUNTIME_PROXY_REQUEST_TIMEOUT_SECONDS)
Warm the local model before the first request (a trivial completion)
Shorten maxTokens in the Converse request so generation finishes in time

Example fix

# before
FLOCI_SERVICES_BEDROCK_RUNTIME_PROXY_REQUEST_TIMEOUT_SECONDS=30
# after
FLOCI_SERVICES_BEDROCK_RUNTIME_PROXY_REQUEST_TIMEOUT_SECONDS=300

Defensive patterns

Strategy: retry

Try / catch

RetryPolicy<Object> p = RetryPolicy.builder()
    .handle(ModelTimeoutException.class)
    .withDelay(Duration.ofSeconds(5))
    .withMaxRetries(2).build();
Failsafe.with(p).get(() -> client.converse(req));

Prevention

Set request-timeout-seconds generously for local LLM backends
Warm the model with a tiny completion at startup
Cap maxTokens per request to bound generation time

When it happens

Trigger: A slow local model (Ollama/vLLM cold start, long generation) exceeding the configured timeout; an overloaded proxy; a timeout set too low (default seconds).

Common situations: First request after starting a local LLM runtime (model load takes minutes); generating long completions; running the emulator on a constrained machine.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of floci-io/floci@62ff490619 (2026-08-14). Data as JSON: /api/errors/fb185d55fe043050. Report an issue: GitHub.