denoland/deno · error

OTEL export timed out after {}ms

Error message

OTEL export timed out after {}ms

What it means

OTLP-over-gRPC export path in Deno's OpenTelemetry runtime: HyperClient::grpc_request wraps the whole request (connect + response + body + trailers, capped at 1 MiB) in tokio::time::timeout(self.timeout). If the collector does not complete the round trip within the budget, the future is dropped and ErrorKind::TimedOut with this message is returned. The timeout comes from OTEL_EXPORTER_OTLP_TIMEOUT in milliseconds (default 10000).

Source

Thrown at ext/telemetry/lib.rs:919

      request: Request<Vec<u8>>,
    ) -> Result<
      (hyper::http::response::Parts, Option<hyper::HeaderMap>),
      Box<dyn std::error::Error + Send + Sync>,
    > {
      let (parts, body) = request.into_parts();
      let request = Request::from_parts(parts, Full::from(body));
      let result = tokio::time::timeout(self.timeout, async {
        let response = self.inner.request(request).await?;
        let (parts, body) = response.into_parts();
        let collected = http_body_util::Limited::new(body, 1024 * 1024)
          .collect()
          .await?;
        let trailers = collected.trailers().cloned();
        Ok::<_, Box<dyn std::error::Error + Send + Sync>>((parts, trailers))
      })
      .await
      .map_err(|_| -> Box<dyn std::error::Error + Send + Sync> {
        Box::new(std::io::Error::new(
          std::io::ErrorKind::TimedOut,
          format!("OTEL export timed out after {}ms", self.timeout.as_millis()),
        ))
      })??;
      Ok(result)
    }
  }

  #[async_trait::async_trait]
  impl opentelemetry_http::HttpClient for HyperClient {
    async fn send_bytes(
      &self,
      request: Request<Bytes>,
    ) -> Result<Response<Bytes>, HttpError> {
      let (parts, body) = request.into_parts();
      let request = Request::from_parts(parts, Full::new(body));
      let response = tokio::time::timeout(self.timeout, async {
        let response = self.inner.request(request).await?;

View on GitHub (pinned to 9ad36f7a2c)

Solutions

  1. Raise the budget: export OTEL_EXPORTER_OTLP_TIMEOUT=30000 (milliseconds).
  2. Verify the endpoint is reachable from the app container: curl http://<collector>:4317/ and check DNS/firewall; for gRPC use port 4317.
  3. Run/health-check a local collector sidecar and point OTEL_EXPORTER_OTLP_ENDPOINT at 127.0.0.1 so only the collector handles the slow WAN hop.
  4. Reduce export size/frequency (OTEL_BSP_* batch settings) so a round trip fits the budget.

Example fix

# before
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com:4317
deno run --unstable-otel app.ts # exports time out after 10000ms

# after
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317  # local sidecar
export OTEL_EXPORTER_OTLP_TIMEOUT=30000                    # 30s budget
deno run --unstable-otel app.ts
Defensive patterns

Strategy: retry

Validate before calling

const ep = process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://127.0.0.1:4317";
const ok = await fetch(ep, { signal: AbortSignal.timeout(2_000) }).then(r => true, () => false);
if (!ok) console.warn(`OTLP collector not reachable at ${ep} — exports will time out`);

Try / catch

try { await exportOnce(); } catch (e) { if (e instanceof Error && /OTEL export timed out/.test(e.message)) console.warn("OTLP gRPC export timed out — batch dropped; check collector 4317 and OTEL_EXPORTER_OTLP_TIMEOUT"); else throw e; }

Prevention

When it happens

Trigger: deno run -- unstable-otel with OTEL_EXPORTER_OTLP_PROTOCOL=grpc and a collector endpoint that is unreachable, slow, or stalling (connection hang, TLS handshake slowness, collector overloaded) so the export exceeds OTEL_EXPORTER_OTLP_TIMEOUT.

Common situations: Remote collector behind slow links or flaky DNS; OTEL collector sidecar not yet up when the app starts; batch exports larger than the link can push in 10s; firewalls dropping packets (black hole) instead of refusing; default timeout too small for high-latency environments.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors


AI-assisted analysis of denoland/deno@9ad36f7a2c (2026-08-20). Data as JSON: /api/errors/ec985d7f4e4c7a1c. Report an issue: GitHub.