neondatabase/neon · error

{err}

Error message

{err}

What it means

The gRPC variant of getpage_latest_lsn maps every tonic transport or status error from Client::get_page into anyhow with the raw message text. It fires when the RPC fails before a valid response is produced: channel closed, connection refused or dropped mid-run, stream reset, deadline exceeded, or an error Status returned by the pageserver handler.

Source

Thrown at pageserver/pagebench/src/cmd/getpage_latest_lsn.rs:759

        rel: RelTag,
        blks: Vec<u32>,
    ) -> anyhow::Result<()> {
        let req = page_api::GetPageRequest {
            request_id: req_id.into(),
            request_class: page_api::GetPageClass::Normal,
            read_lsn: page_api::ReadLsn {
                request_lsn: req_lsn,
                not_modified_since_lsn: Some(mod_lsn),
            },
            rel,
            block_numbers: blks,
        };
        let inner = self.inner.clone();
        self.requests.push(Box::pin(async move {
            inner
                .get_page(req)
                .await
                .map_err(|err| anyhow::anyhow!("{err}"))
        }));
        Ok(())
    }

    async fn recv_get_page(&mut self) -> anyhow::Result<(u64, Vec<Bytes>)> {
        let resp = self.requests.next().await.unwrap()?;
        Ok((
            resp.request_id.id,
            resp.pages.into_iter().map(|p| p.image).collect(),
        ))
    }
}

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Verify the gRPC page_api listener address and that it is enabled in the pageserver configuration
2. Inspect pageserver logs and panics around the failure time
3. Re-run the bench once; transient transport failures are common
4. Raise or disable keepalive and idle timeouts on any proxy between client and server

Example fix

// before
inner.get_page(req).await.map_err(|err| anyhow::anyhow!("{err}"))
// after: retry once on transient transport failure
match inner.clone().get_page(req.clone()).await {
    Ok(resp) => Ok(resp),
    Err(status) if matches!(status.code(), tonic::Code::Unavailable | tonic::Code::DeadlineExceeded) => {
        inner.get_page(req).await.map_err(|err| anyhow::anyhow!("{err}"))
    }
    Err(err) => Err(anyhow::anyhow!("{err}")),
}

async fn endpoint_reachable(addr: &str) -> bool {
    tokio::net::TcpStream::connect(addr).await.is_ok()
}

match inner.get_page(req).await {
    Ok(resp) => resp,
    Err(status) => {
        let code = status.code();
        if matches!(code, tonic::Code::Unavailable | tonic::Code::DeadlineExceeded) {
            // rebuild the channel from the Endpoint and retry the call once
        }
        anyhow::bail!("get_page failed: {status}")
    }
}

When it happens

Trigger: pageserver restart, crash, or OOM during the run; the gRPC endpoint not enabled or the wrong port used; a handler panic surfacing as an Unknown status; proxies or load balancers killing the long-lived channel.

Common situations: Passing the libpq page_service port where the gRPC listener is expected; long benches crossing idle or keepalive timeouts of intermediaries; environments where DNS changes mid-run invalidate the endpoint.

Related errors

• no response
• unexpected status code: {}
• error downloading extension {:?}: {:?}
• no grpc_url for shard {shard_index}
• Failed to upload all blocks {:#?}

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/1c68ee5299f08723. Report an issue: GitHub.