risingwavelabs/risingwave · error · SinkError

redirection occur more than twice when sending stream load r

Error message

redirection occur more than twice when sending stream load request

What it means

send_stream_load_request manually follows FE redirects (to preserve Authorization headers that reqwest would strip) but allows at most 2 rounds: follower FE -> leader FE -> BE. If after two redirected requests the response still redirects (and the redirected URL's port equals the original, so it looks like another FE), this error is thrown.

Source

Thrown at src/connector/src/sink/doris_starrocks_connector.rs:447

                // this is a FE address. Because in practice, no one would deploy their `StarRocks` cluster with the same
                // http port for both FE and BE. However, this is a potentially problematic assumption,
                // we may investigate a better way to do this. For example, we could use the `show backends` command to check
                // if the host of the redirected url is in the list. However, `show backends` requires
                // the system-level privilege, which could break the backward compatibility.
                let redirected_port = be_url.port();
                *request_for_redirection.url_mut() = be_url;
                if redirected_port == original_http_port {
                    // redirected to FE, continue another round.
                    request = request_for_redirection;
                } else {
                    // we got BE address here
                    return Ok(StreamLoadResponse::BeRequest(request_for_redirection));
                }
            }
            None => return Ok(StreamLoadResponse::HttpResponse(resp)),
        }
    }
    Err(SinkError::DorisStarrocksConnect(anyhow!(
        "redirection occur more than twice when sending stream load request"
    )))
}

pub struct MetaRequestSender {
    client: Client,
    request: Request,
    fe_host: String,
}

impl MetaRequestSender {
    pub fn new(client: Client, request: Request, fe_host: String) -> Self {
        Self {
            client,
            request,
            fe_host,
        }
    }

View on GitHub (pinned to 6469eb736d)

Solutions

Ensure FE and BE use different HTTP ports (avoid the same http_port for FE and BE) so the FE/BE redirect detection heuristic works
Check for a redirect loop: curl -v --location-trusted against the stream load URL and inspect the Location chain
Remove or correctly configure any reverse proxy in front of the FE so it does not repeatedly redirect with the same port
Verify the configured URL points to an FE (not a BE) and that only one follower hop is expected; point directly at the leader FE if possible

Example fix

# before: FE and BE both on 8030 -> redirect heuristic misfires, loops twice
# after: give BE a distinct http_port (e.g. 8040) so BE redirects are detected and the loop ends

Defensive patterns

Strategy: validation

Validate before calling

// Before creating the sink, verify the redirect chain terminates:
let client = reqwest::Client::builder().redirect(reqwest::redirect::Policy::none()).build()?;
let mut url = format!("{endpoint}/api/transaction/begin");
for hop in 0..3 {
    let resp = client.post(&url).send().await?;
    match resp.headers().get(reqwest::header::LOCATION) {
        Some(loc) => {
            url = resp.url().join(loc.to_str()?)?.to_string();
            if hop == 2 { return Err("redirect chain exceeds 2 hops; FE/BE port heuristic will fail".into()); }
        }
        None => break,
    }
}

Type guard

fn looks_like_fe_redirect(location: &reqwest::Url, original_port: Option<u16>) -> bool {
    location.port() == original_port // same heuristic the connector uses: same port => FE
}

Try / catch

match send_stream_load_request(client, request, &fe_host).await {
    Err(e) if e.to_string().contains("redirection occur more than twice") => {
        // fall back to querying the leader FE directly (SHOW FRONTENDS) and retry once against it
    }
    other => other?,
}

Prevention

Deploy FE and BE with distinct http_port values so the port-based FE/BE detection works
Avoid placing redirecting reverse proxies in front of the FE; route directly to the FE
Point the sink URL at a stable FE address (VIP/DNS that resolves to the leader) to minimize follower redirect hops
After FE topology changes, re-verify only one follower->leader redirect is needed

When it happens

Trigger: The stream load / transaction endpoint keeps returning a redirect to an FE (Location URL with the same HTTP port as the request) after two hops — e.g. a redirect loop among FE nodes, or a misconfigured proxy that always redirects with the same port, so the FE-vs-BE port heuristic never terminates.

Common situations: Deployments where FE and BE share the same HTTP port (the code's heuristic then mistakes a BE redirect for an FE redirect); a chain of more than one follower FE; a reverse proxy or load balancer in front of FE that re-redirects endlessly; FE leader election flapping.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/26fad9357c12416b. Report an issue: GitHub.