What causes "lambda invocation failed: {}" in quickwit-oss/quickwit?

Verify the Lambda function name/ARN configured for the lambda client exists in the target region (`aws lambda get-function --function-name `).

quickwit-oss/quickwit · error · LambdaInvokeError

lambda invocation failed: {}

Error message

lambda invocation failed: {}

What it means

This error is raised in `invoke_error_to_lambda_error` (quickwit/quickwit-lambda-client/src/invoker.rs:100) when an AWS SDK Lambda `InvokeError` cannot be mapped to a successful invocation result. The raw SDK error is wrapped with `DisplayErrorContext` into the message "lambda invocation failed: {}" and then classified: if it matches known timeout variants (including `EfsMountTimeoutException` and `SnapStartTimeoutException`) it becomes `LambdaInvokeError::Timeout`, otherwise it becomes `LambdaInvokeError::Permanent(SearchError::Internal(...))`. It means the remote Lambda function call itself failed before returning a search response.

Source

Thrown at quickwit/quickwit-lambda-client/src/invoker.rs:100

            | InvokeError::Ec2ThrottledException(_)
            | InvokeError::ResourceConflictException(_) => {
                return LambdaInvokeError::RateLimited(None);
            }
            _ => {}
        }
    }

    let is_timeout = match &error {
        SdkError::TimeoutError(_) => true,
        SdkError::DispatchFailure(failure) => failure.is_io() || failure.is_timeout(),
        SdkError::ServiceError(service_error) => matches!(
            service_error.err(),
            InvokeError::EfsMountTimeoutException(_) | InvokeError::SnapStartTimeoutException(_)
        ),
        _ => false,
    };

    let error_msg = format!("lambda invocation failed: {}", DisplayErrorContext(&error));

    if is_timeout {
        LambdaInvokeError::Timeout(error_msg)
    } else {
        LambdaInvokeError::Permanent(SearchError::Internal(error_msg))
    }
}

/// Create a Lambda invoker for a specific version.
///
/// The version number is used as the qualifier when invoking, ensuring we call
/// the exact published version (not $LATEST).
pub(crate) async fn create_lambda_invoker_for_version(
    function_name: String,
    version: String,
) -> anyhow::Result<AwsLambdaInvoker> {
    let aws_config = aws_config::load_defaults(aws_config::BehaviorVersion::latest()).await;
    let client = LambdaClient::new(&aws_config);

View on GitHub (pinned to a39730c5cd)

Solutions

Verify the Lambda function name/ARN configured for the lambda client exists in the target region (`aws lambda get-function --function-name <name>`).
Check the Quickwit node's IAM role has `lambda:InvokeFunction` permission on the function.
Read the wrapped `DisplayErrorContext` message to identify the specific SDK variant; if it is a Timeout variant, inspect the Lambda's EFS configuration / SnapStart settings and raise timeouts.
Confirm network reachability (VPC, endpoints) and that the payload is under the 6 MB synchronous invoke limit.
Retry transient failures (throttling) with backoff; permanent variants indicate a configuration problem, not a retry case.

Example fix

// before: generic failure with no classification insight
let error_msg = format!("lambda invocation failed: {}", DisplayErrorContext(&error));
// after: log the underlying SDK error kind for faster diagnosis
tracing::error!(error = %DisplayErrorContext(&error), "lambda invocation failed");
let error_msg = format!("lambda invocation failed: {}", DisplayErrorContext(&error));

Defensive patterns

Strategy: try-catch

Validate before calling

// Before invoking, check the function exists and is reachable
aws lambda get-function --function-name "$LAMBDA_SEARCH_FUNCTION" --region "$AWS_REGION"
// and confirm IAM policy allows lambda:InvokeFunction on the function ARN

Type guard

fn is_lambda_timeout(err: &aws_sdk_lambda::Error) -> bool {
    matches!(
        err,
        aws_sdk_lambda::error::InvokeError::EfsMountTimeoutException(_)
            | aws_sdk_lambda::error::InvokeError::SnapStartTimeoutException(_)
    )
}

Try / catch

match invoke_error_to_lambda_error(service_error) {
    LambdaInvokeError::Timeout(msg) => {
        // transient: retry with exponential backoff
        tracing::warn!(%msg, "lambda invoke timed out, retrying");
        backoff_retry()
    }
    LambdaInvokeError::Permanent(SearchError::Internal(msg)) => {
        // configuration/permission problem: do not retry, surface to caller
        tracing::error!(%msg, "lambda invoke failed permanently");
        return Err(anyhow!(msg));
    }
}

Prevention

Pin and validate the Lambda function name/ARN in Quickwit config at startup with a get-function check.
Grant lambda:InvokeFunction explicitly to the Quickwit instance role (least privilege on the exact ARN).
Keep search request payloads well under the 6 MB synchronous invoke limit; batch or shard large requests.
Keep the Lambda in the same region and ensure VPC/endpoints allow egress from Quickwit nodes.
Alert on this error's Permanent variant; retrying configuration errors only masks the problem.

When it happens

Trigger: Calling the Lambda-backed leaf search invoker when the AWS SDK invoke call fails: function does not exist or wrong function name/alias, missing IAM permissions (lambda:InvokeFunction), payload too large, EFS mount timeout, SnapStart timeout, throttling, or the function configuration is unreachable from the network.

Common situations: Misconfigured `LAMBDA_SEARCH_FUNCTION_NAME`/ARN after redeploying the lambda; IAM role lacking invoke permissions; Lambda deployed in a VPC or region unreachable from the Quickwit node; EFS-attached lambda timing out on cold start; request payload exceeding the 6 MB synchronous invocation limit.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/df83b49572d4c812. Report an issue: GitHub.