{"record":{"id":"df83b49572d4c812","repo":"quickwit-oss/quickwit","slug":"lambda-invocation-failed","errorCode":null,"errorMessage":"lambda invocation failed: {}","messagePattern":"lambda invocation failed: (.+?)","errorType":"exception","errorClass":"LambdaInvokeError","httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-lambda-client/src/invoker.rs","lineNumber":100,"sourceCode":"            | InvokeError::Ec2ThrottledException(_)\n            | InvokeError::ResourceConflictException(_) => {\n                return LambdaInvokeError::RateLimited(None);\n            }\n            _ => {}\n        }\n    }\n\n    let is_timeout = match &error {\n        SdkError::TimeoutError(_) => true,\n        SdkError::DispatchFailure(failure) => failure.is_io() || failure.is_timeout(),\n        SdkError::ServiceError(service_error) => matches!(\n            service_error.err(),\n            InvokeError::EfsMountTimeoutException(_) | InvokeError::SnapStartTimeoutException(_)\n        ),\n        _ => false,\n    };\n\n    let error_msg = format!(\"lambda invocation failed: {}\", DisplayErrorContext(&error));\n\n    if is_timeout {\n        LambdaInvokeError::Timeout(error_msg)\n    } else {\n        LambdaInvokeError::Permanent(SearchError::Internal(error_msg))\n    }\n}\n\n/// Create a Lambda invoker for a specific version.\n///\n/// The version number is used as the qualifier when invoking, ensuring we call\n/// the exact published version (not $LATEST).\npub(crate) async fn create_lambda_invoker_for_version(\n    function_name: String,\n    version: String,\n) -> anyhow::Result<AwsLambdaInvoker> {\n    let aws_config = aws_config::load_defaults(aws_config::BehaviorVersion::latest()).await;\n    let client = LambdaClient::new(&aws_config);","sourceCodeStart":82,"sourceCodeEnd":118,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-lambda-client/src/invoker.rs#L82-L118","documentation":"This error is raised in `invoke_error_to_lambda_error` (quickwit/quickwit-lambda-client/src/invoker.rs:100) when an AWS SDK Lambda `InvokeError` cannot be mapped to a successful invocation result. The raw SDK error is wrapped with `DisplayErrorContext` into the message \"lambda invocation failed: {}\" and then classified: if it matches known timeout variants (including `EfsMountTimeoutException` and `SnapStartTimeoutException`) it becomes `LambdaInvokeError::Timeout`, otherwise it becomes `LambdaInvokeError::Permanent(SearchError::Internal(...))`. It means the remote Lambda function call itself failed before returning a search response.","triggerScenarios":"Calling the Lambda-backed leaf search invoker when the AWS SDK invoke call fails: function does not exist or wrong function name/alias, missing IAM permissions (lambda:InvokeFunction), payload too large, EFS mount timeout, SnapStart timeout, throttling, or the function configuration is unreachable from the network.","commonSituations":"Misconfigured `LAMBDA_SEARCH_FUNCTION_NAME`/ARN after redeploying the lambda; IAM role lacking invoke permissions; Lambda deployed in a VPC or region unreachable from the Quickwit node; EFS-attached lambda timing out on cold start; request payload exceeding the 6 MB synchronous invocation limit.","solutions":["Verify the Lambda function name/ARN configured for the lambda client exists in the target region (`aws lambda get-function --function-name <name>`).","Check the Quickwit node's IAM role has `lambda:InvokeFunction` permission on the function.","Read the wrapped `DisplayErrorContext` message to identify the specific SDK variant; if it is a Timeout variant, inspect the Lambda's EFS configuration / SnapStart settings and raise timeouts.","Confirm network reachability (VPC, endpoints) and that the payload is under the 6 MB synchronous invoke limit.","Retry transient failures (throttling) with backoff; permanent variants indicate a configuration problem, not a retry case."],"exampleFix":"// before: generic failure with no classification insight\nlet error_msg = format!(\"lambda invocation failed: {}\", DisplayErrorContext(&error));\n// after: log the underlying SDK error kind for faster diagnosis\ntracing::error!(error = %DisplayErrorContext(&error), \"lambda invocation failed\");\nlet error_msg = format!(\"lambda invocation failed: {}\", DisplayErrorContext(&error));","handlingStrategy":"try-catch","validationCode":"// Before invoking, check the function exists and is reachable\naws lambda get-function --function-name \"$LAMBDA_SEARCH_FUNCTION\" --region \"$AWS_REGION\"\n// and confirm IAM policy allows lambda:InvokeFunction on the function ARN","typeGuard":"fn is_lambda_timeout(err: &aws_sdk_lambda::Error) -> bool {\n    matches!(\n        err,\n        aws_sdk_lambda::error::InvokeError::EfsMountTimeoutException(_)\n            | aws_sdk_lambda::error::InvokeError::SnapStartTimeoutException(_)\n    )\n}","tryCatchPattern":"match invoke_error_to_lambda_error(service_error) {\n    LambdaInvokeError::Timeout(msg) => {\n        // transient: retry with exponential backoff\n        tracing::warn!(%msg, \"lambda invoke timed out, retrying\");\n        backoff_retry()\n    }\n    LambdaInvokeError::Permanent(SearchError::Internal(msg)) => {\n        // configuration/permission problem: do not retry, surface to caller\n        tracing::error!(%msg, \"lambda invoke failed permanently\");\n        return Err(anyhow!(msg));\n    }\n}","preventionTips":["Pin and validate the Lambda function name/ARN in Quickwit config at startup with a get-function check.","Grant lambda:InvokeFunction explicitly to the Quickwit instance role (least privilege on the exact ARN).","Keep search request payloads well under the 6 MB synchronous invoke limit; batch or shard large requests.","Keep the Lambda in the same region and ensure VPC/endpoints allow egress from Quickwit nodes.","Alert on this error's Permanent variant; retrying configuration errors only masks the problem."],"tags":["aws","lambda","grpc","network","serverless"],"backgroundTag":"lambda-invocation-failed","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}