{"record":{"id":"5d886e19d543b9c6","repo":"elastic/elasticsearch","slug":"failure-store-document-has-unexpected-structure-m-5d886e","errorCode":null,"errorMessage":"failure store document has unexpected structure, missing required [error] field","messagePattern":"failure store document has unexpected structure, missing required \\[error\\] field","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/RecoverFailureDocumentProcessor.java","lineNumber":62,"sourceCode":"    public static final String TYPE = \"recover_failure_document\";\n\n    RecoverFailureDocumentProcessor(String tag, String description) {\n        super(tag, description);\n    }\n\n    @Override\n    @SuppressWarnings(\"unchecked\")\n    public IngestDocument execute(IngestDocument document) throws Exception {\n        if (document.hasField(DOCUMENT_FIELD) == false) {\n            throw new IllegalArgumentException(MISSING_DOCUMENT_ERROR_MSG);\n        }\n\n        if (document.hasField(SOURCE_FIELD_PATH) == false) {\n            throw new IllegalArgumentException(MISSING_SOURCE_ERROR_MSG);\n        }\n\n        if (document.hasField(ERROR_FIELD) == false) {\n            throw new IllegalArgumentException(MISSING_ERROR_ERROR_MSG);\n        }\n\n        // store pre-recovery data in ingest metadata\n        storePreRecoveryData(document);\n\n        // Get the nested 'document' field, which holds the original document and metadata.\n        Map<String, Object> failedDocument = (Map<String, Object>) document.getFieldValue(DOCUMENT_FIELD, Map.class);\n\n        // Copy the original index, routing, and id back to the document's metadata.\n        String originalIndex = (String) failedDocument.get(INDEX_FIELD);\n        if (originalIndex != null) {\n            document.setFieldValue(IngestDocument.Metadata.INDEX.getFieldName(), originalIndex);\n        }\n\n        String originalRouting = (String) failedDocument.get(ROUTING_FIELD);\n        if (originalRouting != null) {\n            document.setFieldValue(IngestDocument.Metadata.ROUTING.getFieldName(), originalRouting);\n        }","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/elastic/elasticsearch/blob/db6a809a667c081ca1dc7500389d26975573215f/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/RecoverFailureDocumentProcessor.java#L44-L80","documentation":"Thrown by RecoverFailureDocumentProcessor.execute when the IngestDocument lacks the 'error' field. The processor needs the 'error' metadata (along with 'document' and 'document.source') to fully reconstruct a failed document for reprocessing.","triggerScenarios":"Running the recover_failure_document processor on a document that has 'document' and 'document.source' but is missing the top-level 'error' key describing the original failure.","commonSituations":"Partial failure-store entries where the error metadata was stripped or never written. Custom failure pipelines that drop the error field before recovery. Version mismatch between failure-store writer and reader.","solutions":["Confirm the failure-store entry retains its 'error' field through any intermediate processing.","Avoid stripping the 'error' key in upstream processors when this recovery processor runs downstream.","Validate the three required fields (document, document.source, error) before submitting to the pipeline."],"exampleFix":"// before: error field removed by an upstream script processor\n// after: preserve the error field\nif (ingestDocument.hasField(\"error\") == false) {\n    throw new IllegalStateException(\"failure-store entry missing error metadata\");\n}","handlingStrategy":"validation","validationCode":"// Verify the 'error' field is present before recovery\nif (document.hasField(\"error\") == false) {\n    // not a complete failure-store entry; do not run recovery\n    return;\n}","typeGuard":"boolean hasErrorMetadata(IngestDocument doc) {\n    return doc.hasField(\"error\");\n}","tryCatchPattern":"try {\n    // run pipeline\n} catch (IllegalArgumentException e) {\n    if (e.getMessage().contains(\"missing required [error]\")) {\n        // route to dead-letter; failure-store entry is incomplete\n    } else { throw e; }\n}","preventionTips":["Never strip the 'error' field in upstream processors when recovery runs downstream.","Audit custom failure pipelines to ensure the error metadata survives.","Test recovery pipelines with full failure-store fixtures."],"tags":["elasticsearch","ingest-pipeline","recover-failure-document","validation"],"backgroundTag":null,"analyzedSha":"db6a809a667c081ca1dc7500389d26975573215f","analyzedAt":"2026-08-12T01:39:14.192Z","schemaVersion":2},"datasetVersion":"2026-08-12T12:31:55.035Z"}