{"record":{"id":"e21cb61264828bd5","repo":"elastic/elasticsearch","slug":"failure-store-document-has-unexpected-structure-m","errorCode":null,"errorMessage":"failure store document has unexpected structure, missing required [document] field","messagePattern":"failure store document has unexpected structure, missing required \\[document\\] field","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/RecoverFailureDocumentProcessor.java","lineNumber":54,"sourceCode":"    public static final String ERROR_FIELD = \"error\";\n\n    public static final String MISSING_DOCUMENT_ERROR_MSG =\n        \"failure store document has unexpected structure, missing required [document] field\";\n    public static final String MISSING_SOURCE_ERROR_MSG =\n        \"failure store document has unexpected structure, missing required [document.source] field\";\n    public static final String MISSING_ERROR_ERROR_MSG = \"failure store document has unexpected structure, missing required [error] field\";\n\n    public static final String TYPE = \"recover_failure_document\";\n\n    RecoverFailureDocumentProcessor(String tag, String description) {\n        super(tag, description);\n    }\n\n    @Override\n    @SuppressWarnings(\"unchecked\")\n    public IngestDocument execute(IngestDocument document) throws Exception {\n        if (document.hasField(DOCUMENT_FIELD) == false) {\n            throw new IllegalArgumentException(MISSING_DOCUMENT_ERROR_MSG);\n        }\n\n        if (document.hasField(SOURCE_FIELD_PATH) == false) {\n            throw new IllegalArgumentException(MISSING_SOURCE_ERROR_MSG);\n        }\n\n        if (document.hasField(ERROR_FIELD) == false) {\n            throw new IllegalArgumentException(MISSING_ERROR_ERROR_MSG);\n        }\n\n        // store pre-recovery data in ingest metadata\n        storePreRecoveryData(document);\n\n        // Get the nested 'document' field, which holds the original document and metadata.\n        Map<String, Object> failedDocument = (Map<String, Object>) document.getFieldValue(DOCUMENT_FIELD, Map.class);\n\n        // Copy the original index, routing, and id back to the document's metadata.\n        String originalIndex = (String) failedDocument.get(INDEX_FIELD);","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/elastic/elasticsearch/blob/db6a809a667c081ca1dc7500389d26975573215f/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/RecoverFailureDocumentProcessor.java#L36-L72","documentation":"Thrown by RecoverFailureDocumentProcessor.execute when the failure-store document lacks the required 'document' field. The processor unpacks a failure-store entry that must contain document, source, and error sub-fields; absence of 'document' means the input is not a valid failure document. IllegalArgumentException using MISSING_DOCUMENT_ERROR_MSG constant.","triggerScenarios":"Recover failure processor invoked on a document that does not have the failure-store envelope structure — specifically missing the 'document' key.","commonSituations":"Pipeline misconfiguration running recover_failure_document on regular (non-failure-store) documents; failure store schema changed between versions; manually-crafted failure documents missing fields.","solutions":["Only route genuine failure-store documents to the recover_failure_document processor.","Verify the failure store schema has document/source/error fields (use GET on a sample failure doc to inspect).","If ingesting non-failure data, remove the recover_failure_document processor from that pipeline."],"exampleFix":"// before\n{\"recover_failure_document\": {}}\n// applied to a regular doc with no 'document' field\n// after - gate the processor:\n{\"recover_failure_document\": {\"if\": \"ctx.containsKey('document') && ctx.containsKey('error')\"}}","handlingStrategy":"type-guard","validationCode":"if (!doc.hasField(\"document\")) {\n    // do not route this document through recover_failure_document\n}","typeGuard":"static boolean isFailureStoreEnvelope(IngestDocument doc) {\n    return doc.hasField(\"document\") && doc.hasField(\"source\") && doc.hasField(\"error\");\n}","tryCatchPattern":"try {\n    recoverProcessor.execute(doc);\n} catch (IllegalArgumentException e) {\n    if (e.getMessage().contains(\"missing required [document] field\")) {\n        // skip — not a failure-store document\n    } else throw e;\n}","preventionTips":["Only route genuine failure-store documents to recover_failure_document.","Add a conditional checking for the document/error/source fields before the processor.","Inspect a sample failure document to confirm the expected envelope schema."],"tags":["ingest","failure-store","recover","schema"],"analyzedSha":"db6a809a667c081ca1dc7500389d26975573215f","analyzedAt":"2026-08-12T01:39:14.192Z","schemaVersion":2},"datasetVersion":"2026-08-12T12:17:08.281Z"}