{"record":{"id":"f91c29f4877663db","repo":"NationalSecurityAgency/ghidra","slug":"is-already-ingested-f91c29","errorCode":null,"errorMessage":" is already ingested","messagePattern":" is already ingested","errorType":"exception","errorClass":"DatabaseNonFatalException","httpStatus":null,"severity":"warning","filePath":"Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java","lineNumber":1637,"sourceCode":"\t\t\tthrows ElasticException, LSHException, DatabaseNonFatalException {\n\t\tRowKeyElastic eKey = updateKey(manager, exeRecord);\n\t\tString exeId = eKey.generateExeIdString();\n\n\t\tif (!insertExecutableRecord(exeRecord, exeId)) {\t// Try to insert the executable\n\t\t\tJsonObject exeObj = queryMd5ExeMatch(exeRecord.getMd5());\n\t\t\tif (exeObj != null) {\t\t// Try to retrieve the previous version\n\t\t\t\tExecutableRecord oldrec = makeExecutableRecordTemp(exeObj);\n\t\t\t\tint cmp = oldrec.compareMetadata(exeRecord);\n\t\t\t\tif (cmp != 0) {\n\t\t\t\t\tString fatalerror =\n\t\t\t\t\t\tFunctionDatabase.constructFatalError(cmp, exeRecord, oldrec);\n\t\t\t\t\tif (fatalerror != null) {\n\t\t\t\t\t\tthrow new LSHException(fatalerror);\n\t\t\t\t\t}\n\t\t\t\t\tthrow new DatabaseNonFatalException(\n\t\t\t\t\t\tFunctionDatabase.constructNonfatalError(cmp, exeRecord, oldrec));\n\t\t\t\t}\n\t\t\t\tthrow new DatabaseNonFatalException(\n\t\t\t\t\texeRecord.getNameExec() + \" is already ingested\");\n\t\t\t}\n\t\t\treturn false;\t\t// Indicate this executable already inserted\n\t\t}\n\t\tint newIds = 0;\n\t\tlong baseId = 0;\n\t\tIterator<FunctionDescription> iter = manager.listFunctions(exeRecord);\n\t\twhile (iter.hasNext()) {\t\t\t// Count the functions to insert for this executable\n\t\t\titer.next();\n\t\t\tnewIds += 1;\n\t\t}\n\t\tbaseId = allocateFunctionIndexSpace(newIds);\t\t// Allocated the ids we will need\n\t\titer = manager.listFunctions(exeRecord);\n\t\twhile (iter.hasNext()) {\n\t\t\tmanager.setFunctionDescriptionId(iter.next(), new RowKeyElastic(baseId));\t// Set the (allocated) ids\n\t\t\tbaseId += 1;\n\t\t}\n\t\t// Collect/dedup vectors and update SignatureRecords with vector ids, before writing FunctionDescriptions","sourceCodeStart":1619,"sourceCodeEnd":1655,"githubUrl":"https://github.com/NationalSecurityAgency/ghidra/blob/d5f144c24d6bc53c9cbf4448c6d11143e7696206/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java#L1619-L1655","documentation":"Thrown as a DatabaseNonFatalException in insertExe when an executable already exists in the database with identical metadata. insertExecutableRecord uses op_type=create, which fails on the first insert; the existing record is then retrieved via queryMd5ExeMatch and compareMetadata returns 0 (exact match). This is expected, informational behavior signaling a no-op re-ingestion, not a genuine error.","triggerScenarios":"Calling insertExe (via an ingest command) for an executable whose md5 already exists. The version_conflict from op_type=create is caught, the existing record is fetched, and compareMetadata returns cmp==0 (metadata is identical), so the non-fatal exception is thrown.","commonSituations":"Re-running an ingest script that processes the same binaries; batch ingestion pipeline that includes previously-seen executables; re-ingesting after a partial failure where the executable record was already written.","solutions":["Catch DatabaseNonFatalException at the call site and treat it as informational — skip the executable and continue the batch.","Pre-check for an existing executable via queryMd5ExeMatch before attempting ingestion to avoid the exception entirely.","Log the skipped executable name and proceed with the next item in the batch."],"exampleFix":"// before\ndatabase.query(query);  // DatabaseNonFatalException propagates uncaught\n\n// after\ntry {\n    database.query(query);\n} catch (DatabaseNonFatalException e) {\n    Msg.info(this, \"Skipping already-ingested executable: \" + e.getMessage());\n}","handlingStrategy":"try-catch","validationCode":"// Pre-check before ingestion\nJsonObject existing = queryMd5ExeMatch(exeRecord.getMd5());\nif (existing != null) {\n    // Executable already exists; skip ingestion\n    Msg.info(this, \"Executable already ingested, skipping: \" + exeRecord.getNameExec());\n    return;\n}","typeGuard":null,"tryCatchPattern":"try {\n    database.query(query);\n} catch (DatabaseNonFatalException e) {\n    if (e.getMessage().endsWith(\" is already ingested\")) {\n        // Expected — skip this executable in the batch\n        Msg.info(this, \"Already ingested: \" + e.getMessage());\n        continue;\n    }\n    throw e;\n}","preventionTips":["In batch ingestion scripts, catch DatabaseNonFatalException and continue to the next executable.","Pre-check with queryMd5ExeMatch before attempting ingestion to avoid the exception overhead.","Track ingested md5 values locally to skip duplicates before hitting the server."],"tags":["bsim","ingest","non-fatal","duplicate","expected-behavior","op-type-create"],"backgroundTag":null,"analyzedSha":"d5f144c24d6bc53c9cbf4448c6d11143e7696206","analyzedAt":"2026-08-14T01:00:57.564Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}