NationalSecurityAgency/ghidra · warning · DatabaseNonFatalException

is already ingested

Error message

 is already ingested

What it means

When a non-library executable's metadata matches an existing record, `testExecutableDuplication` throws DatabaseNonFatalException("<name> is already ingested"). It is non-fatal because the overall insert can proceed with other executables; this one is a duplicate that should be skipped. BSim tolerates re-ingesting libraries (partial multi-inserts) but flags duplicate main executables.

Source

Thrown at Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/AbstractSQLFunctionDatabase.java:1070

						throw new LSHException(fatalerror);
					}
					throw new DatabaseNonFatalException(
						FunctionDatabase.constructNonfatalError(cmp, erec, tmp));
				}
				if (erec.getRowId() != null) {
					if (!erec.getRowId().equals(tmp.getRowId())) {
						throw new LSHException(
							"Id mismatch when inserting executable: " + erec.getNameExec());
					}
				}
				else {
					input.setExeRowId(erec, tmp.getRowId());
				}
				input.setExeAlreadyStored(erec);
				// Just because we've seen a library before doesn't mean we should stop function insertion
				// Libraries are likely to be partially inserted multiple times
				if (!erec.isLibrary()) {
					throw new DatabaseNonFatalException(
						erec.getNameExec() + " is already ingested");
				}
				pickout_storedfuncs = true; // At least one executable has previously inserted functions
			}
		}

		if (pickout_storedfuncs) {
			if (!markPreviouslyStoredFunctions(input, input.listAllFunctions())) {
				throw new DatabaseNonFatalException("Already inserted");
			}
		}
	}

	/**
	 * Do the final work of inserting new ExecutableRecords into the database. This function
	 * assumes testExecutableDuplication has already run and marked previously ingested records
	 * 
	 * @param input the executable descriptor

View on GitHub (pinned to d5f144c24d)

Solutions

1. Treat DatabaseNonFatalException as an expected, recoverable skip rather than a hard failure.
2. Deduplicate the input executable list before ingest.
3. Track which executable MD5s are already stored and exclude them.

Example fix

// before: insert throws non-fatal on a known duplicate
try { db.insert(manager); }
  catch (DatabaseNonFatalException e) { /* unhandled, aborts batch */ }
// after: classify non-fatal separately and continue
try { db.insert(manager); }
catch (DatabaseNonFatalException e) {
    log.info("skipping duplicate exe: {}", e.getMessage());
}

// Detect already-ingested executables by MD5 before attempting insert.
for (ExecutableRecord erec : manager.getAllExecutables()) {
    String md5 = erec.getMd5();
    if (exeTable.queryMd5ExeMatch(md5) != null && !erec.isLibrary()) {
        // will be flagged non-fatal; skip or accept the skip downstream
        log.info("already ingested: {}", erec.getNameExec());
    }
}

try {
    db.insert(manager);
} catch (DatabaseNonFatalException e) {
    // 'X is already ingested' is expected for duplicate main executables -- continue
    log.info("skipping duplicate executable: {}", e.getMessage());
} catch (LSHException | SQLException e) {
    throw e; // real errors propagate
}

When it happens

Trigger: Inserting a main executable (not a library) whose metadata already matches a stored record. The metadata `compareMetadata` returned 0 (no fatal/non-fatal diff), IDs agree, but it is not a library, so re-ingestion is flagged.

Common situations: Re-running an ingest over the same binary set; CI pipelines re-ingesting; idempotency assumption that the DB will silently no-op.

Related errors

• Already inserted
• Id mismatch when inserting executable:
• Trying to insert on read-only database
• Did not get desctable sequence number after insertion
• Did not get exetable sequence number after insertion

AI-assisted analysis of NationalSecurityAgency/ghidra@d5f144c24d (2026-08-14). Data as JSON: /api/errors/82ba94ce05207c7a. Report an issue: GitHub.