headroomlabs-ai/headroom · error · ValueError
No valid samples processed
Error message
No valid samples processed
What it means
Error "No valid samples processed" thrown in headroomlabs-ai/headroom.
Source
Thrown at headroom/evals/html_oss_benchmarks.py:376
em_original.append(1.0 if em_orig else 0.0)
em_extracted.append(1.0 if em_ext else 0.0)
question_results.append(
{
"question": question,
"ground_truth": ground_truth,
"answer_original": answer_original,
"answer_extracted": answer_extracted,
"f1_original": f1_orig,
"f1_extracted": f1_ext,
}
)
if (i + 1) % 10 == 0:
logger.info(f" Processed {i + 1}/{len(samples)} questions")
if not f1_original:
raise ValueError("No valid samples processed")
avg_f1_orig = sum(f1_original) / len(f1_original)
avg_f1_ext = sum(f1_extracted) / len(f1_extracted)
avg_em_orig = sum(em_original) / len(em_original)
avg_em_ext = sum(em_extracted) / len(em_extracted)
# Accuracy is preserved if extracted is within 2% of original
accuracy_preserved = avg_f1_ext >= avg_f1_orig - 0.02
return QAAccuracyResult(
total_questions=len(f1_original),
accuracy_original_html=avg_f1_orig,
accuracy_extracted=avg_f1_ext,
accuracy_preserved=accuracy_preserved,
avg_f1_original=avg_f1_orig,
avg_f1_extracted=avg_f1_ext,
exact_match_original=avg_em_orig,
exact_match_extracted=avg_em_ext,View on GitHub (pinned to 322425c43b)
Solutions
- Verify the input dataset actually contains samples matching the benchmark filters
- Check earlier log lines for per-sample errors that caused every sample to be skipped
- Relax sample filters (min/max length, required fields) if they exclude everything
- Verify the LLM/API used for processing is reachable so samples are not all failing
When it happens
Trigger: Raised at the end of an HTML OSS benchmark run when every sample failed processing or was filtered out, leaving nothing to score.
Common situations: See trigger scenarios.
AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15).
Data as JSON: /api/errors/2500ce2612671bc3.
Report an issue: GitHub.