Taya Yakovenko Get in touch
← All projects
Eval study

BlogAI Evals

An evaluation of whether the BlogAI pipeline keeps what matters when notes become a post: the user’s argument and their voice. The question underneath is whether the cheap Qwen→Haiku pipeline holds up against Haiku on its own.

GitHub ↗

Problem

BlogAI is supposed to amplify your thinking, not summarize the article. Whether it actually does that is the kind of claim these tools make and never check. The failure that matters is quiet: the model keeps the topic and the headline claim, then drops the reasoning that made the point worth writing. A single quality score never catches it.

Approach

Measure substance and voice as two separate things, and calibrate both before trusting either. Substance fidelity scores whether the argument survives. Voice fidelity scores whether it still sounds like the author. Both were validated against human-edited gold and a null floor first, then the conditions were compared with paired statistics across 27 real inputs.

What I built

  • Substance fidelity types each note sentence by argument role (claim, evidence, logic, implication), embeds it with all-mpnet-base-v2, and matches it to the output by cosine similarity. A flattening flag fires when the claim survives but the reasoning behind it does not.
  • Voice fidelity runs a two-tier regex and spaCy rubric: surface markers in tier one, structural patterns in tier two. The gap between the tiers is the signal. Passing tier one without tier two means the model copied the surface and missed the voice.
  • The study sat behind a calibration gate. Five human-edited gold outputs and five null baselines set the ceiling and floor, and both rubrics had to clear Spearman ρ ≥ 0.5 before any generation ran (substance ρ = 0.926, voice ρ = 0.876).
  • A factual-consistency floor check (BM25 retrieval into DeBERTa-v3 NLI), per-call cost tracking, and resumable generation across the HuggingFace and Anthropic APIs.

Result

Across 216 outputs, the cheap Qwen→Haiku pipeline matched or beat Haiku alone on substance (LinkedIn: 0.663 vs 0.624, paired Wilcoxon p = 0.030), while the Haiku edit pass lowered substance rather than raising it. That is a usable design finding for about eight cents of total spend. The full report carries the effect sizes, the limitations, and a LinkedIn rubric the data showed was miscalibrated.