Accuracy & methodology
How we measure accuracy.
We want a system that catches real problems while keeping false alarms low enough that your team isn’t dismissing noise. Here’s how we measure that, and the thresholds we hold ourselves to.
Weights recall over precision
Keeps false alarms manageable
Inter-reviewer consistency baseline
What we measure
Recall
Of all the real problems, how many did the system catch?
Precision
Of all the things flagged, how many were truly problems?
F2-score
A harmonic mean of precision and recall that prioritizes catching problems over avoiding false alarms.
The four outcomes
Every finding lands in one of four buckets when we compare the system against a human reviewer.
| Human reviewer | System | Outcome |
|---|---|---|
| Red flag present | Flagged | HitTrue Positive |
| Red flag present | Not flagged | MissFalse Negative |
| No red flag | Flagged | False alarmFalse Positive |
| No red flag | Not flagged | Correct blankTrue Negative |
The math
Recall
TP ÷ (TP + FN)
Hits ÷ (Hits + Misses)
Precision
TP ÷ (TP + FP)
Hits ÷ (Hits + False alarms)
F2-score
5 · P · R ÷ (4 · P + R)
Harmonic mean, weighted toward recall
False-positive rate
FP ÷ (FP + TN)
False alarms ÷ (False alarms + Correct blanks)
Why F2-score?
Why not just use raw accuracy?
Because red flags are rare. A system that never flags anything could look “99% accurate” while missing every real problem. F2 focuses on catching problems, which is what QC is for.
Why weight recall more than precision?
Because missing a real problem costs far more than checking a false alarm. A false alarm is 20 to 40 seconds of a reviewer’s time; a missed problem can reach the client. We tune to minimize false alarms so your team isn’t dismissing noise, but never at the expense of catching real issues.
See these numbers on your own surveys.
We calibrate to your protocol and run a batch of your real calls in the demo.