Model evaluation
Recorded validation evidence for base and fine-tuned classifiers.
| Model | Mode | Accuracy | Macro F1 | Valid outputs | Examples |
|---|---|---|---|---|---|
| Base | Generative | 0.0% | 0.0% | 0.0% | 582 |
| Fine-tuned | Generative | 63.2% | 65.4% | 94.2% | 582 |
| Base | Scored | 39.5% | 28.2% | 100.0% | 582 |
| Fine-tuned | Scored | 67.5% | 67.8% | 100.0% | 582 |
Scored macro F1
+39.6 pp
95% CI +35.4
to +43.9
Accuracy
+28.0 pp
95% CI +23.2
to +33.2
Brier score
−0.196
Lower is better
| Class | Base recall | Fine-tuned recall | Delta | 95% CI |
|---|---|---|---|---|
| bug | 69.6% | 75.7% | +6.1 pp | −0.7 to +14.2 |
| documentation | 0.7% | 73.2% | +72.5 pp | +64.7 to +79.5 |
| feature_request | 92.4% | 65.9% | −26.5 pp | −35.0 to −18.5 |
| question | 2.5% | 56.2% | +53.8 pp | +46.0 to +61.7 |
Threshold tradeoff (scored)
Escalate at ≤30.7% confidence
≤30.7%
20.1% escalated
117 / 582
73.5% retained accuracy
73.5%
66 of 189 validation errors captured
66 / 189
Gate decision
Original gate: failed
Failure reason: feature_request recall regressed by more than 0.05
Explicit override: approved
run_a remains deployed
Provenance
Validation split
val
Examples
582
Seed
42
Bootstrap resamples
1,000
Base model
Qwen/Qwen3-0.6B
Adapter
atakan99/repomedic-qwen3-0.6b-lora-run-a