Field Guide · Craft skills
Evaluate without theater
Lightweight evaluation habits that catch real failures without fake benchmark theater.
Start with the job, not the leaderboard
Public benchmarks are signals, not contracts. For your work, define three failing examples that would hurt if wrong - then keep them.
A weekly eval loop
- Pick one workflow you already run with a model.
- Save five inputs and the outputs you accepted.
- Change one variable (model, prompt, tool, temperature).
- Score pass/fail against your own bar - not vibes.
What to write down
- What must be true for a pass
- What a skeptic would challenge
- Whether a human still owns the last mile
If you cannot name those three, you are not evaluating - you are browsing.