Actually Works · Episode 08
31 / 38A check said it passed. It lied.
It ran clean every time. Nobody had ever given it a case that was actually broken — so "never failed" got mistaken for "works." One test settles which one is true.
Part 2 of This video almost shipped broken — the fix for what that episode named but didn't solve.
The exact clicks
- Take the automated check you're currently trusting — a test, a QA script, a validation step.
- Write one input you already know is broken, on purpose.
- Run the check on that broken input, not on good input.
- Confirm it actually fails. If it doesn't, the check has never been asked the real question.
What changes
- A check that always passes on real input looks identical to a working check — the only way to tell them apart is to feed it something you know is wrong.
- "Never failed" is not the same claim as "works" — it can just mean the check was never tested against a failure.
What it will not do
- Passing on one known-bad case doesn't prove the check catches every bad case — only that it isn't blind to that one.
- This has to be repeated after any change to the check itself, not just after changes to what it checks.
Get the next one
One AI setup a week, straight to your inbox.