Actually Works · Episode 08

31 / 38

A check said it passed. It lied.

It ran clean every time. Nobody had ever given it a case that was actually broken — so "never failed" got mistaken for "works." One test settles which one is true.

Part 2 of This video almost shipped broken — the fix for what that episode named but didn't solve.

Watch on YouTube instead

The exact clicks

  1. Take the automated check you're currently trusting — a test, a QA script, a validation step.
  2. Write one input you already know is broken, on purpose.
  3. Run the check on that broken input, not on good input.
  4. Confirm it actually fails. If it doesn't, the check has never been asked the real question.

What changes

  • A check that always passes on real input looks identical to a working check — the only way to tell them apart is to feed it something you know is wrong.
  • "Never failed" is not the same claim as "works" — it can just mean the check was never tested against a failure.

What it will not do

  • Passing on one known-bad case doesn't prove the check catches every bad case — only that it isn't blind to that one.
  • This has to be repeated after any change to the check itself, not just after changes to what it checks.

Get the next one

One AI setup a week, straight to your inbox.

Also buildingAgently— the catalog for AI agents