Actually Works · Episode 04

35 / 38

This video almost shipped broken

Three real defects reached us before a person caught them, on our own pipeline. None was caught by a tool. That's why every check here now checks the check before it.

There's a part 2: A check said it passed. It lied. — the fix for exactly this.

Watch on YouTube instead

The exact clicks

  1. Before trusting any automated check, feed it a case you already know is broken.
  2. Confirm the check actually flags that known-bad case — not just that it runs without error.
  3. Only then trust it on real, unknown cases.
  4. Re-run this test after any change to the check itself, not just after changes to what it checks.

What changes

  • A level check measured the wrong thing — it read a position in the file, not whether the sound had actually landed.
  • A check ran before the fix it was supposed to verify, so it always passed.
  • A script proved a file existed by checking its path, not its contents — the file was empty.
  • None of these three were caught by a tool. A person listening, reading, and checking by hand caught all three.

What it will not do

  • Testing a check against a known-bad case does not guarantee it catches every bad case — only the ones like the one you tested.
  • This is a discipline, not a one-time fix. The three defects here happened after checks already existed; the checks just weren't checked.

Get the next one

One AI setup a week, straight to your inbox.

Also buildingAgently— the catalog for AI agents