Flaky tests are a design problem, not a retry problem

The first thing you reach for when a test fails once in ten runs is a retry. It works, in the sense that the pipeline turns green. It also means the suite now reports a state that nobody has verified.

Where the flakiness actually came from

Going back through a quarter of failures in our Playwright suite, almost all of them fell into three groups.

1. Waiting on a duration rather than a condition. A fixed waitForTimeout is a bet on the slowest machine that will ever run the test.
2. Sharing data between specs. Two tests that both edit the same record pass alone and fail together, and which one fails depends on scheduling.
3. Asserting on rendering rather than on state. A row appearing in the DOM is not the same claim as the record existing.

What replaced them

Locators that describe intent, not structure:

// brittle: breaks when the table gains a column
await page.locator('table tr:nth-child(2) td:nth-child(3)').click();

// stable: describes what the user is looking for
await page.getByRole('row', { name: /INV-0042/ })
          .getByRole('button', { name: 'Approve' })
          .click();

Web-first assertions, which retry the assertion instead of retrying the whole test:

await expect(page.getByRole('status')).toHaveText('Approved');

And fixtures that create their own data and remove it afterwards, so any spec can run alone, in any order, on a machine that has never seen the app.

The part that is not technical

A flaky test costs more than a failing one, because it teaches the team to disbelieve the suite. Once people start re-running rather than reading, the suite has stopped being evidence. That is the real cost, and no retry setting fixes it.