Taming Recalcitrant Tooling: A Weekend Down the Playwright Rabbit Hole

How I gave my Playwright suite better reporting, smoke tests for every user role, and survived a few tooling gremlins along the way.

Actions on one side, assertions on the other — the whole framework is built around that one split.

I handle QA work during the week, and automation on weekends. It’s a rhythm that fits my current situation — weekdays are for the day-to-day testing grind, and Saturdays are when I get to put on my “engineer” hat and make the boring parts of the job disappear.

This past weekend, I set out to do three things on the Playwright framework I maintain for an internal ERP:

• Take a proper look at the architecture again
• Give my test reports a serious glow-up
• Finally, build a smoke suite that checks the app across every user role

Here’s how it went.

First, the framework (a quick tour)

I won’t pretend I designed something revolutionary. It’s a pretty standard Page Object Model setup — but there are a couple of choices I’m genuinely happy with.

The move that actually saved my sanity

Here’s where I went differently: I split actions and assertions into completely separate classes. Sounds boring, right? It’s not.

One file has all the clicks and fills — ReimbursementPage, LoginPage, whatever. The other file has all the expects — ReimbursementAssertions, LoginAssertions. They each extend a shared base that handles the boring stuff like "wait for the page to actually load."

Why does this matter? Because now, when someone reads a test, it’s obvious what’s happening:

await reimbursementPage.submitRequest(data);
await reimbursementAssertions.assertSubmitted(code);

No detective work. No hunting through 400 lines of page object trying to figure out if a method is changing state or just verifying it. It’s clear.

And clarity, honestly, is half the battle when you’re shipping automation that other people have to touch.

The fixtures layer

The other thing I lean on hard is fixtures. There’s one file, utils/test-fixtures.ts, that wires every page object and assertion object into Playwright's test.

So my specs never do new SomethingPage(page) — they just ask for what they need:

import { test, expect } from '../../utils/test-fixtures';

test('submit reimbursement', async ({ reimbursementPage, reimbursementAssertions }) => {
  // ...
});

Everything else is boring in a good way:

• A FakerFactory so nobody calls Faker directly in a test
• A Winston logger instead of console.log
• A helpers.ts for dates and file uploads
• Joi schemas for validating data

Rules I set for myself so future-me doesn’t make a mess.

Making my reports less ugly

The built-in Playwright HTML report is fine. Allure is genuinely nice for a quick, offline look at a single run.

But the moment you want history — “Is this test getting flakier over time?”, “What did last week’s run look like?” — a single HTML file on your laptop doesn’t cut it.

I didn’t want to stand up a whole CI pipeline just to get a dashboard, though. My automation happens on my machine, on my terms. So I went hunting for something that could take my local runs and give me a proper cloud dashboard with trends and history.

That’s when I found TestDino.

The real payoff

Here’s where it gets practical. I needed TestDino to actually see my reports, so I added a JSON reporter to playwright.config.ts — nothing fancy, just tell Playwright "drop a report.json after every run." Then I point TestDino's CLI at that folder and boom, it picks it up.

Run your tests locally, upload the report in one command, and suddenly you’ve got a cloud dashboard with history and trends. No CI pipeline needed. Just local runs + upload.

Reports that don’t make you squint are worth the effort of setting them up.

The gotcha that wasted twenty minutes

One thing threw me: the HTML reporter has to come before the JSON reporter in your config. HTML wipes its output folder on every run, so if JSON goes first, it’s gone.

reporter: [
  ['html', { outputFolder: 'playwright-report', open: 'never' }],
  ['json', { outputFile: 'playwright-report/report.json' }], // must come after html
],

Uploaded straight from a local run, no CI in sight — pass/fail, history, and trends in one place.

The CLI that fought back

Then came the part where I burned twenty minutes on a maddeningly silly problem.

The docs said the CLI reads your API token from an environment variable. It does not — at least not in the version I had. It kept insisting the token was missing, even though it was right there in my .env.

Turns out I had to pass it explicitly with a --token flag. And when I did that, it rejected the token anyway, grumbling about an extra prefix it didn't like.

Rather than hand-hold a finicky tool on every run, I did the sensible thing: wrote a small cross-platform wrapper that reads the token, strips the offending prefix, and hands the CLI the exact command it wants.

One npm run testdino:upload, and the whole headache was gone.

When it finally worked, and my run showed up in the dashboard — screenshots, pass/fail breakdown, all of it — it was genuinely satisfying.

Smoke-testing the app across every role

This was the part I was most excited about.

Our ERP behaves very differently depending on who’s logged in. An admin sees a huge sidebar; an applicant sees a fraction of it. A smoke test that only logs in as one user is, frankly, a misleading measure of coverage.

So I built a multi-role smoke suite. It has two moving parts.

Part 1: The session setup

A setup step logs in as each role once and saves the session. There are six role labels, but only five real accounts (two of the labels share a login), so it’s smart enough to log in five times instead of six.

Small optimization, but it shaves real seconds off every run.

Part 2: The data-driven test suite

The actual smoke test is data-driven off a menu-map.json — basically a snapshot of every menu path each role is allowed to see. For every role, for every path in its menu, it generates one test that visits the page and checks a few cheap-but-meaningful things:

• Did we get bounced back to /login? (session died = fail)
• Is the page’s heading or breadcrumb actually there and non-empty? (page rendered = pass)
• If the page has a table, did it show rows or a proper “no data” empty state? (not a broken half-render)
• Did the browser throw any uncaught JS errors while loading?

All of those are soft assertions, so one broken page doesn’t abort the whole run. I get the full picture of everything that’s off in a single pass, not just the first failure.

The drift guard

There’s also a little drift guard baked in: each role’s test compares the live sidebar (read straight from the app’s store) against the menu-map.json snapshot, and warns me if they've drifted apart.

If the product team adds a menu item and I forget to update the map, the suite nudges me instead of silently under-testing.

The result

End result: 144 tests across all roles, green, in about eleven minutes — from a single command. That’s coverage I’d never get by hand without losing an entire afternoon and my sanity.

End result: 144 tests across all roles, green, in about eleven minutes — from a single command.

The parts that fought back

Not everything went down without a fight.

The CLI stubbornness

The upload CLI turned out to be gloriously stubborn. Its own docs claimed it would read my API token from an environment variable, but it flatly refused to see it.

When I gave in and passed the token explicitly, it rejected the value anyway, grumbling about an extra prefix it didn’t like.

Rather than hand-hold a finicky tool on every run, I did the sensible thing: wrote a small cross-platform wrapper that reads the token, strips the offending prefix, and hands the CLI the exact command it wants.

One npm run, and the whole headache was gone.

The self-inflicted problems

The other headaches were self-inflicted — which is somehow both more embarrassing and easier to fix.

At one point, I let the token flash across my terminal in plaintext, so I rotated it on the spot. An exposed credential isn’t something you leave lying around.

Then there was the classic git mess: my local main had drifted way behind the remote, which showed up as phantom “missing script” errors, while ESLint complained about carriage returns on every single line.

The fix was mercifully boring — resync, migrate ESLint to its flat config, and normalize line endings with a .gitattributes file plus a one-line Prettier tweak.

By the time the dust settled, I’d opened and merged three pull requests in a single day — the TestDino reporting, a line-ending fix, and a handful of test tweaks — then deleted the stale branches so the repo wouldn’t turn into a graveyard of abandoned work.

Not a bad haul for a Saturday.

What I took away from it

Two things, really.

First, good reports are cheap and worth it. I put off the reporting upgrade for weeks because it felt like a “nice to have.” It took one weekend afternoon, and now I actually look forward to checking results instead of scrolling through a terminal.

Second, tooling friction is normal — so don’t take it personally. Half my day went to a CLI that ignored its own documentation. That’s not a sign you’re doing it wrong; it’s just Saturday. Wrap the annoying thing in a script, write down the gotcha, and move on.

Next weekend, I want to get this smoke suite running on a schedule so I’m not the one kicking it off. But that’s a problem for future-me.

TL;DR

• Framework: Playwright + TypeScript, Page Object Model with a clean split — Page classes do actions, Assertions classes do the checks, all wired through one fixtures file, so specs stay tidy.
• Reporting: Kept Allure for quick local looks, added TestDino for a cloud dashboard with history and trends. Works entirely from local runs.

On automation, judgment, and what actually scales

There’s a thought I keep coming back to whenever someone insists AI is about to make QA obsolete.

It goes like this:

“AI can generate a thousand test cases before your coffee cools, but it still takes a human to ask the one question the machine never thought to. Automation scales the effort; judgment is what scales the trust.”

The tools get sharper every year. I watched that happen during my weekend with Playwright — better reporters, faster test runners. But here’s what doesn’t scale faster: the judgment about what’s actually worth testing.

Think about my multi-role smoke suite. The machine didn’t decide to check the session state across six different user roles. A human did — because a human sat down and thought, “Our app looks completely different depending on who’s logged in. A single-role smoke test is a lie.”

The machine executed that judgment flawlessly. It ran 144 tests in eleven minutes. But it never would have asked why those tests mattered in the first place.

The speed trap

Speed multiplier — that’s all automation is. Your machine runs whatever you fed it flawlessly. But half-baked test design from the start? You just automated mediocrity.

Real QA is human: asking “should we even test this?” and thinking sideways when everyone else is thinking straight. Judgment doesn’t scale with faster runners. It scales with better thinking.

The reporting blind spot

Here’s the uncomfortable part: we’re still treating reporting like it’s incidental.

Static reports are still the norm — HTML files that evaporate the moment you close them. Dynamic dashboards that track trends and flag drift? They exist, just gathering dust while teams squint at yesterday’s terminal output.

Your automation’s only as sharp as the reporting behind it. Garbage reporting, garbage insights. That’s the gap.

What actually matters

I spent this weekend optimizing three things: architecture clarity, report visibility, and coverage breadth. Not one of them was about running tests faster. They were all about thinking better — about what matters, about what changed, about what we’re actually validating.

That’s the part machines still can’t do. And honestly, that’s the part that’s more interesting anyway.

Thanks for reading! If you’re wrestling with similar QA challenges (whether it’s flaky tests, reporting gaps, or coverage blind spots), I’d love to hear how you’re tackling them.