AI Can Write Code Fast. Playwright Tells Me Whether I Can Ship It.

The AI Velocity vs. Regression Risk

Building and refactoring OpsKitPro with AI assistance has fundamentally changed how fast we can ship. Entire modules—like our pure MDX CMS migration, TLS diagnostic live probes, or the new API rate limit guard—were developed, structured, and implemented in days instead of weeks.

But velocity introduces a new kind of risk: regression by side effect.

When AI rewrites a layout component to fix a mobile spacing issue, does it inadvertently break the sticky Knowledge Map navigation on desktop? When it adds a 429 Too Many Requests API contract, does it accidentally trigger a fatal error in the React frontend that was expecting a 200 OK?

AI makes writing code fast. But speed without boundaries is just technical debt waiting to explode in production.

Why Unit Tests Aren't Enough

Unit tests (vitest) are excellent for verifying that our LRUCache rate limiter mathematically increments correctly, or that our checkServerIdentity TLS function correctly parses a SAN mismatch.

But unit tests lie about the user experience.

A unit test doesn't know if the Next.js middleware.ts stripped a locale prefix. It doesn't know if an unhandled hydration mismatch crashed the entire page rendering on the client side.

To trust an AI-assisted release, we need to test the actual product exactly as the user (and search engine) sees it. That's where Playwright comes in.

What the OpsKitPro Smoke Matrix Checks

Instead of writing hundreds of isolated UI tests for every single button, we built a centralized "Smoke Matrix." This matrix is designed to catch the fatal, structural errors that would force an immediate rollback.

Our smoke.spec.ts focuses heavily on:

1. Critical Routes: Can the /tools/website-check page load, accept input, and submit without a console error?
2. SEO Integrity: Does /sitemap.xml return exactly the expected URLs? Do the retired locale redirects (/ja/* → /en/*, /tw/* → /zh/*) correctly execute a 301?
3. i18n Stability: When you switch from English to Simplified Chinese, does the routing matrix hold up without a 404?

The Fear of "Page Loads, But Critical Path is Broken"

For a diagnostic tool site, the worst bug isn't a 500 error page—it's a tool that silently fails to provide a result.

If a user runs the Website Check and hits our backend rate limits (which we artificially enforce at 3 req/min), the test needs to handle it. In our architecture, rate-limit behavior is covered through controlled test paths or mocked limiter states, while production smoke tests verify the public 429 contract. Our Playwright setup doesn't just assert expect(page).toHaveTitle('...'); it asserts that the complex, interactive diagnostic UI renders the three-state findings (🟢 Pass, 🔴 Fail, ⚪ Unknown) correctly, regardless of the network state.

A Quick Anecdote: When Content Breaks Tests

Interestingly, while writing this very article, our Playwright test suite broke.

It wasn't because the application was broken, but because adding this new article altered the UI semantics. Our E2E test originally looked for the Knowledge Map section using page.getByRole('link', { name: /AI engineering/i }). This worked fine until this article appeared on the page, introducing a new card that also matched the AI engineering string!

Playwright caught this "strict mode violation" immediately. It forced us to realize that as content grows, our test selectors need to be more precise. E2E tests don't just catch code bugs—they catch assumptions that no longer hold true as the product scales.

Putting Playwright in the Pre-Release Path

The workflow is now rigid by design:

1. Describe the goal and boundary to the AI.
2. AI generates the implementation.
3. Run npm run verify:fast (which includes unit tests, build checks, and static analysis).
4. Run npm run test:e2e (Playwright smoke matrix).

If the smoke matrix passes, we ship it. If it fails, we paste the Playwright trace back to the AI and say, "You broke the locale redirect. Fix it."

The goal is not coverage. The goal is shipping confidence.

We aren't chasing 100% E2E test coverage. Writing Playwright tests for every micro-interaction is fragile and slows down development.

Instead, we use Playwright to establish a confidence baseline. It tells us that the core application architecture, the localization routing, the API boundaries, and the SEO redirects are intact.

AI writes the code. Playwright tells us if we can confidently deploy it to production.