How agentic loops built on Sigma migrated 1,000+ tests to Playwright in a month
Engineering has changed. When coding agents can write and modify code faster than any team can review it by hand, the only way to trust that code is with fast, reliable, high-fidelity test feedback.
At Sigma, we built an agentic loop to scale out our test feedback. It’s a system that lets AI agents migrate our entire end-to-end test suite to Playwright—the open-source, cross-browser testing framework—without a dedicated QE team.
With this system, we migrated more than 1,000 tests in just four weeks. I've led test framework migrations before, and I'd never seen one move like this.
Read on to learn how we built that loop, how we used Sigma to power and monitor it, and what it delivered once it was running end to end.
Why Playwright, and why we couldn't wait for a QE team
We started by converting a handful of our existing tests to Playwright. Debugging got easier, parallelization made the same tests run 4x faster out of the box, and flakiness dropped compared to our legacy framework. Playwright was the right call, and we needed to move fast: with thousands of complex E2E tests to migrate, a manual effort could have taken quarters and a dedicated Quality Engineering team we didn't have.
As Sigma's first Test Engineer, I worked with Kevin Pham and Alison Lai from our Developer Experience and Web Infrastructure teams to rethink what an org-wide migration could look like without one. That rethink became an agentic loop: a harness that let AI agents translate, run, review, and re-test each spec until it met our bar, with computational and Sigma-powered checks keeping the whole thing honest.
Give agents a stable foundation before you scale them
Telling an AI agent to “migrate our E2E tests to Playwright” would have produced something. It wouldn't have produced tests at the quality, scalability, and reliability our development teams needed to trust their own changes.
Agents needed a stable foundation so they weren't reinventing basic functionality: authentication, working with our product, and the context to understand what a given feature was actually testing. I iterated on an initial Playwright exploration and built our fixtures, parallelization patterns, and page objects around each feature's golden path. That work drew clear boundaries for any agentic work that followed.
It also became the direct translation layer for two skills: our Playwright Migration Skill and our Playwright Review Skill.
The agentic loop: translate, run, review, repeat
An example migration prompt breaks down into six steps:
- Translate it. Map the auth/setup pattern to the equivalent fixture, and convert each step to an existing page-object method before writing anything inline. Prefer accessible selectors (role, test ID) over CSS or text matches.
- Run it locally until it passes deterministically. A failure here is a mistranslation. Compare against the original and fix it; don't tune around it.
- Hand it to a second, independent agent for adversarial review against our E2E conventions: selector fragility, fixture and page-object reuse, dropped assertions or coverage, stale comments. Fix or justify every finding.
- Run the test N times in a row with retries disabled before marking it done. One green run doesn't prove stability; repetition does.
- Treat any failure in that repeat run as a real bug, not noise. Pin down which layer caused it (backend, UI timing, a bad selector, shared-state collision) and fix the root cause. Don't add a retry or widen a timeout to make it go away.
- Delete the original test and move to the next one only once the review is clean and the repeat run is 100% green.
The Migration and Review Skills gave separate agents the context to read and convert existing tests, apply our vetted Playwright patterns, and perform adversarial reviews of the output to ensure that it met our quality standards.
Working with Web Infrastructure and DevEx, we wired all of it into CI and fed it feedback on new pull requests. In harness-engineering terms, we mixed computational checks (linting, flaky-test reruns) with inferential ones (the Migration and Review Skills) into one loop: trigger a migration for a given feature area's specs, let agents translate with the Migration Skill, run the new tests, confirm the assertions, then run the Reviewer Skill and iterate until the PR was solid.
A single passing test still isn't the finish line. E2E testing's oldest enemy is flake, so every migrated spec had to clear CI's flakiness detection and self-healing before it counted as done. Kevin's flakiness-detection loop reran newly converted specs across multiple parallel CI shards, zero retries, up to 30 repeats per test. That gave us confidence the tests held up under load rather than leaning on in-framework retries to paper over instability.
Our definition of done: a full, retry-free pass of that loop, followed by sign-off from the owning feature team.
How we used Sigma to power and monitor the migration
We had a loop. We still needed visibility into how the whole effort was going, and since our legacy test data already lived in Sigma, Alison used Sigma Assistant to build our Playwright Migration Dashboard directly on top of it.
The dashboard tracked remaining tests to migrate, trends over time, outstanding PRs, and which teams had migrated the most. More importantly, it let us trigger the agentic migration loop directly from inside Sigma.
Using Sigma Workbooks, Actions, and API connectivity, a workbook could call out to Cursor to kick off the loop for any remaining legacy specs, then surface the agent link and resulting PR right back in the dashboard once the work was done.
With everything connected and agents running around the clock, the progress was hard to miss. The initial work to stand up the loop migrated roughly 10% of our specs; every week after that added another ~25%, until we hit 100% in four weeks.
This is a small, real example of what building on Sigma looks like: the same platform holding your data can also hold the workflow that acts on it.
The results: what 4 weeks of agentic migration delivered
Finishing so fast almost feels too fast; how do we know it actually worked?
Speed wasn't our success criteria on its own. Going in, I expected Playwright to give us less flake and faster, more resource-efficient testing than our legacy framework. The agentic loop's repeat-run requirement drove the flake reduction, and the parallelization work we did upfront drove the resource efficiency.
A few weeks post migration, our Sigma-powered agentic loop delivered:
- 90% less test flake compared to our previous framework
- 6,000 CI minutes saved per week
- A better developer and agent experience for E2E testing, with easier debugging, exploratory testing, and faster feedback
Since the migration, the two Playwright Skills have been consolidated and now power PR reviews of new specs, self-healing, and exploration of new features. Our flake rate has stayed a fraction of what it was, and the team is no longer buried in a constant churn of failure notifications, which has made everyone more attentive to the signals that remain.
That attentiveness already paid off. One test with a steady 3% flake rate had gone unnoticed for a long time; once flake stopped being background noise, it stood out as an outlier. Running Playwright's exploratory testing against it surfaced a real bug that only showed up in 3% of attempts. Because we could now reproduce it deterministically, we fixed it within a day.
What's next for Playwright at Sigma
The migration generated a lot of data about how our development process actually works, and that's the next opportunity: loops that generate and curate acceptance criteria with clear intent, then use Playwright to close the loop on building, testing, and fixing until the criteria pass and the agent has feedback for further exploration.
The 3% flake-rate bug is proof we can already debug and root-cause issues this way. Next, we're building toward signals that get fully investigated, debugged, and self-healed, not just detected.
Power your own agentic engineering loop with Sigma
If you're weighing whether an agentic loop could take on a backlog like this one, request a Sigma demo to see how Workbooks, Actions, and Sigma Assistant connect the pieces.
