How a four-person QA team with no automation engineer got to 79% automated regression
A B2B SaaS company with 48 employees and a 4-person QA team
A field-service software company had one abandoned Playwright suite, a spreadsheet checklist and developers pulled into every release. This worked example shows what a small team can realistically change with one extra seat and no new hire, and what it cannot.
- Regression pack automated
- 19% → 79%
- 35 up to 130 of 165 scenarios
- Regression cycle
- 2 days → ½ day
- About two hours of automated run, in parallel with manual checks
- Person-hours per release
- −71%
- 42 h down to 12 h
- Releases with full regression
- 6 of 6
- Up from 2 of 6
The team and the product
- Company
- B2B SaaS, scheduling and dispatch software for field-service contractors
- Size
- About 48 employees, 14 developers
- QA team
- 4 people: 1 QA lead and 3 QA analysts. No dedicated automation engineer
- Release cadence
- Every two weeks (26 a year)
- Before
- A 180-scenario regression checklist in a spreadsheet; a Playwright suite of 49 tests, 14 of them skipped
- Plan
- TestOwl Starter ($299/month) plus one extra seat
The product is a web application for scheduling jobs, dispatching technicians and sending invoices. Its schedule board is a drag-and-drop calendar, and technicians use a responsive mobile view in the field.
A contract developer wrote the Playwright suite about 18 months earlier and left. Since then nobody has owned it. The SDET role had been open for six months without a suitable hire, and the company was not going to pay for two.
Challenges
1Automation nobody owned
Of 49 Playwright tests, 14 were skipped and several of the rest failed after CSS changes. One developer spent about 12 hours a month nursing the 35 tests that still ran, 0.34 hours per test, and had stopped trusting them.
2A spreadsheet checklist as the real regression process
The 180-scenario checklist was run before each fortnightly release by the QA team and took about two working days. Of the last six releases, only two got the full pass; four got a reduced one because of schedule pressure.
3Developers pulled into testing
Four developers each spent about two hours per release helping with the checklist, 8 hours in total, which the engineering lead counted as a hidden cost of every release.
4No budget for the standard fix
A Playwright rewrite needed an SDET the company could not hire, and asking developers to own it competed with the roadmap.
5A small team with a lot of surface area
Four people covered the web app, integrations with accounting and messaging, and the mobile technician view. Time to learn and maintain a framework was not available.
Why they considered TestOwl, and how they use it
The QA lead wanted a tool the three analysts could use on day one, with a price that did not need a budget cycle. The Starter plan includes three seats; one extra seat covered the fourth person.
The 7-day free trial, which needs no card, let the team test the claim that analysts could record real flows before anyone committed. They recorded the twelve flows behind most past customer-reported regressions during that week.
The workflow
- 1
Analysts record from the checklist
Each analyst works down the spreadsheet, records a scenario in the browser, adds assertions and ticks it off. A recorded scenario that took ten minutes to run by hand typically took about fifteen to record and tidy once.
- 2
Reusable Flows for the common starts
Login, create customer and create job became Flows. A change to the job form is fixed in one place, and the Flow’s usage count shows how many tests it touches.
- 3
Import what was still useful
Twenty of the 35 running Playwright tests were imported after previewing. Fifteen were retired as duplicates of newer flows or tests of removed screens, and the 14 skipped tests were already covered by the checklist.
- 4
A schedule sized to the plan
The full 130-test suite runs three times a week plus on every release candidate, about 1,950 cloud runs a month, inside Starter’s 2,500. Two parallel workers finish a run in about two hours.
- 5
Failures reviewed by QA, not developers
Healed steps and failures are reviewed by the analyst who recorded the flow. Developers only see confirmed product failures.
Implementation
- Week 1
Trial
Recorded the twelve highest-risk flows on the free trial and compared runs with a manual pass.
- Weeks 2–5
Import and record
Imported 20 Playwright tests, then recorded 110 scenarios from the checklist, about seven per person each week alongside regular testing.
- Week 6
Schedule and switch over
Set the schedule, ran one release with both the checklist and TestOwl, then reduced the checklist to the 35 manual scenarios.
What changed
- Automated regression grew from 35 scenarios to 130 (19% to 79% of the pack) with no automation engineer.
- Every one of the next six releases got the full regression pass, where before four of six had been cut down.
- Developers were no longer part of the regression process: 8 hours per release became 1.
- Total maintenance hours barely moved because the old suite had already been neglected. What changed is how many tests those hours cover.
Before and after
| Measure | Before | After | Change |
|---|---|---|---|
| Automated regression scenarios | 35 of 180 (19%) | 130 of 165 (79%) | +95 scenarios |
| Manual regression scenarios per release | 145 | 35 | −76% |
| Regression cycle, elapsed | 2 working days | ~½ working day | −75% |
| Person-hours per release regression | 42 h | 12 h | −71% |
| Developer hours pulled into regression | 8 h per release | 1 h per release | −7 h |
| Releases with the full regression pass | 2 of 6 | 6 of 6 | +4 |
| Test maintenance, hours per month | 12 h | 9 h | −3 h |
| Maintenance hours per test per month | 0.34 h | 0.07 h | −80% |
| Tooling cost | $0 licence; ~12 h/month of developer time | $378 / month | Starter + 1 seat |
How it was measured
- Measurement period
- Baseline: the six fortnightly releases (12 weeks) before adoption. After: the six releases following a 6-week ramp. Per-release figures annualized over 26 releases; maintenance over 12 months. The sample is small, so treat the percentages as indicative.
- Team measured
- 4 QA staff plus 4 developers who helped with regression. The developer’s Playwright maintenance time is counted in the maintenance row.
- Regression hours, before
- 145 manual scenarios × 10 min = 24 h, + 6 h retest and triage, + 8 h developer help, + 4 h triaging the Playwright run = 42 h.
- Regression hours, after
- 35 scenarios × 10 min = 6 h, + 3 h retest, + 1 h developer review, + 2 h triaging failures and healed steps = 12 h.
- Runtime
- 130 tests × about 2 minutes ÷ 2 parallel workers ≈ 130 minutes.
- Rates and capacity
- $62 blended fully loaded hourly cost across QA and developers; 1,800 productive hours per FTE.
- Not included
- About 110 person-hours of one-time effort (about $6,800 at $62/h) over the six-week ramp are excluded from the annual figure.
How the savings were calculated
| Item | Calculation | Annual |
|---|---|---|
| Regression effort | (42 h − 12 h) = 30 h × 26 releases | 780 h |
| Maintenance effort | (12 h − 9 h) = 3 h × 12 months | 36 h |
| Total capacity freed | 780 h + 36 h = 816 h (≈ 0.45 FTE at 1,800 h) | 816 h |
| Value of freed capacity | 816 h × $62 | $50,592 |
| TestOwl subscription | Starter $299 + 1 extra seat $79 = $378 × 12 | −$4,536 |
| Estimated net annual value | $50,592 − $4,536 | $46,056 |
No cash cost disappeared here: this is capacity, mostly developer and QA hours returned to other work. The extra-seat rate is the published add-on price at the time of writing; confirm current rates. The one-time ramp cost is recovered in under two months of steady state.
Business impact
- Skipping regression stopped being a way to meet a release date. The engineering lead can now count on the full pass happening every time.
- Developers recovered about 182 hours a year (7 hours × 26 releases) that they had been spending on regression checks, and they no longer need to stop feature work when a release candidate is cut.
- The company avoided a six-month search for an SDET it could not easily afford, and the cost of the tool is under 10% of the value of the hours freed.
- The team now has a coverage number it can report, rather than a spreadsheet that records what someone intended to check.
Lessons learned
Start with the flows that have already hurt you
The twelve flows behind past customer-reported regressions gave the first week visible value and justified the rest of the effort.
Let the spreadsheet drive the order
Working down the existing checklist meant no planning overhead and no new test-design debate. Each recorded scenario retired a row.
Size the schedule to the plan
Choosing a run frequency that fits the monthly run allowance avoided both overage worries and a daily full run nobody would read.
Expect maintenance to stay a line item
Nine hours a month is still nine hours. The gain is coverage per hour, not zero upkeep.
What did not improve
- Starter covers desktop browsers only, so the technician mobile view stays in the manual 35 until the team moves to a plan with mobile emulation.
- The drag-and-drop schedule board was the hardest area to record reliably and remains partly manual, along with PDF invoice output and SMS delivery checks.
- Six releases before and after is a small sample, and the figures should not be extrapolated beyond this kind of team.