# How a fintech QA team cut release regression by 63% and stopped hand-assembling audit evidence

*A fintech product company with 275 employees and a 12-person QA team*

A business-payments platform had to prove regression coverage for every fortnightly release, handled sensitive data in test accounts, and had a Cypress suite that failed for the wrong reasons. This worked example focuses on release confidence and evidence rather than raw speed.

## Company profile

- **Company:** Fintech product company, B2B payments platform (invoices, approvals, payouts)
- **Size:** About 275 employees
- **QA team:** 12 people: 1 QA manager, 3 SDETs, 8 functional testers
- **Release cadence:** Every two weeks (26 a year), plus hotfixes
- **Before:** Cypress suite of 300 tests in the application repository, plus 400 manual regression scenarios
- **Constraints:** SOC 2 controls requiring documented regression evidence per production release; staging reachable only from an allow-listed network

The platform lets finance teams approve and pay supplier invoices. Most journeys involve a maker-checker approval chain, asynchronous settlement statuses and a third-party card and bank form rendered inside iframes. The product is audited, and change-management controls require QA to attach regression evidence to every production release ticket.

The QA manager signs off each release. Three SDETs own the Cypress suite, which lives in the application repository and runs in CI. Eight functional testers run the manual regression pack.

## Challenges

### Evidence was assembled by hand

For each release, testers screenshotted passing scenarios, redacted account numbers and compiled a sign-off pack. That took about 32 person-hours per release and was the part of the process testers liked least.

### CI failures that were not product failures

In the 410 pipeline runs of the baseline period, 58 (14%) had at least one failure that was not a product defect, mostly asynchronous settlement statuses and the third-party iframe. Engineers got used to re-running until green. An internal audit note flagged that practice on a release gate.

### Ownership concentrated in three people

Only the three SDETs could fix the suite. Maintenance took about 200 hours a month, roughly 42% of their time, for 300 tests, or 0.67 hours per test.

### Credentials and sensitive values in the wrong places

Test-user credentials were CI variables plus a shared spreadsheet, and captured screenshots and logs could include account numbers. Security reviews asked for better handling.

### A three-day window and a contractor for release week

Manual regression of 400 scenarios, evidence assembly and retests took about 155 person-hours and three working days. The team brought in a contract tester for roughly 24 hours each release to hit the date.

### Hotfixes got a smoke test, not regression

Because the full pass was so expensive, urgent production fixes were verified with about 30 smoke tests. The QA manager considered that the largest residual risk in the process.

## Why they considered TestOwl

The team’s requirements were different from a pure speed goal: encrypted handling of credentials, masking of sensitive values in artifacts, the ability to run against the allow-listed staging environment, and a report the QA manager could attach to a release ticket as evidence.

They compared TestOwl with continuing to harden the Cypress suite. Hardening would have kept ownership with three SDETs and left evidence assembly manual. After a security review of how Secrets, Masking Rules and local execution work, they ran a two-week pilot on the payee and payment flows.

## How the team uses TestOwl

1. **Secrets and Masking Rules before any test was recorded.** Test-user credentials moved into encrypted Secrets, and Masking Rules were defined for account numbers, routing numbers and tokens so those values stay out of screenshots, reports and logs. The security team reviewed this configuration before the pilot started.
2. **Testers record maker-checker journeys; SDETs own the hard cases.** Functional testers recorded flows such as creating a payee, submitting a payment and approving it as a second user. The third-party iframe steps and settlement-status checks were set up by an SDET using condition-based waits, then reused by everyone.
3. **Reusable Flows for approval chains.** Flows such as Create payee, Submit payment for approval and Approve as second approver are used in about 150 tests. When the approval screen changed in the following quarter, the Flow was updated once.
4. **Cypress tests imported with a preview.** Of 300 Cypress tests, 240 were imported after reviewing what the importer would create. Sixty were retired as duplicates or tests of removed features.
5. **Runs from the pipeline, locally behind the allow-list, and a report per release.** The release branch triggers the suite from CI. Runs against allow-listed staging execute locally through the desktop app, and public sandbox runs execute in the cloud. The run report, including which steps healed, is attached to the release ticket, and the QA manager reviews failures and healed steps instead of compiling screenshots.

## Implementation

- **Security review and pilot (Weeks 1–2):** Set up Secrets and Masking Rules with the security team, recorded the payee and payment flows and checked masked artifacts for leaked values.
- **Import and conversion (Weeks 3–5):** Imported 240 Cypress tests, retired 60, and recorded 260 flows from the manual pack, starting with the scenarios that took longest by hand.
- **CI integration and dual running (Weeks 6–8):** Triggered runs from CI, ran two releases with TestOwl and the manual pack side by side, and agreed the report format with the compliance lead as the release evidence.

## What changed

- The release gate now rests on a stable automated signal: non-product failures dropped from 14% to 3% of CI runs, and re-run-until-green stopped being a norm.
- Evidence is a by-product of the run rather than a manual task, with sensitive values masked at source.
- Functional testers contributed 260 new automated scenarios, so the SDETs’ maintenance load fell while coverage rose.
- Hotfixes get a tagged critical-path set of 120 tests instead of a 30-test smoke check.

## Measurable outcomes

| Measure | Before | After | Change |
| --- | --- | --- | --- |
| Automated regression scenarios | 300 of 700 (43%) | 500 of 640 (78%) | +200 scenarios |
| Manual regression scenarios per release | 400 | 140 | −65% |
| Release regression window | 3 working days | 1 working day | −67% |
| Person-hours per release regression | 155 h | 57 h | −63% |
| Evidence preparation per release | 32 h | 8 h | −75% |
| CI runs with a non-product failure | 58 of 410 (14.1%) | 13 of 430 (3.0%) | −79% rate |
| SDET maintenance, hours per month | 200 h | 95 h | −53% |
| Maintenance hours per test per month | 0.67 h | 0.19 h | −72% |
| Full suite runtime (release gate, Chrome) | ~45 min (300 tests) | ~90 min (500 tests) | Longer: 67% more tests |
| Hotfix verification | ~30 smoke tests | 120 critical-path tests, ~22 min | Broader coverage |
| Contract tester hours per release | 24 h | 0 h | Eliminated |

### How it was measured

- **Measurement period:** Baseline: 13 fortnightly releases (about six months) before rollout. After: 13 releases following the 8-week rollout. Per-release figures annualized over 26 releases; maintenance annualized over 12 months.
- **Team measured:** 12 QA staff (1 manager, 3 SDETs, 8 testers). Regression hours cover testers and SDETs.
- **Regression hours, before:** 400 scenarios × 14 min = 93 h execution, + 32 h evidence, + 12 h retest, + 18 h triaging Cypress failures = 155 h.
- **Regression hours, after:** 140 scenarios × 14 min = 33 h execution, + 8 h evidence review, + 8 h retest, + 8 h triaging failures and healed steps = 57 h.
- **Maintenance hours:** 3 SDETs × 160 h/month = 480 h. Before: 200 h (about 42%). After: 95 h (about 20%), on 67% more tests.
- **Rates and capacity:** $64 blended fully loaded hourly cost for internal staff; $52 per hour for the contract tester; 1,800 productive hours per FTE. Internal hours are capacity redeployed, not headcount removed.
- **Not included:** About 300 person-hours of one-time rollout effort (about $19,200 at $64/h) and the security review time are excluded from the annual figure.

### How the savings were calculated

| Item | Calculation | Annual |
| --- | --- | --- |
| Regression effort | (155 h − 57 h) = 98 h × 26 releases | 2,548 h |
| Maintenance effort | (200 h − 95 h) = 105 h × 12 months | 1,260 h |
| Total capacity freed | 2,548 h + 1,260 h = 3,808 h (≈ 2.1 FTE at 1,800 h) | 3,808 h |
| Value of freed capacity | 3,808 h × $64 | $243,712 |
| Contract tester no longer needed | 24 h × 26 releases = 624 h × $52 | $32,448 |
| TestOwl subscription | Scale plan, 12 seats within the 15 included: $1,999 × 12 | −$23,988 |
| Estimated net annual value | $243,712 + $32,448 − $23,988 | $252,172 |

The $32,448 of contractor spend is cash that stops being spent. The remaining value is internal capacity at a blended rate. The suite got slower in absolute terms because it covers more tests; it still fits inside the release window.

## Business impact

- Release sign-off moved from three days to one, so a Thursday code freeze can be signed off Friday without a contractor.
- The release gate no longer depends on re-running failures. That addresses the audit observation directly, and the report format was accepted by the compliance lead as release evidence.
- Sensitive values are masked in screenshots and logs by rule, rather than by testers redacting them afterwards, and credentials no longer sit in a spreadsheet.
- Hotfixes now get a 120-test critical-path run in about 22 minutes, which reduced the largest residual risk the QA manager had identified.
- About 2.1 FTE of capacity moved into exploratory testing of new payment methods and approval rules.

## Lessons learned

- **Settle data handling before you record anything.** Defining Secrets and Masking Rules first meant no artifact needed rework later, and it made the security review a short conversation instead of a blocker.
- **Agree the evidence format with compliance early.** Dual running for two releases let the compliance lead compare the report with the old sign-off pack and agree what could replace it.
- **Give the hard steps to one person, then share them.** Third-party iframes and asynchronous statuses took SDET effort once. Because they sit in Flows, eight testers reuse them without touching the details.
- **Measure false failures, not just speed.** The 14% figure was the clearest argument to engineering leadership that the old gate was not a reliable signal.

### What did not improve

- Maintenance fell by about half, not to near zero: third-party form changes and new payment methods still need SDET attention.
- 140 scenarios stay manual, mainly new-feature exploration and bank-specific edge cases where a person needs to judge the outcome.
- The release-gate suite is slower than before in wall-clock terms (about 90 minutes versus 45) because it covers 67% more tests.
- Safari and mobile viewports are not part of the release gate.

## Related TestOwl pages

- [Test data and Secrets](https://testowl.ai/features/test-data): Generated data, encrypted credentials and masking.
- [Execution environments](https://testowl.ai/features/execution-environment): Local runs behind a firewall and cloud runs.
- [End-to-end testing](https://testowl.ai/end-to-end-testing): Causes of flakiness and rules for stable suites.
- [Self-healing tests](https://testowl.ai/features/self-healing): What heals and how it is reported.