# A pilot at a partner company found LLM-generated acceptance tests helpful most of the time, but a third of the generated scripts needed a person to fix, regenerate or discard them

Page: https://flyvolo.ai/en/changes/ev-20250409-business-analyst-9
Date: 2025-04-09 · Stage: Pilot
Can this record move a task judgement? only together with another independent record
Occupation: https://flyvolo.ai/en/careers/business-analyst

## Scope

A two-tool pilot (AutoUAT for test scenarios, Test Flow for executable scripts, both on GPT-4 Turbo) was integrated into one partner company's workflow and evaluated on real-world projects, researchers report. Users found AutoUAT's natural-language acceptance test scenarios, drafted from user stories, helpful 95% of the time, sometimes surfacing cases the team had overlooked. Of Test Flow's executable Cypress scripts, 92% were considered helpful, but only 60% were usable as generated: 8% needed minor fixes, 24% needed regeneration with more input, and 8% were discarded for major issues. The paper frames this as LLMs helping the acceptance-test process only with appropriate tooling and supervision. It establishes that at this one company, generating the scenarios and scripts used to check delivered work against requirements still needed a person to accept, fix or reject roughly a third of the output; it does not name the company, measure how analysts' or testers' overall hours changed, or say anything about change-request handling or preparing people for a new way of working, the rest of this task.

## Source

- Ferreira, Viegas et al. — Acceptance Test Generation with Large Language Models: An Industrial Case Study (arXiv) — https://arxiv.org/abs/2504.07244 (primary source)

## What this means

At one real company, LLM tools drafted the acceptance test scenarios and scripts used to check delivered work against requirements, and were helpful most of the time (95% and 92% respectively), but a person still had to accept, fix or reject roughly a third of the generated scripts before they were used.

## What it does not show yet

It does not say how acceptance testing changes when a person is not checking the output, does not cover change-request handling or preparing people for a new way of working, and does not say whether this holds outside one company's web-application projects.

## How to verify it yourself

Before accepting an AI-drafted acceptance test as done, re-run it yourself against the actual requirement it claims to check, the way this pilot's reviewers did for the third that needed fixing.


More: https://flyvolo.ai/llms.txt