LO
PilotCognitive automation2024-08-23
Australia's Treasury found staff in its 14-week Copilot trial used the tool less than expected, citing high initial expectations, technology limits and a lack of education and training
Business systems owneroccupation page →
Event date / reported
2024-08-23 · reported 2025-02-05
Evidence stage
PilotSmall-scale trial in a real setting. Tells us the deployment conditions are being tested, not that they hold — so one pilot is never enough on its own; two independent ones are.
Tasks this bears on
Getting the team to use it properly
Training, answering the same question for the fifth time, and finding out that the reason a field is always empty is that filling it costs someone twenty seconds they do not have.
Still human-led✓ Evidence-backed
Where this applies
The Treasury's in-house evaluation unit reporting on a trial of Microsoft 365 Copilot by 218 staff from 20 May to 23 August 2024, part of an Australian Government trial. It says the product was used less than originally expected, with about a quarter using it frequently by the end; that lower use likely came from initial high expectations that diminished in practice, technology limitations, and a lack of education and training on how to use the product; and that 42 participants self-nominated as trial champions tasked with supporting their teams. Findings rest on surveys (136 post-trial responses) and focus groups. It is an office AI assistant in a policy department, not a CRM or order system.
What this means
A second government trial, in another country, found the same thing: people used the assistant less than planned, and part of the reason given was that nobody had taught them how. Getting a team to use a new system properly is still work somebody has to do.
What it does not yet show
A small trial in one policy department, based on surveys; it does not show how CRM or order-system rollouts go, or whether these reasons hold elsewhere.
What you can check
Open the Treasury's "Evaluation of a trial of generative artificial intelligence (Copilot)" and find "a lack of education and training on how to use the product".
Does it change the assessment?
No. The impact index is never moved by a single event, and this stage does not move one on its own: a "Pilot" record counts toward a judgement but needs a second, independent record before the judgement rests on evidence. This one is counted; on its own it changed nothing.
Source
Australian Centre for Evaluation, Department of the Treasury — "Evaluation of a trial of generative artificial intelligence (Copilot) in The Department of the Treasury" (evaluation report, February 2025) · verified 2026-09-28 · Claude (VOLO agent) · interpreted 2026-09-28 · Claude (VOLO agent)
Primary source — published by the party that did this, or the authority of record. No co-signature needed.