The pilot-to-production playbook for finance agents
Published 9 May 2026
I have been referring to the pilot-to-production playbook for finance agents in posts for the last month without ever publishing it. That has been a mistake, because the pilot-to-production transition is the part of the deployment lifecycle where most agent projects quietly fail, and a published playbook is more useful than a referenced one.
Gartner’s June 2025 prediction is that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing “escalating costs, unclear business value, or inadequate risk controls.” (Source: Gartner press release, 25 June 2025.) A March 2026 enterprise survey of 650 organisations found that only 14% had scaled an AI deployment to production. The pilot-to-production gap is the most expensive gap in the deployment lifecycle, and the least documented.
This post is what I would actually do, step by step, with citations to the public frameworks and the published case studies.
What “pilot” and “production” actually mean
The terms are used loosely. The framework I find most useful is the autonomy tier classification from the Cloud Security Alliance’s Agentic NIST AI RMF Profile v1, which extends NIST AI 100-1 for autonomous agents. (Source: CSA Labs Agentic Profile.) The profile defines four tiers, from fully supervised (Tier 1) to full autonomy (Tier 4), and requires every agent to be classified “according to an autonomy tier that reflects both the agent’s operational independence and the consequences of its potential failures” before any Tier-2-or-higher agent is deployed.
That gives a working definition. A pilot is a Tier 1 deployment with humans approving every output. Shadow mode is a Tier 1 deployment running in parallel with the existing manual workflow without taking action. Limited production is Tier 2, where the agent acts on low-risk decisions and escalates the rest. Full production is Tier 3, where the agent acts autonomously with monitoring and a defined escalation path. Tier 4 is full autonomy, and I would not deploy any finance agent at Tier 4 inside the next eighteen months.
The five-stage playbook
Stages, intended cadence, and the public-source evidence for each stage.
Stage 1: Pre-pilot governance (week 0)
Before any agent code is written or any template is configured, three things have to be on paper.
Autonomy tier classification under CSA AG-GV.1. The classification has to be signed off by the function head and the head of IT. The Berkeley CLTC Agentic AI Risk-Management Standards Profile (February 2026) adds explicit risk categories to consider at this stage: unintended goal pursuit, privilege escalation, self-replication, resistance to shutdown, misinformation, and amplified bias. (Source: CLTC Profile.) These read as engineering concerns but they map cleanly to finance failure modes: an agent that posts journal entries it should escalate, an agent that pulls data outside the connectors it should use, an agent that produces plausible commentary on the wrong period.
Tool risk inventory under CSA AG-MP.1, covering the consequence scope, reversibility, authentication, and compositional risk of every tool the agent will use.
Accountability lineage under CSA AG-GV.2, which “connects every agent action to a responsible human officer.”
The function that skips this stage is the function whose pilot will produce useful data and whose production deployment will lack the documentation an auditor or regulator will want to see twelve months later.
Stage 2: Pilot (weeks 1-6)
Contained sandbox. Tier 1. The agent runs on real but limited inputs with human approval of every output.
The FCA’s AI Live Testing programme is the closest public analogue to a credible regulator-grade pilot. (Source: FCA FS25/5, AI Live Testing.) The first cohort, announced October 2025, includes NatWest, Monzo, Santander, Scottish Widows, Gain Credit, Homeprotect, and Snorkl. The second cohort, announced April 2026, includes Barclays, Lloyds Banking Group, UBS, Experian, Aereve, Coadjute, Go-Cardless, and Palindrome. (Source: FCA press release on cohort 2.) The structure is small cohort, defined evaluation rubric, technical partner (Advai), and output-driven validation. That is what a serious pilot looks like.
For an internal finance pilot, the equivalent: agree the success criteria before the pilot starts, define the population of work the agent will see, log every input and output, and review with the function head weekly.
Stage 3: Shadow mode (weeks 6 to 14)
The agent runs in parallel with the existing manual workflow. The agent’s outputs are produced and logged but not acted on. The team continues to do the work. A daily or weekly comparison surfaces where the agent and the team agree, where they disagree, and what the disagreements reveal.
This is the stage most agent projects skip, and it is the stage that most strongly predicts production success. Ramp’s published approach is the cleanest finance-specific example I have seen: the agent runs on real transactions in shadow mode, an LLM-judge compares its predictions to what the human did, and the agent only graduates to live action when shadow accuracy crosses a defined threshold. (Source: Ramp engineering blog and industry coverage.)
A defensible shadow window is eight to twelve weeks and at least 5,000 decisions before any production cutover. The reason is straightforward: the agent’s failure modes only appear in the specific edge cases of your data, and the long tail of those edge cases takes time and volume to surface. Anthropic’s “Building Effective Agents” engineering post from December 2024 makes the same point in different language: “the most successful implementations weren’t using complex frameworks or specialized libraries… they were building with simple, composable patterns” and iterating against real outputs. (Source: Anthropic Building Effective Agents.)
Stage 4: Canary cutover (weeks 14 to 18)
The agent moves to limited production on a defined subset of the workflow. Tier 2 deployment. Reversible actions only. Ten per cent of the population by default, with automated rollback triggers if monitoring detects drift.
The canary pattern is borrowed from software reliability engineering, and it works for finance agents for the same reasons it works for backend deployments: a small live population exposes failures that a sandbox cannot, with bounded blast radius. The CSA profile’s AG-MS.1 requires runtime telemetry covering action velocity, permission escalation rates, cross-boundary invocations, delegation depth, and exception rates. That telemetry is the canary’s instrumentation. Without it, the canary is just a pilot with more risk.
Stage 5: Production (week 18 onward)
Full FRC mitigations live. The Financial Reporting Council’s March 2026 Generative and Agentic AI Guidance, the first regulator-issued guidance globally specifically on agentic AI in audit engagements, sets four mitigation categories that should be live at production: system design and development, certification, staff education and governance, and human-in-the-loop review and oversight. (Source: FRC Guidance, 30 March 2026.) The FRC’s framing is for audit engagements; the principles apply to any AI deployment in regulated finance work.
The CSA profile’s AG-MG.3 requires a formal decommissioning plan, including credential revocation and audit-log preservation, to be in place before production. Most teams build this plan after they need it. Build it before.
What the case studies tell us
Three production deployments worth knowing about.
Goldman Sachs GS AI Assistant. Pilot to ten thousand employees, scaled to forty-six thousand firm-wide during 2025. The pilot ran for months, not weeks. The cutover was gradual, not all-at-once. (Source: CNBC, January 2025; Slashdot, June 2025.) The deployment is now described as updating “every eight weeks with new data and workflows,” a continuous improvement cadence that is more important than the initial cutover.
JPMorgan LLM Suite. Launched August 2024 to approximately sixty thousand employees, scaled to one hundred and forty thousand, now at two hundred and fifty thousand. The Suite combines OpenAI and Anthropic models with internal RAG layers. (Source: CIO Dive on the rollout.) The cadence is similar to Goldman’s: pilot, careful cutover, continuous expansion.
TD Bank. Public statement that it takes “about six to nine months building out the engineering infrastructure to take a model from proof of concept to production.” (Source: TD Bank cited via industry coverage; verify before quoting directly.) That number is consistent with what I would tell a client. The pilot is the easy part. The infrastructure is what determines whether production lasts.
What the failures tell us
Three failures worth knowing about.
Moffatt v Air Canada, BCCRT, 14 February 2024. Chatbot in production produced a refund policy that contradicted Air Canada’s actual policy. Tribunal held Air Canada liable. The chatbot was not piloted in shadow mode against the live policy database. (Source: Moffatt v Air Canada via American Bar Association.) The lesson is not “do not deploy chatbots in customer-facing roles.” The lesson is “the chatbot is your company’s voice, and the policy-sync and review layers are not optional.”
Klarna AI customer service rollback, mid-2025. Replaced approximately seven hundred customer service agents with AI. CSAT dropped 22 percentage points. The company rehired humans. Sebastian Siemiatkowski, CEO: “We focused too much on efficiency and cost. The result was lower quality, and that’s not sustainable.” (Source: Entrepreneur on the Klarna reversal.) The deployment graduated to full production without a shadow-mode CSAT validation and without a rollback plan for empathy-sensitive workflows.
McDonald’s IBM AI drive-thru, June 2024. Two-year pilot across one hundred-plus US locations ended. Order accuracy plateaued at 80 to 85% against a human baseline of approximately 90%. (Source: CNBC, 17 June 2024.) The deployment failed at the pilot stage, never crossed the accuracy threshold for production, and the published lesson is that “the gap between controlled demonstrations and real-world deployment was wider than the marketing suggested.”
The pattern across the failures is consistent. Two of three failed at the production stage because the shadow-mode validation that would have caught the failure was not done. The third failed at the pilot stage because the accuracy threshold for production was never met.
What I would do this quarter
If you are deciding whether to move a finance agent pilot to production this quarter, the checklist is short.
Did you classify the autonomy tier before the pilot started? If not, do it now.
Did you run shadow mode for at least eight weeks against at least 5,000 real decisions? If not, do that before any production cutover.
Do you have the runtime telemetry to detect drift in production? Action velocity, exception rates, permission escalation. If not, the production deployment is flying blind.
Do you have a rollback plan that has been tested? Most rollback plans are written and never tested. Test yours before production.
Do you have a written autonomy demotion procedure for the moments when the agent starts producing outputs the team does not trust? CSA AG-MG.2 calls this “tier demotion.” It is the production analogue of pulling a junior off a piece of work that is going wrong. The function that has the procedure written before production is the function whose production survives a bad week.
The CFO 11 questions post covers the broader pre-deployment governance frame. The five-stage playbook above is the operational layer beneath those questions.
Where this lands
The pilot-to-production transition is where finance agent deployments mostly live or die, and the public evidence is now thick enough to build a defensible playbook on. The frameworks exist (NIST, CSA, FRC). The case studies exist (Goldman, JPMorgan, Air Canada, Klarna, McDonald’s). The cadence is now reasonably consistent across the credible deployments: six to eighteen weeks of pilot and shadow mode, a canary cutover, then production with telemetry.
The function that follows the playbook will not avoid every failure. It will avoid most of the failures the published case studies have already paid for. That is the highest-value deliverable a finance leader can get from reading about other people’s mistakes.
Maebh Collins is a Fellow Chartered Accountant (FCA, ICAEW) with Big 4 training and twenty years of operational experience as a founder and senior finance leader.