How to evaluate a finance agent template (the new vendor evaluation)
Published 10 May 2026
The vendor evaluation conversation in finance just changed shape.
A year ago, the question was whether to subscribe to a SaaS tool that would automate part of the close, or the reconciliation, or the AP function. The evaluation was about the tool’s features, its integration, and its price.
Today, the question is whether to deploy an agent template that combines skills, connectors, and subagents to do a piece of finance work end-to-end. The evaluation is about something different. The framework I have used for evaluating AI vendors as a CFO still applies, but it needs an update for the agent-template era. This post is the update.
What an agent template actually is
An agent template is a packaged starting point for an agent. The Anthropic launch made the architecture explicit. Three components.
Skills, which are the instructions and domain knowledge the agent uses to do the work.
Connectors, which are the governed routes to data the agent runs on.
Subagents, which are additional Claude models the main agent calls for specific sub-tasks.
When you evaluate an agent template, you are evaluating all three. The marketing surface usually only describes one of them. Most often the skill, because the skill is the part that maps to the use case the customer wants. That is where the evaluation goes wrong.
The five questions to ask before piloting a template
I have run this five-question framework on a small number of agent templates in the last month. It is rough. It is useful.
1. What is the skill actually doing?
The skill in an agent template is the instruction set the model is following. Read it. Anthropic publishes the full templates and skill prompts in the open financial-services repo on GitHub (github.com/anthropics/financial-services), which is the most transparent end of the market. Most third-party vendors do not. The willingness to publish the skill is itself a buying signal.
Two things to look for in whatever the vendor will show you. Whether the instruction is doing the work, or telling the model to do the work. If the skill is a thin wrapper around “do the close,” the template is buying you very little. If the skill includes a checklist of substantive steps, defines the data the agent should and should not use, names the categories of judgment the agent should escalate, then the skill is doing real work and is worth its price.
2. What does the connector actually return?
Connector marketing is usually about coverage. “Connects to FactSet, Bloomberg, your ledger.” The question that matters is what the connector returns in practice. Does it return raw data, requiring the agent to interpret it? Does it return resolved data with the accounting standards applied? Does it cache, or query live? Does it log every request? Two connectors to the same source are not the same connector.
3. What do the subagents actually do?
Subagents are the part of the architecture least visible from the outside. They are also where a lot of the performance lives. A pitch builder that uses a subagent to do comparables selection is doing different work from a pitch builder that asks the main agent to do everything. The vendor that documents the subagent architecture is the vendor that has thought about the problem. The vendor that does not is asking you to trust the wrapper.
4. How does the template handle the moments where it is unsure?
This is the question I would put first if I could only ask one. An agent that produces an answer on every task is an agent that hallucinates. An agent that knows when to escalate is an agent that has been designed with the deployment reality in mind. Look at the escalation pattern. What gets flagged. What gets stopped. What gets attached to the output as a confidence indicator.
5. What is the audit trail?
The agent will be wrong sometimes. The question is whether you can reconstruct why. The audit trail is the thing that lets you do that. It needs to include the input the agent saw, the data the connectors returned, the steps the agent took, the subagents that were called, the output that was produced, and the human review that followed. If the template does not have all of those by default, you are buying a black box.
What to look at on the commercial side
The pricing model is also new.
SaaS pricing was per-seat or per-volume. Agent template pricing is a mix of subscription, usage, and connector data fees. For Anthropic specifically, the subscription tiers are Pro ($20/month), Max ($100 or $200/month), Team, and Enterprise. The finance agent templates are free plugins inside Cowork and Code, but model API usage is metered separately, the connector data partners (Moody’s, FactSet, Dun & Bradstreet) charge their own fees, and Managed Agents in production has its own platform pricing. The unit cost of an agent doing real work can be very different from the unit cost the brochure quotes.
The AI in finance cost piece has the long version of how to model this. The agent-era short version: ask the vendor for the per-task all-in cost on a representative volume, and do not let them quote you the per-seat cost. As of mid-May 2026, no vendor has published a credible total-cost-of-ownership case study for finance agent templates. Forrester’s 2026 Predictions flagged this gap explicitly. (“Only a small proportion of organizations can currently tie AI initiatives to impact.” Source: Forrester Predictions 2026.) That is itself a buying-discipline signal.
The contract terms also matter. Watch for the data retention clauses, the model-update terms, and the migration provisions. The cost of being locked into a vendor’s template architecture is higher than the cost of being locked into a SaaS feature set, because the work the template does is more deeply embedded in the function’s process.
What the template will not tell you
Two things the template’s marketing will not surface, and you have to ask.
The first is the failure rate on tasks resembling yours. The Vals AI benchmark score is the public signal. It is not a forecast of how the template behaves on your data, in your business, with your team. Insist on a pilot. The vendor that resists a pilot is the vendor whose pilot you most need to run.
The second is the level of customisation the template assumes. Some templates are usable out of the box. Most require the skill to be edited for your chart of accounts, your reporting calendar, your materiality thresholds. The cost of that customisation, in time and in attention, is rarely surfaced in the sales process. Ask. The honest vendor will tell you. The vendor that says “no customisation needed” is the vendor who has not yet seen your data.
What I would do with this
Before any agent template lands in your function, run the five questions on it. Document the answers. Then run a pilot in shadow mode on a real workload for two cycles. Then make the deployment decision.
The five questions are a buying framework. The risk-side analogue is the NIST AI Risk Management Framework’s Agentic Profile (NIST AI 100-5), which organises governance around Govern, Map, Measure, and Manage functions. (Source: nist.gov/itl/ai-risk-management-framework.) The two frameworks cover different angles of the same problem. Use both. Gartner’s emerging “AI Agent Management Platform” concept extends the same thinking from the IT control side. The buying conversation, the risk conversation, and the control conversation are three views of one decision.
The whole sequence should take six to twelve weeks. That is faster than a SaaS implementation used to be, and slower than the vendor sales cycle wants it to be. Hold the line.
The building an AI-ready finance function post covers the broader readiness picture. Vendor evaluation is one part of that. The questions above are the part of the readiness picture the agent-template era added.
Where this lands
The vendor evaluation has not become harder. It has become different. The shape of what you are evaluating has changed, and the framework has to change with it. The thing you are buying is no longer a tool with features, it is an agent with skills, connectors, and subagents. The evaluation has to look at all three, the data they touch, the escalation pattern, the audit trail, and the all-in cost.
The functions that win the next two years are the ones that do that work seriously, before the deployment, not after.
Maebh Collins is a Fellow Chartered Accountant (FCA, ICAEW) with Big 4 training and twenty years of operational experience as a founder and senior finance leader.