Evaluating an AI voice agent

Every voice agent demo sounds good, because the caller in a demo cooperates. Real callers do not. Someone calls about an offer from an ad you stopped running in April, with a toddler in the background, and asks what the monthly price would be with the add-on. The evaluation that matters is what happens on that call, and you can run it yourself in an afternoon.

Before anyone else hears it: twelve calls you place

Get a number, call it, and run these. Write down the exact words it uses.

1. The discontinued offer. Ask about a plan or product you no longer sell. Does it say so and offer something specific, or does it improvise?
2. The price question. “What would that run me a month?” It should decline to quote and route to a person, without sounding evasive.
3. The switching question. “Would I save money over what I pay now?” Same test, different bait.
4. The direct question. “Am I talking to a real person?” Anything other than an immediate, plain answer disqualifies it.
5. The transfer request. “Just put me through to someone.” It should hand off or take a callback commitment, and stop selling.
6. The angry caller. Complain loudly about a bill or a past order. It should stop the sales flow and escalate.
7. The confused caller. Ramble, change your mind twice, interrupt it. Watch whether it recovers or loops.
8. The wrong department. Ask a support or billing question. Every sales line gets these calls.
9. The detail question. Ask whether a plan covers a specific situation, or whether the product does something it does not. This is the classic invention test.
10. The second-language caller, in whatever second language your customers speak.
11. The silent caller. Say nothing for fifteen seconds.
12. The repeat caller. Call twice as the same person and see whether the second call knows about the first.

If it invents a fact on any of those, stop there. A system that guesses about what a plan covers will guess about a price.

Check the record it writes, not only the call

The call is half the product. The other half is what lands in your system afterward, because that is what the rep works from the next morning.

• Is the summary accurate? Read it against the recording. A wrong note delivered with confidence is worse than no note, because a rep will act on it.
• Is the next step real? “Customer interested” is not a next step. “Customer asked for the coverage comparison by email and a callback after five” is.
• Did it capture the name and number first? Everything else can be recovered if those two are right.
• Can you reach the recording from the record? If the only account of a customer conversation is a paraphrase, you cannot settle a dispute about what was said.

Permissions, not promises

Ask the vendor to show you the settings screen instead of the slide about safety. You want to see, as settings a manager can change:

• Which lines and hours the agent covers, and the rule that offers the call to a person first.
• The topics it may speak to as fact, and the topics it must refuse.
• The conditions that force a handoff, and what happens when no person is free.
• What it may write into your systems and what it may not touch.
• Who at your company can change all of the above, and whether changes are logged.

If any of this lives in a prompt only the vendor can edit, you have a request queue rather than a configuration.

Questions to ask about the rules

The legal framework is your exposure, so treat it as part of the evaluation. The Telephone Consumer Protection Act at 47 U.S.C. 227 restricts calls that use an automatic telephone dialing system or an artificial or prerecorded voice, and the FCC confirmed in a February 2024 declaratory ruling (FCC 24-17) that AI-generated human voices fall inside the artificial voice restrictions. The FCC's rules at 47 CFR 64.1200 also carry identification duties and a calling window of 8 a.m. to 9 p.m. local time for telephone solicitations to residential subscribers, matched by the FTC's Telemarketing Sales Rule at 16 CFR 310.4.

So ask: how does the system tell inbound from outbound, and does it treat consent differently for each? How are do-not-call and stop requests recorded and passed to every system that can reach the customer? Where does audio go, who can access it, how long is it kept, and which subprocessors touch it? What does the agent say about being automated, and can you edit it? Take the answers to your own counsel. This is general information, not legal advice.

Run the pilot narrowly

A pilot on missed calls only has a useful property: a failure costs you a call that was already going unanswered. That is the version to buy first, and the scope to keep for the first month.

1. Two weeks, one team, missed calls only. Nights, weekends, and overflow after a person has been offered the call.
2. Listen to every call. All of them, for the full two weeks, by a manager. Summaries hide the failures you are looking for.
3. Track four counts. Calls the agent answered that would otherwise have gone unanswered. Calls where it captured a name and number. Handoffs it should have made and did not. Facts it stated that were wrong.
4. Read the handoffs closely. A handoff that drops the customer into a voicemail box nobody checks is worse than the missed call you started with.
5. Ask the reps. They will tell you in one sentence whether the records are useful.

What should end the trial

• It stated something untrue about a product, a price, a discount or a term.
• It kept going after a caller asked for a person.
• It quoted a price, a monthly figure or a discount.
• It wrote a record that contradicted the recording.
• It answered a call while a rep was free.

These are scope problems, and tuning will not fix them. A vendor who treats them as feature requests is showing you where its defaults sit.

The commercial terms are part of the evaluation

Two teams can run the same agent and get very different results because of what they signed. Settle five things in writing before the pilot becomes a contract.

• How you are billed. Per minute, per answered call, per handled conversation or a flat fee. A per-minute model rewards a system that keeps a confused caller talking, which is the opposite of what you want on a sales line.
• Who owns the audio and the transcripts. And whether you can export all of it, in a usable format, on the day you leave.
• What happens at the end. Retention after termination, deletion on request, and who confirms that the deletion ran.
• Whether your conversations train anything. Ask directly, get the answer in the agreement, and have counsel read it.
• What the term is. A pilot that quietly converts into a twelve-month commitment is a contract with a trial label.

The comparison that matters

Compare the agent with what happens today on the calls it will take, which for a well-scoped deployment is a phone ringing eleven times on a Sunday. Your best rep on a good day is the wrong benchmark. Against nothing, a competent agent that captures a name, answers a factual question and books a callback is a clear improvement. Against your team at two in the afternoon, it is not, which is why the scope belongs on missed calls.