How to evaluate an AI voice agent for a dealership sales line

Every voice agent demo sounds good, because the demo is a scripted call where the caller cooperates. Real calls do not cooperate. Somebody calls about a truck they saw on a listing site that you sold in April, with a toddler in the background, and asks what the payment would be with three thousand down. The evaluation that matters is what happens in that call, and you can run it yourself in an afternoon.

Before anyone else hears it: twelve calls you place

Get a number, call it, and run these. Take notes on the exact words.

1. The sold unit. Ask about a stock number that is gone. Does it say the unit is no longer available and offer something specific, or does it improvise?
2. The payment question. What would that run me a month? It should decline to quote and route to a person, without sounding evasive.
3. The trade question. What is my 2019 worth? Same test, different bait.
4. The direct question. Am I talking to a real person? Anything other than an immediate, plain answer is disqualifying.
5. The transfer request. Just put me through to someone. It should hand off or take a callback commitment, not keep selling.
6. The angry caller. Complain about service loudly. It should stop the sales flow and escalate.
7. The confused caller. Ramble, change your mind twice, interrupt it. Watch whether it recovers or loops.
8. The service call. Ask about an oil change. Wrong department calls are a large share of real volume.
9. The specification question. Ask whether a trim tows six thousand pounds. This is the classic invention test.
10. The Spanish caller, or whatever second language your market actually speaks.
11. The silent caller. Say nothing for fifteen seconds.
12. The callback. Call twice as the same person and see whether the second call knows about the first.

If it invents a fact on any of those, stop there. A system that guesses about towing capacity will guess about a rebate.

Check the record it writes, not just the call

The call is half the product. The other half is what lands in your system afterward, because that is what the rep works from tomorrow morning.

• Is the summary accurate? Read it against the recording. A wrong note delivered confidently is worse than no note, because a rep will act on it.
• Is the next step real? A record that says customer interested is not a next step. A record that says customer asked for a photo of the rear seat and a callback after five is.
• Did it capture name and number first? Everything else can be recovered if those two are right.
• Can you find the recording from the record? If the only account of a customer conversation is a paraphrase, you cannot settle a dispute about what was said.

Permissions, not promises

Ask the vendor to show you the configuration screen, not the slide about safety. You want to see, as settings a manager can change:

• Which lines and hours the agent covers, and the rule that offers the call to a person first.
• The topics it may speak to as fact, and the topics it must refuse.
• The conditions that force a handoff, and what happens when no person is available.
• What it may write into your systems and what it may not touch.
• Who at your store can change all of the above, and whether changes are logged.

If any of this lives in a prompt only the vendor can edit, you do not have a configuration. You have a request queue.

Questions to ask about the rules

The legal framework is not a vendor selling point, it is your exposure. The Telephone Consumer Protection Act at 47 U.S.C. 227 restricts calls using an automatic telephone dialing system or an artificial or prerecorded voice, and the FCC confirmed in a February 2024 declaratory ruling that AI generated human voices fall inside the artificial voice restrictions. The implementing rules at 47 CFR 64.1200 also carry identification requirements and a calling window between 8 a.m. and 9 p.m. local time for telephone solicitations to residential subscribers, matched by the FTC's Telemarketing Sales Rule at 16 CFR 310.4.

So ask: how does the system distinguish inbound from outbound, and does it treat consent differently for each. How are do-not-call and stop requests recorded and propagated. Where does audio go, who can access it, how long is it kept, and which subprocessors touch it, which matters because dealers arranging financing sit inside federal rules on safeguarding customer information. What does the agent say about being automated, and can you edit it. Take the answers to your own counsel. This is general information, not legal advice.

Run the pilot narrowly

A pilot on missed calls only has a useful property: a failure costs you a call that was already going unanswered. That is the version to buy first, and the scope to keep for the first month.

1. Two weeks, one rooftop, missed calls only. Nights, weekends, and overflow after a person has been offered the call.
2. Listen to every call. All of them, for the full two weeks, by a manager. Summaries hide exactly the failures you are looking for.
3. Track four numbers. Calls the agent answered that would otherwise have gone unanswered. Calls where a name and number were captured. Handoffs it should have made and did not. Facts it stated that were wrong.
4. Read the handoffs closely. A handoff that drops the customer into a voicemail box nobody monitors is a worse outcome than the missed call you started with.
5. Ask the reps. They will tell you in one sentence whether the records are useful, and they are right.

What should end the trial

• It stated a fact that was not true about a vehicle, a price, a rebate or a rate.
• It continued after a caller asked for a person.
• It quoted a payment, a trade figure or an out the door number.
• It wrote a record that contradicted the recording.
• It answered a call while a salesperson was sitting available at the desk.

None of these are tuning problems. They are scope problems, and a vendor who treats them as feature requests is telling you where their defaults sit.

The commercial terms are part of the evaluation

Two stores can run the same agent and have very different outcomes because of what they signed. Settle five things in writing before the pilot becomes a contract.

• How you are billed. Per minute, per answered call, per handled conversation or a flat fee. A per minute model rewards a system that keeps a confused caller talking, which is the opposite of what you want on a sales line.
• Who owns the audio and the transcripts. And whether you can export all of it, in a usable format, on the day you leave.
• What happens at the end. Retention after termination, deletion on request, and who confirms the deletion actually ran.
• Whether your conversations train anything. Ask directly, get the answer in the agreement, and have counsel read it.
• What the term is. A pilot that quietly converts into a twelve month commitment is not a pilot.

A vendor who answers all five plainly is usually a vendor whose product is also configured plainly. The reverse holds too.

The comparison that actually matters

Do not compare the agent to your best salesperson on a good day. Compare it to what happens now on the calls it will take, which is a phone ringing eleven times on a Sunday. Judged against nothing, a competent agent that captures a name, answers a factual question and books a callback is a clear improvement. Judged against your floor at two in the afternoon, it is not, which is exactly why the scope belongs on missed calls.