Back-office AI fails quietly. A document router misfiles something, a person catches it, and the member never knows. Member-facing AI fails in public, one conversation at a time, with the exact people your institution exists to serve. That asymmetry should drive the entire evaluation, and it rarely does. Most chatbot and voice vendor selections we see run on a demo, a price sheet, and a reference call with a credit union the vendor picked.

The CFPB studied what happens when financial institutions get this wrong. Its report on chatbots in consumer finance reviewed consumer complaints and found the recurring damage: wasted time, people stuck in loops with no path to a human, inaccurate answers about fees and disputes. None of those outcomes started at the chatbot. They started months earlier, in a vendor evaluation that never asked the right questions. We have sat through enough of these demos to know which questions those are, so this piece walks through the five we always ask, plus the demo script we use when a vendor’s answers get slippery.

In our 2026 vendor map, member-facing tools span both the point-solution tier and the horizontal-platform tier, so this evaluation applies whether you are looking at a CU-specific vendor or configuring something general purpose. As with everything on this site, no vendor named here paid for placement, and our About page explains the methodology.

First: make the vendor define its headline number

Every vendor in this category leads with a percentage. The problem is that the percentages measure different things, and the differences are large enough to change a buying decision.

A useful live example: when KeyPoint Credit Union ($1.6 billion, 55,000 members) moved to Eltropy’s agentic platform, the announced results included a 94 percent AI resolution rate and, in the same release, a 58 percent request containment rate. Both numbers are real and both are defensible. They differ because the denominators differ: resolution rate measures inquiries the AI handled among those it attempted, while containment measures requests that never needed a human at all. A vendor quoting 94 and a vendor quoting 58 could be describing identical performance.

So skip “what is your containment rate” entirely and ask the vendor to define the numerator and the denominator, then to show the figure for institutions your size on your core. Push further: does an abandoned session count as contained? Does a member who gives up and calls the branch count as resolved? A vendor that instruments for those cases will answer in a sentence. A vendor that keeps steering back to the headline number just told you something too.

Escalation design, which is really the whole product

Years of scripted bots trained members to type “agent” over and over and hope for the best. If the vendor treats escalation as a failure state to be minimized, walk away, because some meaningful share of member conversations will always need a human, and what happens at that handoff moment is what the member remembers about your credit union.

Evaluate four specifics. First, trigger design: does the system escalate on detected frustration, repeated rephrasing, and high-stakes topics (fraud claims, disputes, hardship requests), or only on explicit demand? Second, context transfer: does the human agent receive the full conversation, the member’s authentication status, and the attempted answers, or does the member start over? Third, after-hours behavior: a bot that escalates into a void at 9 pm has not escalated, it has dead-ended, and the acceptable version captures a callback commitment with a specific window. Fourth, the effort ceiling: count the turns a determined member needs to reach a person. More than two is a complaint generator.

Integration depth, which decides what the tool can do at all

There is a hard line in this category between tools that answer questions and tools that do things. Answering “what is my routing number” requires a knowledge base. Answering “why did my payment bounce” requires reading the member’s account. Executing “move $200 to savings” requires authenticated write access to your core.

Each step up that ladder multiplies both the value and the risk, and each step depends on integration depth that varies enormously by core platform. A vendor with a certified, in-production integration on your specific core and version is a different purchase from a vendor with an API roadmap and confidence. Ask for the list of live institutions on your core, then call one the vendor did not offer.

Authentication is the gate for anything past generic answers. How does the system verify the member in channel, and how was that method validated? The KeyPoint deployment is again instructive: the credit union and vendor co-built a debit card based authentication workflow for the AI channel, which signals both real integration depth and the fact that this problem is new enough that institutions are still inventing solutions to it. If a vendor is vague about authentication, everything it does is either low-value or high-risk.

Data rights, in writing, before the pilot

Member conversations are member data. Before anything deploys, your contract should answer: does conversation data train the vendor’s models or any underlying provider’s models, who is the underlying model provider and what happens when the vendor swaps it, what are retention and deletion terms, and what performance reporting arrives monthly without asking. Our AI vendor contract guide covers the negotiating detail, and the short version is that silence in the contract means yes, your data trains someone’s model.

The provider-swap clause deserves emphasis for this category. Member-facing tools sit on top of foundation models the vendor does not control, and a quiet model change can shift tone, accuracy, and failure behavior overnight. Contractual notice of material model changes is the difference between managing that risk and discovering it in member complaints.

Members should know they are talking to AI, and the disclosure has to survive contact with reality: stated at conversation start, restated in voice channels where there is no screen to read, never designed to pass as human. NCUA’s published concerns about AI, which we covered in our read of the agency’s expectations, put transparency alongside fair lending and privacy, and a member-facing deployment is where transparency stops being abstract.

Then plan for the members it fails. Voice systems degrade for heavy accents, speech differences, and bad phone lines. Chat assumes literacy, language, and device access. The evaluation question: what is the opt-out path, and does it cost the member anything in wait time or capability? A deployment that quietly taxes your most vulnerable members is a field-of-membership problem before it is a technology problem.

Eight demo questions that expose weak products

  1. Define your containment metric’s numerator and denominator. Show it for institutions our size on our core.
  2. Show me the escalation, live: I will play a frustrated member, and I want to see what your system does and what the human agent receives.
  3. What happens at 9 pm when escalation has nowhere to go?
  4. Ask it something adjacent to your knowledge base but wrong, such as a fee your institution does not charge. Does it hallucinate a confident answer or admit the gap?
  5. Who is the underlying model provider, and what notice do we get when that changes?
  6. Does our member conversation data train your models? Show me the contract language.
  7. How does a member authenticate in this channel, and which live institutions use that method today?
  8. Show me your worst month at a comparable institution and what you changed.

Question eight is the one we watch. Good vendors have an answer ready because a bad month taught them something, and they will tell you the story without much prompting. We have also asked it and gotten a promise to circle back that never arrived, which answered the question fine.

The pattern behind member complaints

Reading the complaint patterns in the CFPB report, one design decision predicts most of the damage: deploying the tool as a gate in front of humans rather than an option beside them. Institutions that used chatbots to deflect contact generated the loops, the wasted time, and the fee misinformation. Institutions that used them to absorb the routine 70 percent while making the human path obvious kept the value without the complaints. Buy the second architecture, even when the first one demos better.

More vendor evaluation work lives in the vendor landscape pillar.

If you are evaluating member-facing AI vendors this quarter and want a structured second opinion on the shortlist, Advisor Labs runs a 45-minute AI readiness audit: book a conversation.

Get one substantive analysis like this every Tuesday: subscribe to the AiForCU newsletter.