Public agencies are being pitched AI tools faster than they can build the procurement frameworks to evaluate them. Chatbots for benefits intake, virtual assistants for public health hotlines, automated triage for 311 systems: the sales cycle moves in weeks. Governance does not. What a public agency should ask before putting AI in front of residents is not an abstract ethics question. It is a concrete due diligence problem, and administrators who skip it inherit the liability when the system fails in public.
This is not a piece about whether agencies should adopt AI. That decision is largely being made for them by budget pressure and staffing shortages. This is a checklist for the harder question: which vendor questions separate a defensible deployment from a reputational and legal liability.
AI Vendor Evaluation for Public Agencies: Why a Different Procurement Standard Is Needed
Standard IT procurement asks about uptime, security certifications, and cost per seat. Resident-facing AI needs a different standard because the failure mode is different. A misconfigured expense report tool produces an internal headache. A chatbot giving incorrect Medicaid eligibility information produces a resident who loses coverage, and a news story.
Real cases illustrate the stakes. State unemployment and benefits systems in Michigan, Colorado, and elsewhere have faced lawsuits and federal scrutiny over automated decision systems that wrongly flagged or denied residents, with agencies later admitting error rates that took years to unwind. Air Canada was held liable in a 2024 tribunal ruling after its chatbot gave a customer incorrect bereavement fare information; the airline argued the bot was a separate legal entity, and the tribunal rejected that argument outright. Public agencies do not get to make that argument either. If your chatbot tells a resident something wrong about a benefit, a deadline, or a health service, the agency owns that answer, not the vendor.
This means AI vendor evaluation for public agencies has to start from a liability posture, not a features posture. Before any conversation about capabilities, ask: when this system is wrong, who is legally and politically accountable, and what does the vendor's contract say about that.
Escalation to Humans: Setting the Rules Before Launch
Every resident-facing AI system will eventually encounter a question it cannot or should not answer alone. The question is whether escalation is designed in advance or discovered after a bad interaction goes viral.
Ask vendors directly: what specific triggers force a handoff to a human, and how fast does that handoff happen. "The AI will use judgment" is not an answer. You want a list: self-harm disclosures, allegations of abuse, legal threats, requests for a supervisor, repeated failed attempts to resolve an issue, and any topic outside a defined scope. Each trigger should map to a specific action and a specific response time, not a vague commitment to "human oversight."
Then ask for evidence, not assurance: "Show me your escalation logs from the last 90 days." A vendor with a mature product can produce this. A vendor still building the feature cannot. This single request filters out most unready systems faster than any technical spec sheet.
This is also where resident-facing AI accountability needs to be contractually explicit before launch, not litigated after an incident. Confide's approach to sensitive conversations in HIV/PrEP navigation and campus support programs, where escalation paths are defined before a single resident or student interacts with the system, is one illustration of building the handoff rule first. The same discipline applies to government resident support programs evaluating any AI vendor, not just this one.
Records, Retention, and Accountability Requirements
Public agencies operate under records laws that most AI vendors were not built around. Every conversation a resident has with a government-deployed AI system is a potential public record, a potential discovery item, and a potential privacy exposure, sometimes all three at once.
Before signing anything, get clear answers on four points. What gets logged: full transcripts, metadata only, or both. How long it is retained by default, and whether that default can be overridden by agency policy rather than vendor convenience. Who can access records, internally at the vendor and externally under a public records request. And how the system handles a resident's request to have their data deleted, if your state's privacy law grants that right. Ask plainly: "What is your data retention default, and can we override it?" Many vendors default to long retention because it improves their own model training or analytics. That default may directly conflict with your agency's public records obligations or resident privacy commitments. This is a negotiation point, not a fixed term.
PHI-free design, where a system is built so no protected health information is captured in the first place, is one way agencies handling sensitive health topics reduce this exposure. It does not eliminate records obligations, but it narrows what is even in scope for a records request or a breach notification. Agencies running PrEP, HIV, or behavioral health programs should ask vendors specifically whether the system was designed PHI-free or whether PHI avoidance is a policy layered on top of a system that technically captures it.
Language Access and Accessibility: Non-Negotiable Vendor Questions
Language access is a legal obligation for most public agencies under Title VI of the Civil Rights Act, not a nice-to-have feature. An AI system that performs well in English and poorly in Spanish, Vietnamese, or Haitian Creole is not a technical limitation. It is a compliance gap, and it disproportionately fails the residents most dependent on public services. The vendor question here is specific: "What language pairs have you load-tested for accuracy, not just translation?" Translation and accuracy are different claims. A system can translate a sentence correctly and still give the wrong benefits information because it misunderstood an ambiguous phrase, a regional dialect, or a culturally specific way of describing a symptom or situation. Ask for accuracy testing data by language, not a marketing claim of "50 languages supported."
Accessibility is the same category of non-negotiable, not an afterthought bolted on after launch. WCAG 2.1 AA compliance should be a procurement gate, tested before contract signature, not a promise to fix post-launch. Ask whether the system has been tested with actual screen readers (JAWS, NVDA, VoiceOver), not just automated accessibility scanners, which catch a fraction of real usability problems for residents using assistive technology. A system that fails a blind resident trying to apply for benefits is as much a liability event as one that gives wrong information, and it is far more likely to trigger a formal complaint under the ADA.
AI Pilot Design Government Teams Should Insist On
The most common failure in public sector AI deployment is not a bad system. It is a pilot with no defined end. A "soft launch" that quietly becomes permanent because no one set criteria for success or failure in advance, and no one wants to be the person who shuts down a program that already has political momentum. AI pilot design government teams should insist on four elements before any resident interacts with the system. First, a defined population: a specific service area, a specific caseload segment, or a specific time window, not "everyone who calls." Second, explicit success and failure metrics set before launch, covering both accuracy and resident satisfaction, not just call volume handled. Third, a fixed evaluation date with a real decision point, not an indefinite extension. Fourth, a kill switch: a documented, tested process to shut the system off and revert to prior workflows within a defined number of hours, not days.
A pilot without a kill switch is not a pilot. It is a deployment with a delayed announcement. Agencies that build the off-ramp before the on-ramp protect both residents and their own credibility if the system underperforms.
Programs serving students face a parallel version of this problem: campus systems fielding sensitive questions need the same defined scope and escalation clarity as public agency deployments, which is why student support programs evaluating AI tools tend to apply comparable due diligence around escalation and privacy, even though the resident population and legal framework differ from a county or state agency.
