A vendor wants thirty minutes of your time to demo an "AI health navigator." IT is already piloting a chatbot on the department intranet. Leadership wants to know, in the next budget cycle, whether this technology cuts costs. Somewhere in that sequence, a public health program manager is expected to have an informed opinion on large language models in public health, often without a technical background and without having asked for the job.
This is the primer that should exist before that first vendor meeting: what these tools actually do, why this decision has landed on your desk now, and what to ask before anyone signs anything.
What Large Language Models Actually Do (And Don't Do)
A large language model is a system trained on enormous volumes of text that predicts the most likely next words in a response. That's it. It is not a database of verified facts, and it is not a clinical decision-maker. It generates text that sounds plausible based on patterns it has seen, not text that has been checked against current guidance.
That distinction matters enormously in public health. A model can produce a confident, well-formatted, completely wrong answer about PrEP eligibility or isolation guidance. This is commonly called hallucination, and it is a known, documented limitation, not an edge case. The model has no mechanism for knowing what it doesn't know.
Public health informatics guidance generally expects AI tools to have human oversight structures, not blind trust in outputs. Data governance frameworks from federal health agencies tend to assume AI-generated content needs review, not that it replaces review. Any vendor who doesn't bring this up unprompted is a flag worth noting.
None of this means the technology is useless. It means it does one thing well, pattern-based text generation, and that one thing has to be matched to a task where being occasionally, confidently wrong is an acceptable and catchable risk.
Why Public Health Program Managers Are Facing This Decision Now
Three pressures are converging at once. Budgets are tight, and anything marketed as a staffing multiplier gets leadership attention fast. Case managers and navigators are stretched across growing caseloads, particularly in HIV and PrEP programs where retention work is labor-intensive and relationship-driven. And software vendors have moved quickly to rebrand existing chat tools as "AI-powered," often faster than procurement and compliance teams can evaluate the claims.
Program managers are now the ones in the room when IT proposes a pilot or a vendor pitches a tool, expected to weigh in on workflow fit and risk even though nobody trained them on how to evaluate AI claims in public health. That's a structural gap, not a personal failing. It's also exactly why the procurement conversation needs a different set of questions than "does this work."
The Questions to Ask Before Any AI Procurement Conversation
Before any serious conversation about AI procurement public health teams should walk in with, these five categories of questions should be answered in writing, not verbally promised in a demo.
Where does input text go? If a navigator types a client's question into a chatbot interface, does that text leave your system? Is it sent to a third-party model provider, stored, or used to further train the vendor's model? If the answer is unclear, treat it as a yes to the worst case until proven otherwise.
Is PHI ever exposed to the model? Many tools are marketed as HIPAA-compliant when no such blanket certification exists for AI vendors. Ask specifically whether protected health information is ever passed into the model itself, versus handled by PHI-free design that keeps identifying clinical details out of the system entirely. These are very different architectures with very different risk profiles.
What evidence exists from a comparable setting? Not a case study from a tech conference. Ask for performance data from another public health program, ideally one with a similar population and use case. If no comparable deployment exists, you are the pilot, and your budget is funding the vendor's R&D.
What does human-in-the-loop review actually look like? Get specifics. Who reviews outputs, how often, and what's the escalation path when a response is flagged as wrong? A vague answer here usually means no real review process exists.
Who is liable when the model is wrong? If a chatbot gives a student or patient inaccurate guidance and harm results, where does accountability sit? Get this in contract language, not in a sales conversation. Legal and compliance teams will ask this regardless of whether you do.
Where Large Language Models Help Public Health Programs, and Where They Don't
The useful applications tend to be low-stakes and high-volume: drafting first-pass outreach messages for staff to edit, summarizing long intake notes for a case manager's review, answering frequently asked logistical questions like clinic hours or appointment rescheduling. In these cases, a wrong output is annoying, not dangerous, and a human is already in the loop before anything reaches a client.
The risky applications are the ones where the model's output goes directly to a patient or student without review, especially around clinical guidance, medication information, or anything touching mental health or crisis-adjacent topics. A model should never be positioned as a crisis service or a substitute for a clinician, and any vendor framing it that way should be a dealbreaker.
This is where workflow design matters more than model quality. A well-built tool reduces navigator workload by handling routine triage and freeing staff for complex cases. A poorly scoped one creates new oversight work, because now someone has to monitor the AI on top of doing the job it was supposed to lighten. For campus-facing programs, where student health and support services often field sensitive, non-clinical questions, the design principle that works best is routing anything ambiguous or high-stakes to a human, by default, not by exception.
Governance, Data, and Accountability: What to Lock Down Before You Sign
Three things need to be settled before a contract is signed, not after an incident. First, data residency and retention: where is conversation data stored, for how long, and under whose jurisdiction. This matters for state health departments bound by their own data-sharing restrictions and for any program subject to funder audit requirements.
Second, an audit trail. If a state health department or funder reviews your program, can you show how the AI tool was used, what it was and wasn't permitted to say, and what review process caught errors? If the vendor can't produce this documentation, you can't produce it either when asked.
Third, a clear statement of what decision is being automated. This is the question underneath all the others. It's not "should we use AI." It's "what decision are we automating, and who is accountable when it's wrong." A chatbot that drafts outreach language is automating a low-stakes decision. A chatbol that tells a student whether their symptoms warrant a clinic visit is automating a much higher-stakes one, even if it's built on the exact same underlying model.
Government and public agency programs face a similar version of this question when evaluating any confidential communication platform for public agencies, where the same principle applies: tools built to avoid exposing sensitive data by design hold up better under scrutiny than tools that rely on after-the-fact compliance claims. The architecture, not the marketing language, is what survives an audit.





