← Confide Blog
ai_overall

Large Language Models in Public Health: What Program Managers Should Know Before Adopting

by the Confide Team · November 19, 2026 · 8 min read

Program manager, illustrating large language models in public health: what program managers should know before adopting

A vendor wants thirty minutes of your time to demo an "AI health navigator." IT is already piloting a chatbot on the department intranet. Leadership wants to know, in the next budget cycle, whether this technology cuts costs. Somewhere in that sequence, a public health program manager is expected to have an informed opinion on large language models in public health, often without a technical background and without having asked for the job.

This is the primer that should exist before that first vendor meeting: what these tools actually do, why this decision has landed on your desk now, and what to ask before anyone signs anything.

What Large Language Models Actually Do (And Don't Do)

A large language model is a system trained on enormous volumes of text that predicts the most likely next words in a response. That's it. It is not a database of verified facts, and it is not a clinical decision-maker. It generates text that sounds plausible based on patterns it has seen, not text that has been checked against current guidance.

That distinction matters enormously in public health. A model can produce a confident, well-formatted, completely wrong answer about PrEP eligibility or isolation guidance. This is commonly called hallucination, and it is a known, documented limitation, not an edge case. The model has no mechanism for knowing what it doesn't know.

Public health informatics guidance generally expects AI tools to have human oversight structures, not blind trust in outputs. Data governance frameworks from federal health agencies tend to assume AI-generated content needs review, not that it replaces review. Any vendor who doesn't bring this up unprompted is a flag worth noting.

None of this means the technology is useless. It means it does one thing well, pattern-based text generation, and that one thing has to be matched to a task where being occasionally, confidently wrong is an acceptable and catchable risk.

Why Public Health Program Managers Are Facing This Decision Now

Three pressures are converging at once. Budgets are tight, and anything marketed as a staffing multiplier gets leadership attention fast. Case managers and navigators are stretched across growing caseloads, particularly in HIV and PrEP programs where retention work is labor-intensive and relationship-driven. And software vendors have moved quickly to rebrand existing chat tools as "AI-powered," often faster than procurement and compliance teams can evaluate the claims.

Program managers are now the ones in the room when IT proposes a pilot or a vendor pitches a tool, expected to weigh in on workflow fit and risk even though nobody trained them on how to evaluate AI claims in public health. That's a structural gap, not a personal failing. It's also exactly why the procurement conversation needs a different set of questions than "does this work."

The Questions to Ask Before Any AI Procurement Conversation

Before any serious conversation about AI procurement public health teams should walk in with, these five categories of questions should be answered in writing, not verbally promised in a demo.

Where does input text go? If a navigator types a client's question into a chatbot interface, does that text leave your system? Is it sent to a third-party model provider, stored, or used to further train the vendor's model? If the answer is unclear, treat it as a yes to the worst case until proven otherwise.

Is PHI ever exposed to the model? Many tools are marketed as HIPAA-compliant when no such blanket certification exists for AI vendors. Ask specifically whether protected health information is ever passed into the model itself, versus handled by PHI-free design that keeps identifying clinical details out of the system entirely. These are very different architectures with very different risk profiles.

What evidence exists from a comparable setting? Not a case study from a tech conference. Ask for performance data from another public health program, ideally one with a similar population and use case. If no comparable deployment exists, you are the pilot, and your budget is funding the vendor's R&D.

What does human-in-the-loop review actually look like? Get specifics. Who reviews outputs, how often, and what's the escalation path when a response is flagged as wrong? A vague answer here usually means no real review process exists.

Who is liable when the model is wrong? If a chatbot gives a student or patient inaccurate guidance and harm results, where does accountability sit? Get this in contract language, not in a sales conversation. Legal and compliance teams will ask this regardless of whether you do.

Where Large Language Models Help Public Health Programs, and Where They Don't

The useful applications tend to be low-stakes and high-volume: drafting first-pass outreach messages for staff to edit, summarizing long intake notes for a case manager's review, answering frequently asked logistical questions like clinic hours or appointment rescheduling. In these cases, a wrong output is annoying, not dangerous, and a human is already in the loop before anything reaches a client.

The risky applications are the ones where the model's output goes directly to a patient or student without review, especially around clinical guidance, medication information, or anything touching mental health or crisis-adjacent topics. A model should never be positioned as a crisis service or a substitute for a clinician, and any vendor framing it that way should be a dealbreaker.

This is where workflow design matters more than model quality. A well-built tool reduces navigator workload by handling routine triage and freeing staff for complex cases. A poorly scoped one creates new oversight work, because now someone has to monitor the AI on top of doing the job it was supposed to lighten. For campus-facing programs, where student health and support services often field sensitive, non-clinical questions, the design principle that works best is routing anything ambiguous or high-stakes to a human, by default, not by exception.

Governance, Data, and Accountability: What to Lock Down Before You Sign

Three things need to be settled before a contract is signed, not after an incident. First, data residency and retention: where is conversation data stored, for how long, and under whose jurisdiction. This matters for state health departments bound by their own data-sharing restrictions and for any program subject to funder audit requirements.

Second, an audit trail. If a state health department or funder reviews your program, can you show how the AI tool was used, what it was and wasn't permitted to say, and what review process caught errors? If the vendor can't produce this documentation, you can't produce it either when asked.

Third, a clear statement of what decision is being automated. This is the question underneath all the others. It's not "should we use AI." It's "what decision are we automating, and who is accountable when it's wrong." A chatbot that drafts outreach language is automating a low-stakes decision. A chatbol that tells a student whether their symptoms warrant a clinic visit is automating a much higher-stakes one, even if it's built on the exact same underlying model.

Government and public agency programs face a similar version of this question when evaluating any confidential communication platform for public agencies, where the same principle applies: tools built to avoid exposing sensitive data by design hold up better under scrutiny than tools that rely on after-the-fact compliance claims. The architecture, not the marketing language, is what survives an audit.

Frequently Asked Questions

No. LLMs generate plausible text based on patterns, not verified clinical judgment, and they cannot take accountability for a client outcome. The realistic use case is offloading routine, low-stakes tasks so navigators have more time for complex casework, not replacing the relationship-driven parts of the job.
No. There is no HIPAA certification body for software products, so any vendor using that exact phrase should be questioned closely. The more accurate framing to look for is whether a tool is PHI-free by design or whether it has a signed business associate agreement covering specific data handling practices.
The biggest risk is deploying a tool in a patient- or student-facing role without a human review loop, which exposes the program to liability if the model gives confidently wrong guidance. A close second is signing a contract without clarity on data handling, which can create compliance exposure that surfaces later during a funder or state audit.
Ask for performance evidence from a comparable public health deployment, not a general product demo or unrelated industry case study. If the vendor cannot produce this, assume your program would be the first real-world test and budget, staff time, and oversight accordingly.
It depends entirely on scope. Tools confined to low-stakes, high-volume tasks like drafting outreach messages or answering logistical questions tend to reduce workload. Tools positioned to handle sensitive or clinical questions directly often create new oversight burden, because staff then have to monitor the AI's outputs in addition to their existing responsibilities.
Insist on explicit terms covering where input data goes, whether it's used for further model training, what the human review process looks like, and who is liable if the model produces harmful or inaccurate output. If a vendor is reluctant to put these terms in writing, treat that reluctance as the answer.
Student privately messaging a confidential support channel
Get Started

See how Confide works for your organization

A 20-minute walkthrough of the platform, tailored to your program.

Book a Demo