Do I have to tell patients I used AI?

FSMB's 2024 policy recommends that physicians disclose to patients when and how AI is used in their care, and communicate its capabilities and limitations [13]. Requirements vary by state and by institution, and some organizations now have specific consent language. Check your local policy rather than relying on national guidance alone.

Does AI actually make physicians better at diagnosis?

The evidence so far says not automatically. In a randomized trial, physicians given LLM access scored a median 76% on diagnostic reasoning versus 74% with conventional resources, a difference that was not statistically significant, while the model alone scored 92% [3]. The gap points at how clinicians integrate the output, not at raw model capability.

Which clinical questions are safe to ask AI?

Questions where you can verify the answer quickly and where being wrong is recoverable: drafting patient-facing language, orienting yourself in an unfamiliar area, generating a differential to react to, summarizing a long chart you will still read. Dosing, interactions, and time-critical decisions are not in that category [5,10].

How do I know if a clinical AI tool is FDA cleared?

Check FDA's list of AI-enabled medical devices, which is publicly searchable and was last updated September 22, 2026, with more than 1,600 authorized devices [7]. Most general-purpose chatbots and many documentation tools are not on it, because they are not regulated as devices. Absence from the list is information, not a verdict.

Back to blogs

AI Point of Care Safety: A Physician's Guide

Where clinical AI fails at the bedside, the checks that catch it, and when to close the tab. Evidence-based guidance for practicing physicians.

Reviewed by O. Daniel Odutola, MD, MBA, MPHNovember 5, 202614 min read

AI point of care safety comes down to one habit: never act on an output whose basis you cannot check. Use these tools to speed up retrieval, widen a differential, or draft language you will edit. Do not let them be the last step before an order goes in, and do not ask them a question you have no way to verify.

Easy to agree with in a conference room. Harder at 2 a.m. with eleven patients waiting and a nurse at the workstation holding a syringe.

Most of us are past the question of whether to use AI. In the AMA's 2026 survey of nearly 1,700 physicians, 81% reported using it professionally, roughly double the 2023 share [1]. The live question is narrower and more uncomfortable: which tasks can you hand to a model, and what has to be true before you sign your name under the answer.

The failure mode is not the model being wrong

Everyone expects AI to be wrong sometimes. What gets underestimated is what a wrong answer does to a competent clinician who reads it.

A single-blind randomized trial published in NEJM AI put 44 physicians through 20 hours of AI literacy training, then had them work six clinical vignettes with ChatGPT-4o assistance. Half received clean recommendations. Half received recommendations seeded with deliberate errors in three of the six cases. Diagnostic reasoning accuracy fell from 84.9% with error-free suggestions to 73.3% with flawed ones, an adjusted difference of 14.0 percentage points (95% CI, -18.9 to -9.1; P < .0001). Top-choice diagnostic accuracy fell from 90.5% to 76.1% [2].

Read that again with the training in mind. These physicians had been taught, formally, that the model could be wrong. It did not protect them.

This is automation bias, and it is not a character flaw. It is what happens when a fluent, well-organized answer arrives before you have finished forming your own. It anchors you, and you end up evaluating its answer instead of building yours, which is a different and much easier task.

The companion finding is just as sobering. In a randomized trial of 50 physicians in JAMA Network Open, LLM access did not significantly improve diagnostic reasoning versus conventional resources (median 76% vs 74%; adjusted difference 2 points; 95% CI, -4 to 8; P = .60). The model alone scored a median 92% [3]. The tool was good. The handoff leaked the value.

ECRI ranked misuse of AI chatbots the number one health technology hazard for 2026, ahead of cybersecurity and device recalls [4]. That ranking is about the handoff, not the technology.

What AI point of care safety actually requires

There is a useful legal test hiding in plain sight that doubles as a bedside heuristic.

FDA's Clinical Decision Support Software guidance, revised January 6, 2026, lays out four criteria that keep software outside device regulation. The fourth is the interesting one: the software must enable health care professionals to independently review the basis for its recommendations, so that they do not rely primarily on it for clinical decisions [5]. FDA also holds that software intended to support time-critical decision making does not qualify as non-device CDS, though software used in time-critical settings can still qualify if it is doing supportive work like surfacing relevant patient data [6].

Strip out the regulatory language and you get a working rule for the clinic: if you cannot independently reconstruct why the answer is the answer, you cannot safely use the answer.

This is the difference between a tool that tells you ceftriaxone plus azithromycin and one that tells you ceftriaxone plus azithromycin, here is the guideline, here is the year, here is the population it was derived from. The first asks for your trust. The second offers a path to your own judgment. Only the second belongs near an order.

It also matters whether the tool was ever evaluated for clinical use at all. FDA has authorized more than 1,600 AI-enabled medical devices and maintains a searchable list, last updated September 22, 2026 [7]. A general-purpose chatbot is not on it. That does not make it useless, but it means nobody has vouched for it in your use case except the vendor and you.

For a broader treatment of where these tools fit in practice overall, see our guide to AI in clinical medicine. For how these tools compare with the reference sources you already use, see our breakdown of point-of-care tools.

See how your offer compares

Compare real pay data before you sign anything

Physicians' Copilot Clinical AI

Four clinical AI pitfalls that show up at the bedside

Fabricated sources that look real. In a blinded cross-sectional evaluation of three LLMs queried on ten neurocritical care topics under zero-shot, retrieval-disabled conditions, 165 of 300 generated references (55.0%) contained an inaccuracy and 85 (28.3%) were complete fabrications. Rates ranged from 23% for one model to 73% for another [8]. Retrieval-grounded clinical tools behave differently from a bare chatbot, which is exactly the point: the mechanism matters more than the brand. If citations are generated from memory rather than retrieved from a corpus, treat every one as unverified until you open it. Our deeper dive on AI hallucinations in medicine covers how to catch these before they land in a note.

Agreeableness dressed up as reasoning. Researchers at Mass General Brigham tested five frontier models with deliberately illogical drug requests, such as asking for a reason to switch patients from one drug to another when the two were the same compound under different names. GPT-4o, GPT-4o-mini and GPT-4 complied with the misinformation request in 100% of cases; a Llama3 model complied 94% of the time. Explicitly permitting the model to refuse, combined with a factual recall cue, raised rejection rates substantially, in two models to 94% [9]. Models are optimized to be helpful. If your prompt contains a flawed premise, many will build on it rather than correct you.

Confidence that does not track accuracy. A clinician-curated evaluation of five LLMs on 250 medication lists, each containing one clinically relevant interacting pair, found overall drug-drug interaction accuracy ranging from 54.1% to 83.7%. More importantly, the alignment between when a model declined to answer and when it was actually likely to be wrong was weak to moderate, and prompt-based mitigation did not reliably fix it [10]. The model does not hedge when it should. Its tone gives you no signal.

Silent omission. No study attached, because it is hard to measure, and it is the one that worries me most. A model that lists eight items on a differential and leaves off the ninth gives you nothing to react to. Fabrications are detectable. Gaps are invisible. This is why AI is safer as a second pass over a differential you already built than as the thing that builds it.

A 60-second verification loop you can actually run

Run this before any AI-influenced decision reaches a patient. It fits in the time you already spend double-checking a dose.

Form your own answer first, even a rough one. Ten seconds of independent thought before you read the output is the single highest-yield step, because it gives you something to compare against instead of something to agree with.
Ask what the recommendation rests on. If the tool cannot show you a source, a guideline, or a chain of reasoning you could audit, you are outside FDA's independent-review criterion and outside safe practice.
Open one source. Not the summary of the source. Check that it exists, that it says what the model says it says, and that the population resembles your patient.
Check the date on the evidence, not on the output. Models reproduce the guideline version that dominated their training data, which may be two revisions behind the one your department follows.
Test the premise deliberately. Ask the model what would make this recommendation wrong, or state the opposite and see whether it folds. Given the sycophancy data, a model that instantly agrees with a contradiction has told you something useful about its answer.
Stop at the high-risk categories. Anticoagulation, insulin, opioids, chemotherapy, pediatric weight-based dosing, and anything involving renal or hepatic adjustment get verified against your formulary reference or pharmacy, every time, regardless of how confident the output reads.
Document your reasoning, not the tool's. Your note should show why you did what you did. It should not read as though a model decided.

If a task fails step 2 or step 3, close the tab and use the resource you would have used in 2019. That is not the workflow failing. That is the workflow working.

See PCP's guardrails for point-of-care use.

Triage your tasks before you triage your patients

Not every clinical question carries the same risk. Sorting them once saves you from relitigating the decision a dozen times a shift.

Risk tier	Example tasks	What AI can do	What you must do
Low	Drafting patient instructions, rephrasing a plan for a family, summarizing a long chart, generating a broad differential to react to	Produce a first draft you will rewrite	Read every line; correct clinical content; own the final text
Moderate	Recalling a guideline threshold, comparing two treatment options, checking a diagnostic criterion, explaining an unfamiliar result	Point you toward the right source and vocabulary	Open the primary source and confirm before acting
High	Dosing, drug-drug interactions, anticoagulation decisions, oncologic regimens, pediatric weight-based calculations, anything time-critical	Nothing you rely on	Use a validated reference or pharmacy; AI output is a prompt to check, never the check itself
Off limits	Entering identifiable patient information into a consumer tool with no business associate agreement	Nothing	Do not do it

That last row deserves its own paragraph.

The privacy line, stated plainly

HIPAA requires a covered entity to obtain satisfactory assurances, in the form of a contract or other written arrangement, before disclosing protected health information to a business associate [11]. A consumer chatbot you signed up for with your personal email is not operating under a business associate agreement with your employer. Pasting a real history and physical into it is a disclosure.

The practical version: use the AI your organization has contracted for, inside the EHR or the enterprise account. To think out loud with a general-purpose model, strip identifiers first and keep the scenario generic. Unglamorous advice, and also the thing most likely to end up in a deposition.

Who is actually accountable, and it is not the vendor

State medical boards regulate physicians. They do not regulate AI developers. That asymmetry is the whole liability story, and a 2026 review in the Journal of the National Medical Association makes it explicit: clinicians remain fully accountable when AI-assisted care contributes to patient harm [12].

The Federation of State Medical Boards said the same thing more bluntly in its April 2024 policy, "Navigating the Responsible and Ethical Incorporation of Artificial Intelligence into Clinical Practice." Its language is worth memorizing: "Failure to apply human judgment to any output of AI is a violation of a physician's professional duties." The policy also holds that the physician is ultimately responsible for the use of AI and should be held accountable for any harms that occur, that physicians should disclose to patients when and how AI is used in their care, and that the duty to review AI-assisted records for accuracy stays with the physician [13].

On the institutional side, The Joint Commission and the Coalition for Health AI released initial guidance on the Responsible Use of AI in Healthcare on September 17, 2025, covering governance, local validation, and ongoing monitoring [14], and The Joint Commission announced a voluntary Responsible Use of AI in Healthcare certification in May 2026 [15]. If your organization has neither a governance committee nor a validation process for the AI already running in your workflow, that gap is worth raising in writing.

This article is educational and is not legal advice. For questions about your own liability exposure or your employment agreement's AI provisions, consult a licensed attorney.

No AI tool replaces clinical judgment. Verify all outputs against primary sources and your own assessment.

Key takeaways

The documented harm is not model error in the abstract. It is trained physicians agreeing with confident wrong answers: diagnostic reasoning dropped 14 percentage points when LLM advice contained errors, in clinicians who had completed AI literacy training [2].
FDA's fourth non-device CDS criterion is also the best bedside test: if you cannot independently review the basis for a recommendation, do not act on it [5].
Fabricated citations, sycophantic agreement with flawed premises, and confidence that does not track accuracy are measured, reproducible failure modes, not edge cases [8,9,10].
Sort tasks by risk before the shift starts. Dosing, anticoagulation, and pediatric calculations never come off an AI output unverified.
Accountability sits with you. Boards regulate physicians, not developers, and FSMB considers failure to apply human judgment to AI output a breach of professional duty [12,13].

Where this leaves you

The clinicians I worry about are not the ones who refuse to touch AI. They are the ones who have quietly started trusting it because it has been right for three months and nothing bad has happened. That is the point at which the 14-point drop finds you.

Keep the habit of forming your own answer first. It costs ten seconds and it is the only thing in this article that has been shown to work no matter which model you are using.