AI health advice may backfire despite accuracy

More than 300 million people each week turn to ChatGPT for health-related queries. The U.S. government now treats artificial intelligence as a key tool for public health improvements, calling it the “holy grail” of medical progress. Yet mounting evidence indicates that even when AI delivers accurate health information, its messaging can sometimes worsen outcomes.

The problem lies in how AI frames advice. Public health research shows that truthful messages about risky behaviors—such as binge drinking or adolescent drug use—have occasionally produced unintended consequences, including lower vaccination rates, increased smoking, or reduced cancer screenings. The focus on factual correctness alone overlooks behavioral risks that arise from presentation.

A study recently examined three leading AI chatbots, ChatGPT, Gemini, and Claude

ChatGPT also framed vaccination as a “socially responsible” act, positioning it as a duty to protect others. However, research demonstrates that altruistic appeals do not consistently boost vaccination rates and can sometimes backfire by creating resistance.

Two factually identical responses can produce opposite behavioral effects. For example, a smoker inquiring about lung cancer risks might receive two different answers: one noting that most smokers never develop the disease, a statement that could reassure them and reinforce the habit, and another highlighting risks while offering cessation strategies. Both responses are accurate, but their impact on behavior differs dramatically.

Regulators miss AI’s subtle health risks

Regulators and tech firms are beginning to address AI risks, but current efforts concentrate on preventing outright misinformation or immediate harm. A draft federal bill supported by Anthropic, the developer of Claude, proposes allowing the government to block AI models if they could cause serious injury or death to more than 50 people. However, this approach overlooks the broader, long-term health risks posed by AI advice. Nearly half of U.S. cancer deaths, for instance, stem from preventable factors like smoking and vaccine-preventable infections, areas where AI’s messaging could either mitigate or worsen harm over time.

Behavioral science offers a pathway forward. Companies already use randomized experiments to measure user engagement online. Applying the same method to AI health responses could reveal whether advice increases or decreases protective actions. This approach would not require new regulations, as the insurance industry’s crash tests demonstrate: voluntary safety standards can drive innovation beyond what laws mandate. If adopted, behavioral benchmarking would ensure AI health advice aligns with proven strategies for improving outcomes.

AI cannot remain neutral. Every response prioritizes certain facts while omitting others, shaping behavior regardless of intent. Physicians make similar choices, but effective doctors do more than recite facts; they guide patients toward evidence-based actions. Behavioral benchmarking would extend this principle to AI, ensuring its health advice supports better decision-making.

Behavioral science could make AI advice safer

Some argue AI should merely aggregate information without influencing behavior. Yet AI already shapes decisions, whether by framing vaccination as a common practice, reassuring harmful habits, or presenting medical advice in ways that either empower or discourage action. The debate over AI safety has largely focused on avoiding harm, but less attention has been given to the ethical obligation of doing good. Behavioral benchmarking could address this imbalance by ensuring AI actively helps people make healthier choices.

In October, several states filed lawsuits against OpenAI, alleging that ChatGPT provides dangerous medical advice. While the company has pledged to enhance its models, legal and industry responses remain centered on correcting blatant errors rather than addressing the subtler risks of accurate but counterproductive guidance.

The challenge now is whether AI’s role in healthcare can expand beyond accuracy to a standard where its advice not only avoids harm but actively promotes well-being.