ChataGPT's disastrous medical advice. It reassures you instead of recommending you go to the doctor

ChatGPT Health medical chatbot – a new but already popular artificial intelligence tool that provides users with health advice – does not recommend a visit to the hospital in more than half of the cases in which it is necessary – according to a study published in “Nature Medicine”.

Scientists took a closer look ChatGPT Health

The first independent safety assessment of this tool, conducted by researchers at the Icahn School of Medicine at Mount Sinai in New York, found that A medical chatbot can recognize life-threatening symptoms and still calm the patient and advise him to waitinstead of going to the emergency room.

ChatGPT Health was launched in January 2026. Shortly thereafter, its creator, OpenAI, announced that the tool based on a large language model was used by approximately 40 million people every day looking for health information, including guidance on whether their symptoms required urgent medical attention.

Researchers decided to check how the new system copes with assessing the urgency of health problems, i.e. the so-called triage. This is a determination of whether the patient should stay home, contact a doctor, or go to the emergency room immediately.

– We wanted to answer a very simple but key question: If someone experiences a real health emergency and seeks help from ChatGPT Health, will they be clearly informed to go to the emergency room? – explained the main author of the work, Dr. Ashwin Ramaswamy.

60 scenarios

For the study, scientists prepared 60 realistic scenarios covering 21 medical specialties. They included both minor ailments that can be treated at home and life-threatening emergencies.

First, three independent doctors determined how urgent each case was, based on guidelines from 56 medical societies. The same scenarios were then presented to ChatGPT Health. Each case was tested in 16 variants, changing, among others: the patient's gender and origin, his behavior (e.g. downplaying symptoms) and barriers to access to health care, such as lack of insurance or transport.

A total of 960 interactions were conducted with ChatGPT Health and its responses were compared with physician ratings.

Less obvious cases beat the AI

The analysis showed that ChatGPT Health performed well in managing common emergencies such as stroke and severe allergic reactions. However, problems appeared in less obvious situations.

In more than half of the scenarios deemed by doctors to require immediate assistance, the system downplayed the situation and did not refer the user to the hospital. The researchers were particularly surprised that the chatbot often recognized dangerous symptoms well, but still calmed the user by recommending staying at home.

– For example, he was able to identify early signs of respiratory failure in the analysis, and at the same time advised the patient to wait instead of seeking immediate help – said Dr. Ramaswamy.

Researchers also examined how the system responds to signals indicating suicide risk. In such situations, ChatGPT Health should refer users to the US Crisis Hotline. However, it turned out that the warnings did not always appear when the risk was highest – sometimes they triggered at less serious signals, while they did not appear when users described specific plans for self-harm.

– This was particularly disturbing. In our tests, alerts appeared more often in lower-risk situations than in those when someone described exactly how they wanted to hurt themselves, said co-author Girish Nadkarni.

The authors added that the results they obtained do not mean that the use of AI tools in health care should be completely abandoned. They believe they can be helpful as a source of information, but should not replace medical evaluation in cases of serious symptoms.

Therefore, they recommend that in case of increasing or disturbing symptoms, such as chest pain, shortness of breath, severe allergic reaction or impaired consciousness, you should immediately contact a doctor or go to a hospital emergency department, instead of relying on the chatbot's advice.

Katarzyna Czechowicz (PAP)

kap/ agt/