ChatGPT-4 answers medical questions better thank doctors

Study Finds Chatbot Answers Medical Questions Better Than Doctors

June 2023

There's a lot of hype surrounding ChatGPT (chat.openai.com), currently the most widely used artificial intelligence text generator. Of course, I'm a progenitor of such hype. But when the hype comes from a journal of the American Medical Association, it lends a certain credibility. And, if anything, the results of a recent study published in JAMA Internal Medicine actually exceed the hype.

The authors of the study wondered how ChatGPT's answers to medical questions from individuals would compare to answers given by medical professionals. They selected 195 medical questions that had been answered in Reddit's AskDocs forum by verified healthcare professionals. They then asked ChatGPT to also answer each of those 195 questions.

Then they had a panel of three licensed healthcare professionals read the two answers to each question without knowing anything about the source of the answer. These physicians were simply asked which of the two answers was best in terms of quality and in terms of empathy.

ChatGPT excelled on both counts. In terms of quality, the raters found that 79% of ChatGPT's responses were good or very good compared to 22% of the responses from doctors. In terms of empathy, the raters found that 45% of ChatGPT's responses were empathic or very empathic compared to 5% of the doctors' responses.

This doesn't mean you should use ChatGPT instead of a doctor. ChatGPT itself always cautions that it's important for a person to consult a healthcare professional or specialist.

What it does mean is that ChatGPT can do astonishing things.

A friend of mine recently mentioned in passing how grateful he was that I had written about ChatGPT--and went on to tell me some extraordinary things it's done for him. Years ago, in his master's thesis in geometry at the University of California Davis, he discussed a general theorem that he determined would be difficult to prove and would be well beyond the scope of his proof of a specific instance of invariant convexity. So recently, out of curiosity, he asked ChatGPT to prove this more general theorem. ChatGPT did so in seconds, and my friend was thrilled.

He was also excited about his use of it to translate articles on his Chinese friend's website into English. He gave one of the translations to this friend, who is fluent in both languages, and she said the translation was perfect and better than a human could do.

Let's remember, though, that everything ChatGPT does is gleaned from human texts. It's not itself intelligent. It's as if it has assimilated the world's knowledge and can, on demand, apply it your specific prompt.

Google is very afraid. It seems antiquated in comparison, and Google knows that unless it comes up with something revolutionary to compete with the new Large Language Model chatbots, its business could vanish. To that end, Google is obsessed with modernizing it search engine, and, as I write this in early May, the company is planning to begin rolling out some changes this month. Features they're exploring include using a chatbot to search for music and the ability to interact with a chatbot while using their Chrome browser.

They've also rolled out their Bard chatbot to a limited number of users, but when I asked it to write a history of Fairfield, the result was bizarre. It said that Fairfield was located near the Chisholm Trail and that it was a "major center of the cattle trade and was home to several stockyards and packing plants." Also, I haven't been able to figure out how to see my previous conversations with Bard.

Microsoft, too, is desperate to compete, but their Bing Chat feature, which is gradually being made available, is inferior as I write this. There doesn't seem to be any way to see its previous responses. That's a key feature that ChatGPT does well. When I asked a specific medical question to ChatGPT-4, I was able to go back weeks later and ask follow-up questions in the conversation.

Microsoft is so desperate to take advantage of the current upheaval in search that they've begun offering rewards for using their Bing search engine. You can earn points every time you search with Bing, and then use those for gift cards or other prizes.

Meanwhile, ChatGPT-4 forges ahead, offering more features, such as plugins that increase its capabilities. ChatGPT-4 is also now available on smartphone apps. My favorite so far is ChatOn, which gives you access to ChatGPT-4 for $40 per year compared to $20 per month when you access it via your browser.

I've subscribed to both, of course. I don't ever want to be without ChatGPT-4 nearby. But if you're reluctant to pay, you can still use version 3.5 for free (chat.openai.com). It's actually the version my friend used for his geometry proof and Chinese translation.

E-mail Jim Karpen