By I. Edwards HealthDay Reporter<\/b>
\n<\/b><\/p>\n
TUESDAY, May 13, 2025 (HealthDay News) \u2014 OpenAI has unveiled a large dataset to help test how well artificial intelligence (AI) models answer health care questions.<\/p>\n
Experts call it a major step forward, but they also say more work is needed to ensure safety.<\/p>\n
The dataset \u2014 called HealthBench \u2014 is OpenAI’s first major independent health care project. It includes 5,000 \u201crealistic health conversations,\u201d each with detailed grading tools to evaluate AI responses, STAT News <\/em>reported.<\/p>\n \u201cOur mission as OpenAI is to ensure AGI is beneficial to humanity,\u201d Karan Singhal<\/a>, head of the San Francisco-based company’s health AI team, said. AGI is shorthand for artificial general intelligence.<\/p>\n \u201cOne part of that is building and deploying technology,” Singhal said. “Another part of it is ensuring that positive applications like health care have a place to flourish and that we do the right work to ensure that the models are safe and reliable in these settings.\u201d<\/p>\n The dataset was created with help from 262 doctors who have worked in 60 countries. They provided more than 57,000 unique criteria to judge how well AI models answer health questions.<\/p>\n HealthBench aims to fix a common problem: Comparing different AI models fairly.<\/p>\n \u201cWhat OpenAI has done is they have provided this in a scalable way from a really big, reputable brand that\u2019s going to enable people to use this very easily,\u201d Raj Ratwani<\/a>, a health AI researcher at MedStar Health, said.<\/p>\n The 5,000 examples in HealthBench were made using synthesized conversations designed by physicians.<\/p>\n \u201cWe wanted to balance the benefits of being able to release the data with, of course, the privacy constraints of using realistic data,” Singhal told STAT News.<\/em><\/p>\n The dataset also includes a special group of 1,000 hard examples where AI models struggled. OpenAI hopes this group \u201cprovides a worthy target for model improvements for months to come,” STAT News<\/em> reported.<\/p>\n OpenAI also tested its own models as well as models from Google, Meta, Anthropic and xAI. OpenAI\u2019s o3 model scored the best, especially in communication quality, STAT News<\/em> reported.<\/p>\n But models performed poorly in areas like context awareness and completeness, experts said.<\/p>\n Some warned about OpenAI grading its own models.<\/p>\n “In sensitive contexts like health care, where we are discussing life and death, that level of opacity is unacceptable,” Hao explained.<\/p>\n Others noted that AI itself was used to grade some of the responses, which could result in errors being overlooked.<\/p>\n It \u201cmay hide errors shared by both model and grader,\u201d Girish Nadkarni<\/a>, head of artificial intelligence and human health at the Icahn School of Medicine at Mount Sinai in New York City, told STAT News.<\/em><\/p>\n He and others called for more reviews to ensure models work well in different countries and among different demographics.<\/p>\n \u201cHealthBench improves large language model health care evaluation but still needs subgroup analysis and wider human review before it can support safety claims,\u201d Nadkarni said.<\/p>\n More information<\/strong><\/p>\n The National Institutes of Health has more on artificial intelligence in health care<\/a>.<\/p>\n