Becker's Hospital Review

16 INNOVATION Mass General Brigham pits AI models against each other in arena-style challenge By Naomi Diaz A s regulatory frameworks for AI in healthcare remain in their infancy, Mass General Brigham, based in Somerville, Mass., is taking a proactive role with its Healthcare AI Challenge. is initiative aims to establish an unbiased, scalable platform for evaluating AI models, providing insights to guide institutional decision-making and support the development of broader regulatory standards. Keith Dreyer, PhD, chief data science officer at Mass General Brigham, traced the roots of this initiative to the organization's early integration of AI into clinical practice in 2016-17, initially focusing on medical imaging. "Back then, convolutional neural networks were driving breakthroughs in narrow AI solutions," Dr. Dreyer told Becker's. "ese models tackled specific clinical problems, such as detecting strokes or precancerous pulmonary nodules, leading to the FDA approving nearly 1,000 algorithms for clinical use over time." With the emergence of large language models, AI's potential has expanded dramatically — but so have the challenges. A key obstacle is the absence of regulatory frameworks for these advanced models. "e FDA has yet to clear any large language model solutions, despite their transformative potential," Dr. Dreyer said. "Healthcare providers see the immense promise, but safety and efficacy must be thoroughly evaluated before deploying these tools. at's precisely what the Healthcare AI Challenge and its collaborative testing platform, the Arena, are designed to address." Mass General Brigham partnered with Emory Healthcare, based in Atlanta; the University of Wisconsin-Madison Department of Radiology; the University of Washington Department of Radiology, based in Seattle; and the American College of Radiology. Together, they launched the Healthcare AI Challenge — a virtual, interactive series of events aimed at uniting healthcare professionals across institutions to explore and assess AI technologies in real-world clinical scenarios. Participants work with curated AI solutions, tackling tasks like medical image interpretation in a simulated environment known as "the Arena." "We knew early on that for this initiative to succeed, it needed to extend beyond our walls," Dr. Dreyer said. "By adopting a cloud- based architecture, we created a platform accessible to multiple organizations, allowing them to contribute their data, expertise, and unique challenges." e initiative employs a scalable, event-driven framework to evaluate AI models across various healthcare use cases. e first challenge centers on medical imaging, testing AI's ability to analyze X-rays and generate outputs such as dra radiology reports, differential diagnoses, and follow-up recommendations. Models are graded on a scale that mirrors clinical expertise, ranging from attending physicians to medical students. "is process generates a leaderboard that shows which models perform at expert levels," Dr. Dreyer said. "e top-performing models then undergo head-to-head comparisons with human clinicians in subsequent challenges." Future challenges will expand beyond imaging, targeting areas like summarizing electronic health records or predicting cancer risk. "e design is endlessly adaptable," Dr. Dreyer said. "Each challenge is tailored to the specific data, experts, and outcomes we aim to evaluate." Dr. Dreyer envisions the challenge as a learning opportunity for organizations, helping them understand the potential and limitations of AI. n Health systems opening AI hubs By Naomi Diaz A s hospitals and health systems seek to integrate AI into healthcare operations, many are establishing dedicated AI centers to test and develop new AI tools. Here are four hospitals and health systems that have announced the development of AI centers, as reported by Becker's in 2024: 1. New York City-based Mount Sinai opened the Hamilton and Amabel James Center for Artificial Intelligence and Human Health. The center will focus on developing AI tools to improve diagnosis, treatment, and overall patient care. 2. BJC Health System and Washington University School of Medicine, both based in St. Louis, have established a joint Center for Health AI. 3. In March, Nashville, Tenn.-based Vanderbilt University Medical Center announced the creation of the AI Discovery and Vigilance to Accelerate Innovation and Clinical Excellence center. This center will focus on artificial intelligence. 4. In February, Hartford (Conn.) HealthCare announced the creation of the Center for AI Innovation in Healthcare. This center will focus on the research and development, education and training, collaboration and partnerships, and ethical and regulatory guidelines of artificial intelligence. n

• Cover