Roche asks:
Which organization's large language model (LLM) will be ranked first as of 30 September 2025, according to MedHELM's medical domain LLM leaderboard?
Closing Sep 30, 2025 07:01AM UTC
MedHELM is "a comprehensive healthcare benchmark to evaluate language models on real-world clinical tasks" created by a collaboration between, among others, Stanford's Center for Research on Foundational Models (CRFM) (Stanford - MedHELM). The question will be suspended on 29 September 2025 and the outcome determined using the ranks as reported by MedHELM at approximately 5:00p.m. ET on 30 September 2025 (Stanford - MedHELM Leaderboard, see "Mean win rate" on the "Accuracy" sheet tabbed near the top of the page; this page will be used for resolution). As of 2 June 2025, DeepSeek was ranked first with its "R1" scoring 0.663, followed by OpenAI with its "o3-mini (2025-01-31)" scoring 0.641. In the event of a tie for first place by LLMs of different organizations, the LLM with the higher "Mean win rate" on the "Efficiency" sheet tabbed near the top of the page will be considered first. If the named source changes the way it presents the data, further instructions will be provided.
Confused? Check our FAQ or ask us for help. To learn more about Good Judgment and Superforecasting, click here.
To learn more about how you can become a Superforecaster, see here. For other posts from our Insights blog, click here.
| Possible Answer | Crowd Forecast | Change in last 24 hours | Change in last week | Change in last month |
|---|---|---|---|---|
| Anthropic (e.g., "Claude") | 11.04% | 0% | +1.04% | +1.04% |
| DeepSeek | 28.32% | +0.28% | -3.68% | +1.32% |
| Google (e.g., "Gemini") | 19.52% | -0.20% | +0.52% | -5.48% |
| OpenAI (e.g., "03-mini," "GPT") | 30.96% | 0% | +1.96% | -0.04% |
| Another organization | 10.16% | -0.08% | +0.16% | +3.16% |