# What will be the highest success rate on ExploitGym's full 869-task suite reported for any AI model or agent system by 30 June 2027?

> As of 13 November 2026, Prolepsia’s median forecast is 70.3, with an 80% chance it falls between 57.7 and 88.3.

- Forecaster: Prolepsia (https://prolepsia.com)
- Forecast made: 2026-11-13
- Question type: Number
- Resolves: 2027-07-15
- Status: Open
- Forecasts: 1
- Page: https://prolepsia.com/forecasts/isotherm-quasar-theta/what-will-be-the-highest-success-rate-on-exploitgym-s-full-869-task-suite-report
- Open in the app: https://app.prolepsia.com/library/2513a554-81f4-4959-aac3-c33598db122b

## Forecast

- 10th percentile: 57.7
- Median: 70.3
- 90th percentile: 88.3

## How Prolepsia reasons

## TL;DR
The forecast for the highest qualifying ExploitGym success rate by 30 June 2027 is a median of 70.3%, with a 10th percentile of 57.7% and a 90th percentile of 88.3%. Google’s documented full-suite result of 54.8% has already surpassed the question’s GPT-6 Astra baseline ([safety report](https://storage.googleapis.com/deepmind-media/gemini/gemini_4_carbon_safety_report.pdf)). The central expectation is further substantial progress, with the attainable task ceiling as the main constraint.

## Context
ExploitGym tests whether an agent can turn a known vulnerability into a working exploit. Each task supplies vulnerable source code, a bug-triggering input and a test environment. Success requires reading a secret flag through the intended vulnerability, not simply finding another way to capture it. The frozen suite contains 502 userspace tasks, 181 Chrome V8 tasks and 186 Linux kernel tasks ([benchmark](https://www.cybergym.io/exploitgym/)).

The question concerns the best qualifying single submission publicly disclosed by 30 June 2027 at 23:59 UTC. Any model or named agent system can count, with any time budget or number of attempts. But the submission must cover the original 869-task suite in the main, mitigations-off configuration. Separate runs cannot be combined after the fact. These rules make both technical progress and public documentation decisive.

## Evidence
The historical backbone is a rising record on the frozen suite. There is no broad empirical base rate that establishes how quickly this particular benchmark must improve. Its own history is the most relevant evidence, but it mixes different models, budgets and agent designs. It shows progress, not a controlled learning curve.

The dated official results and subsequent developer disclosures give the following history. Not every row set a new record; retaining the lower results shows that newer releases do not automatically outperform the leader.

| Date | Model | Reported full-suite result | Source |
|---|---|---:|---|
| 28 May 2026 | Claude Opus 4.8 | 120 successes | [Leaderboard](https://www.cybergym.io/exploitgym/) |
| 9 June 2026 | Claude Mythos 5 | 247 successes | [Leaderboard](https://www.cybergym.io/exploitgym/) |
| 13 July 2026 | GPT-5.6 Sol | 293 successes | [Leaderboard](https://www.cybergym.io/exploitgym/) |
| 1 September 2026 | Claude Mythos 5.1 | 264 successes | [Leaderboard](https://www.cybergym.io/exploitgym/) |
| 3 September 2026 | GPT-6 Astra | 368 successes; 42.35% | [Leaderboard](https://www.cybergym.io/exploitgym/) |
| 22 September 2026 | Claude Opus 5.5 | 300 successes | [Leaderboard](https://www.cybergym.io/exploitgym/) |
| 29 September 2026 | GPT-6.1 Sol | 305 successes | [Leaderboard](https://www.cybergym.io/exploitgym/) |
| 28 October 2026 announcement | Gemini 4 Carbon | 54.8% | [Announcement](https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/) |
| 5 November 2026 | GPT-6 Cyber | 48.45% | [System card](https://deploymentsafety.openai.com/gpt-6-cyber/gpt-6-cyber.pdf) |
| 9 November 2026 | Claude Mythos 5.2 | 331 successes at two hours; 341 at six hours | [System card](https://www.anthropic.com/claude-fable-5-2-mythos-5-2-system-card) |

**The apparent plateau is partly a reporting lag.** Google’s November safety report explicitly identifies the full denominator, intended-vulnerability grading and mitigations-off setting. It reports Carbon at 53.5% with a six-hour limit and 54.8% with a ten-hour limit. These are different budgets, not conflicting measurements of the same run. The higher aggregate meets the question’s stated reporting provisions; no unpublished numerator needs to be inferred ([Google safety report](https://storage.googleapis.com/deepmind-media/gemini/gemini_4_carbon_safety_report.pdf)).

The suite’s history also creates a comparability trap. The original paper studied 898 tasks. The June public release reduced that to 869 by filtering non-exploitable cases. Earlier paper aggregates therefore cannot simply be placed alongside frozen-suite results or used to extend the same trend backward ([official changelog](https://raw.githubusercontent.com/sunblaze-ucb/exploitgym/main/CHANGELOG.md)). The forecast concerns the frozen denominator even if a later benchmark version appears.

Runtime evidence supports continued gains, but not effortless saturation. Carbon’s improvement from six to ten hours is 1.3 percentage points. In the same report, Gemini 4 Argon scores 39.5% at six hours and 42.0% at ten hours. Both models benefit from more time, but neither comparison supports assuming that much longer runs solve nearly everything ([Google safety report](https://storage.googleapis.com/deepmind-media/gemini/gemini_4_carbon_safety_report.pdf)).

A separate full-suite AIOS-Cyber run provides a complete set of cumulative checkpoints: 152 intended successes at two hours, 200 at four hours and 218 at six hours. The successive increments shrink from 48 to 18. These are checkpoints within one run, not independent trials, but they show both the value of persistence and diminishing returns ([evaluation documentation](https://github.com/AIOS-Cyber/AIOS-Cyber/blob/main/README.zh-CN.md)). The inference is that large further gains need better models or better systems, not just more runtime under an unchanged approach.

Agent design is a credible source of those gains. In DoGNAVY’s matched 40-task ablation, the complete system solved 30 tasks. Removing stages produced 24 successes; removing expertise also produced 24; removing parallel execution produced 18; removing memory produced 28. This supports a real role for organization and parallel work. The sample is small and selected, so applying the same improvement factor to the full frontier result would overstate the evidence ([DoGNAVY paper](https://arxiv.org/html/2611.04196v1)).

Subset results need similar discipline. VARAS-OneMind reports 206 intended successes using DeepSeek-V4.1-Flash on 314 tasks. That demonstrates useful capability, but it does not establish performance over all 869 tasks. The omitted tasks cannot be assumed equally easy, and the subset’s headline success rate is not a qualifying full-suite record ([VARAS report](https://varas-onemind.github.io/posts/exploitgym/)).

Scoring matters as much as coverage. The AIOS-Cyber full-suite submission reports 261 flag captures but only 218 intended-vulnerability successes, with 43 off-target captures. An alternate judging exercise produced 213 successes rather than 218. This is evidence of correction risk, not an additional capability gain ([submission](https://github.com/sunblaze-ucb/exploitgym-results/pull/7); [judging documentation](https://github.com/AIOS-Cyber/AIOS-Cyber/blob/main/README.zh-CN.md)). A result can look stronger under raw flag capture while making no corresponding progress on the metric that resolves this question.

The strongest evidence against a high outcome concerns exploitability. The paper warns that a reproducible bug does not guarantee a usable exploit and that ground-truth exploits are unavailable for every task ([benchmark paper](https://arxiv.org/html/2605.11086v1)). A July account of correspondence with the authors reports their informal estimate that 60–70% of tasks are solvable with mitigations disabled ([firsthand account](https://abstatisticalconsulting.substack.com/p/brief-notes-on-the-openaihugging)). That estimate is a serious constraint on the optimistic reading. It is not a demonstrated hard cap: no complete task-level solvability census establishes it.

There is also an upside signal beyond established results. A contemporaneous report describes an OpenAI keynote slide showing 77.0% for an upcoming cyber model. Its model identity, task population and grading are not established by an original methods statement. The same report gives a GPT-6 Cyber comparator that differs from its published system-card result, adding a comparability warning. The preview is evidence of a possible development pipeline, not a qualifying record or a floor ([keynote coverage](https://www.gizmodo.jp/article/2611-openai-next-gen-cyber-ai-exploitgym/)).

## What's non-obvious
The obvious reading asks whether a model will finally beat Astra. The better reading starts from the fact that a documented developer result already has. The question accepts more than the official leaderboard, so leaderboard inactivity is not evidence of capability stagnation. Conversely, a dramatic preview or selected-subset score is not enough. Documentation can turn a real technical advance into an eligible result—or leave it outside the outcome entirely.

The other distinction is between difficult tasks and impossible tasks. More capable agents can overcome the former, but not the latter under a frozen environment and intended-vulnerability rule. A missing known exploit does not prove impossibility. Yet repeated improvements in model strength do not prove that the whole suite is exploitable either. The wide forecast range reflects this unresolved distinction, rather than assuming either a firm low ceiling or inevitable saturation.

## Uncertainties
The largest gap is a task-level account of what remains solvable. A validated set of intended exploits, or rigorous explanations of why particular tasks cannot be exploited under the benchmark settings, would narrow the uncertainty more than another aggregate headline. The informal authors’ estimate weighs against the forecast’s upper range without replacing it with a hard bound.

The leading developer result is documented, but public task-level replication of Carbon is not established. Exact success logs, reproducible configurations and independent review would reduce the risk of grading or reporting corrections. Percentage-only reporting and a longer budget are allowed; the remaining concern is whether the reported aggregate survives validation.

Finally, technical capability and qualifying disclosure are separate events. A stronger system must publish a sufficiently identified full-suite result before the deadline. There is no verified release roadmap establishing when that will happen, and progress on a successor suite would not resolve this question. Those publication constraints, together with the unknown solvability ceiling and uncertain transfer of scaffold gains, remain the main reasons for the forecast’s breadth.

## Forecast history

| Date | Forecast |
|---|---|
| 2026-11-13 | Median 70.3, 80% range 57.7 to 88.3 |

## Sources Prolepsia read

- [Gemini 4 Carbon Safety & Security Report](https://storage.googleapis.com/deepmind-media/gemini/gemini_4_carbon_safety_report.pdf)
- [https://www.cybergym.io/exploitgym/](https://www.cybergym.io/exploitgym)
- [Introducing Gemini 4 Carbon](https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon)
- [GPT-6 Cyber System Card](https://deploymentsafety.openai.com/gpt-6-cyber/gpt-6-cyber.pdf)
- [System card](https://www.anthropic.com/claude-fable-5-2-mythos-5-2-system-card)
- [official changelog](https://raw.githubusercontent.com/sunblaze-ucb/exploitgym/main/CHANGELOG.md)
- [AIOS-Cyber/README.zh-CN.md at main · AIOS-Cyber/AIOS-Cyber · GitHub](https://github.com/AIOS-Cyber/AIOS-Cyber/blob/main/README.zh-CN.md)
- [DoGNAVY-Exploit: Harnessing Parallel Open-Weight Agents for Stage-Aware Vulnerability Exploitation](https://arxiv.org/html/2611.04196v1)
- [VARAS-OneMind on ExploitGym: Capability-Led Exploitation at Scale \| VARAS](https://varas-onemind.github.io/posts/exploitgym)
- [Brief notes on the OpenAI/Hugging Face incident](https://abstatisticalconsulting.substack.com/p/brief-notes-on-the-openaihugging)

## The question

ExploitGym is a benchmark from UC Berkeley and partners (https://www.cybergym.io/exploitgym/, paper https://arxiv.org/abs/2605.11086). Each of its 869 tasks gives an AI agent vulnerable source code, an input that triggers the bug, and a test environment. The agent succeeds when it writes a working exploit that uses the intended vulnerability to read a secret flag. The suite has 502 userspace tasks, 181 Chrome V8 tasks and 186 Linux kernel tasks. As of 13 November 2026 the official leaderboard shows: GPT-6 Astra 368 of 869 (42.35%, 3 September 2026), the highest result of any model; GPT-6.1 Sol 305; Claude Opus 5.5 300; Claude Mythos 5.1 264. The best full-suite open-weight result on the leaderboard is GLM-5.3 with Claude Code at 130 of 869 (15.0%, rescaled six-hour budget, 18 August 2026). Developers report 20.7% for Kimi-K3.1-Pro and 17.8% for MiMo-V2.6-Pro-RL, but their full-suite coverage is not established. The agent system VARAS-OneMind reports 206 successes with DeepSeek-V4.1-Flash, but on a 314-task subset. This question asks whether the best result keeps rising after GPT-6 Astra, which no model has beaten since September 2026.

### How it resolves

The question resolves to the highest success rate of any qualifying result publicly disclosed on or before 30 June 2027 at 23:59 UTC, including GPT-6 Astra's 368 of 869. Any model counts, open or closed, and any single named agent system counts, including one that uses several models, as long as the result is reported as one submission. Combining the best tasks from separately reported runs does not count. A qualifying result uses the benchmark's main configuration, which is the main bar on the official leaderboard, not the configuration with exploit mitigations switched on. The success rate is the number of tasks solved through the intended vulnerability, as ExploitGym's method counts them, divided by 869 and multiplied by 100. Only results on the original frozen 869-task suite count; results on a subset, on a different benchmark version, or with a changed denominator do not count, and unsolved tasks stay in the denominator. Where a source gives exact counts, the counts decide; a percentage-only result counts when the source presents it as an official aggregate for the full 869-task suite. Any time budget, number of attempts, tool use and agent scaffolding is allowed. Sources, in order: the official ExploitGym website and its public results repository (https://github.com/sunblaze-ucb/exploitgym-results); then original developer technical reports, model cards, system cards and papers that identify the model, the configuration and full-suite coverage. Results later shown to be fabricated, miscalculated or on a materially different benchmark are excluded.

### Fine print

Disclosure dates are in UTC. A result counts by the date it was first published, not the date a leaderboard later lists it. The benchmark stays frozen at the original 869-task suite even if ExploitGym adds tasks or publishes a successor. If no source can establish any qualifying result, the question resolves to the best qualifying result already public on 13 November 2026.

---

Prolepsia’s forecasting engine made this forecast: it researched the question, weighed the evidence and wrote the report on this page. A probability is not a promise; this one changes as the evidence does.
