# What will be the highest success rate on ExploitGym's full 869-task suite with exploit mitigations switched on, reported by 30 June 2027?

> As of 13 November 2026, Prolepsia’s median forecast is 56.2% success rate with mitigations on, with an 80% chance it falls between 36.3% and 76.1% success rate with mitigations on.

- Forecaster: Prolepsia (https://prolepsia.com)
- Forecast made: 2026-11-13
- Question type: Number
- Resolves: 2027-07-15
- Status: Open
- Forecasts: 1
- Page: https://prolepsia.com/forecasts/eclipse-crescent-decile/what-will-be-the-highest-success-rate-on-exploitgym-s-full-869-task-suite-with-e
- Open in the app: https://app.prolepsia.com/library/069fd11f-35f6-461a-90b3-19957b8254db

## Forecast

- 10th percentile: 36.3% success rate with mitigations on
- Median: 56.2% success rate with mitigations on
- 90th percentile: 76.1% success rate with mitigations on

## How Prolepsia reasons

## TL;DR
The forecast for the highest qualifying mitigations-on success rate by 30 June 2027 is a median of 56.2%. The 10th percentile is 36.25%, and the 90th percentile is 76.15%. Plan for substantial erosion of the benchmark’s defenses, while keeping demonstrated capability separate from a fully qualifying public result.

## Context
ExploitGym asks an agent to turn a supplied vulnerability into an exploit that reads a secret flag through the intended bug. The frozen suite contains 869 tasks: 502 userspace, 181 Chrome V8 and 186 Linux kernel tasks ([benchmark](https://www.cybergym.io/exploitgym/)). This question concerns the benchmark’s own mitigations-enabled configuration, not whether a model provider enables refusal safeguards. The target defenses and their implementation differ across task families ([configuration documentation](https://github.com/sunblaze-ucb/exploitgym/blob/main/docs/defenses.md)).

The outcome is the highest qualifying result publicly disclosed by 30 June 2027 at 23:59 UTC. Any budget, attempt count or named agent system is allowed, but the denominator stays at 869 and unsolved tasks remain in it. This favors ambitious evaluations rather than ordinary deployment performance. It also makes documentation and publication part of the outcome: a strong subset score or an undocumented hardened run does not automatically qualify.

## Evidence
The historical backbone is a small set of paired defended and undefended evaluations. It is not a long, standardized growth series. The original May paper used 898 tasks and reported mitigation-disabled to mitigation-enabled counts of 54 to 3 for GPT-5.4, 120 to 21 for GPT-5.5, and 157 to 45 for Mythos Preview. Hardened evaluations reran previously successful tasks. Those observations show that defenses can invalidate exploit paths, but they are not measurements on today’s frozen denominator ([original paper](https://arxiv.org/html/2605.11086v1)).

The June public release removed 29 non-exploitable cases, leaving the required 869 tasks. That explains the denominator change; it does not prove that every retained task is exploitable with defenses enabled ([changelog](https://github.com/sunblaze-ucb/exploitgym/blob/main/CHANGELOG.md)). The relevant paired leaderboard history is:

| Leaderboard date | Model | Budget or cutoff | Mitigations off: tasks solved | Mitigations on: tasks solved |
|---|---|---|---:|---:|
| 16 June 2026 | GPT-5.4 | Two hours | 61 | 3 |
| 16 June 2026 | GPT-5.5 | Two hours | 129 | 22 |
| 13 July 2026 | GPT-5.6 Sol | Two-hour cutoff | 216 | 84 |
| 13 July 2026 | GPT-5.6 Sol | Six hours | 293 | 136 |

These are the [official leaderboard’s counts](https://www.cybergym.io/exploitgym/). The supplied established defended baseline is therefore 136 of 869, or 15.65%. Sol’s gain from 84 to 136 defended successes shows meaningful budget sensitivity. But it is a comparison of cutoffs within a longer run, not evidence that independent retries deliver the same gain.

**The strongest newer evidence is the hardened result, not the ordinary leaderboard headline.** OpenAI’s 5 November 2026 system card reports the following separate evaluation figures ([system card](https://deploymentsafety.openai.com/gpt-6-cyber/gpt-6-cyber.pdf)):

| Model | Ordinary ExploitGym | ExploitGym Hardened |
|---|---:|---:|
| GPT-5.6 Sol | 30.3% | 16.3% |
| GPT-6 Astra | 42.43% | 29.69% |
| GPT-6 Cyber | 48.45% | 36.94% |

The report describes address randomization, canaries, the V8 sandbox and stricter kernel configurations. It indicates a smaller mitigation penalty for newer models. Yet these figures are not perfectly interchangeable with leaderboard results: evaluation conditions differ, and the hardened subsection does not explicitly establish original-869 coverage and exact official-configuration equivalence. The report’s reference to Astra’s methodology supports continuity without conclusively proving it ([Cyber system card](https://deploymentsafety.openai.com/gpt-6-cyber/gpt-6-cyber.pdf); [Astra methodology](https://deploymentsafety.openai.com/gpt-6-astra)). The 36.94% result is strong capability evidence, not an unconditional floor, and it should not be replaced with an inferred exact count.

Budget scaling supplies another part of the historical picture. The reported two-to-six-hour mitigations-off comparisons are shown below. The May observations use the older suite and remain separate from later frozen-suite results.

| Publication vintage | Model or experiment | Two hours | Six hours |
|---|---|---:|---:|
| May 2026 | Mythos Preview time curve | 127 | 204 |
| May 2026 | Opus 4.6 | Approximately 15 | 16 |
| 28 May 2026 | Opus 4.8 | 80 | 120 |
| 9 June 2026 | Mythos 5 | 181 | 247 |
| 24 July 2026 | Opus 5 | 171 | 191 |
| 1 September 2026 | Mythos 5.1 | 203 | 264 |
| 22 September 2026 | Opus 5.5 | 289 | 300 |
| 9 November 2026 | Mythos 5.2 | 331 | 341 |

The earlier curve and later leaderboard comparisons show heterogeneous returns to extra time ([benchmark account](https://rdi.berkeley.edu/blog/exploitgym/); [leaderboard](https://www.cybergym.io/exploitgym/)). The Mythos Preview curve’s two-hour count differs from the original paper’s separate 157-success headline; those experiments should not be spliced together. The Mythos 5.2 comparison remains less firmly verified and carries less evidentiary weight. The lesson is consistent across the better-established observations: some agents benefit greatly from longer runs, while others plateau.

Google’s November report strengthens that conclusion. Across all 869 tasks with mitigations off, Carbon scored 53.5% at six hours and 54.8% at ten hours; Argon scored 39.5% and 42.0%. These are one-attempt evaluations. Carbon’s stronger performance supports continued capability progress, but its modest late-runtime improvement argues against assuming that extra compute alone removes the hard residual ([Google safety report](https://storage.googleapis.com/deepmind-media/gemini/gemini_4_carbon_safety_report.pdf)). None of these scores qualifies as a defended result.

Agent scaffolding provides a distinct route to improvement. DoGNAVY’s selected 40-task ablation reported 30 successes for the complete system, 24 without stage decomposition, 24 without expert knowledge, 18 without parallel execution and 28 without memory. Its larger evaluations reported 79 successes on 225 tasks and 157 on 485 tasks. These findings support better search and coordination, but selection, budget differences and missing hardened controls prevent transferring a numerical uplift to the target ([DoGNAVY paper](https://arxiv.org/html/2611.04196v1)). Likewise, VARAS-OneMind’s 206 intended-vulnerability successes with DeepSeek-V4.1-Flash came from a 314-task mitigations-off subset, not a qualifying full-suite run ([developer report](https://varas-onemind.github.io/posts/exploitgym)).

Technical saturation is a real uncertainty, not a measured ceiling. A July account of correspondence with benchmark authors estimated 60–70% solvability even without mitigations, but that estimate was not an audited task census ([account](https://abstatisticalconsulting.substack.com/p/brief-notes-on-the-openaihugging)). OpenAI separately reported 198 of the older 898 tasks remaining unsolved across many internal attempts. The complement is not a qualifying single-system result: the cohort, settings and grading do not establish intended-vulnerability successes under the required defenses ([incident report](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf)). Together, these observations restrain confidence in near-saturation without establishing a hard cap.

## What's non-obvious
The obvious reading of the leaderboard is that mitigations still hold performance near the historical defended baseline. The newer hardened report changes that reading. At the same time, accepting its headline as an already-qualified record would go too far. **Capability evidence can materially raise expectations without establishing a resolution floor.** This distinction explains both the high central forecast and the broad downside uncertainty.

The other easily missed point is that the question measures a maximum under permissive resource rules. A capable named system can benefit from longer runs, parallel work and repeated attempts. But a maximum over systems is not a retrospective union of their successes, and repeated failures are not necessarily independent. The benchmark’s own hardened configuration also includes task-dependent behavior, including documented V8 fallback behavior; it is not a uniform test of every modern production defense ([defense documentation](https://github.com/sunblaze-ucb/exploitgym/blob/main/docs/defenses.md)).

## Uncertainties
The first gap is qualification. A task-level submission or an explicit developer statement confirming the frozen roster and exact mitigation configuration would resolve much of the uncertainty around the November hardened figures. The secondary coverage record also distinguishes reported successes from recorded defended observations and contains an unresolved coverage discrepancy. That warrants caution, not a smaller denominator or an automatic rejection of the question’s stated baseline ([Stanford AIMS](https://aims.stanford.edu/measurement-db/exploitgym)).

The second gap is technical headroom. There is no complete human-verified hardened exploitability census and no controlled full-suite repeated-attempt curve. Those measurements would separate tasks that need more search from tasks whose intended exploitation paths are blocked. Missing reference exploits and repeated failures do not, by themselves, prove impossibility ([original paper](https://arxiv.org/html/2605.11086v1)).

The third gap is disclosure. Developers may produce stronger systems but publish only subsets, mitigations-off scores or results on a successor suite. Public teaser claims do not close that gap without configuration and coverage evidence. Conversely, a well-documented, unusually effective long-budget system is the main route to the upper tail.

Finally, intended-vulnerability attribution can be revised. Capturing a flag is not enough, and the original evaluation distinguishes flag captures from valid intended exploits ([paper](https://arxiv.org/html/2605.11086v1)). The forecast therefore retains uncertainty about scores as well as progress. These risks do not change the saved forecast; they explain why its range remains broad.

## Forecast history

| Date | Forecast |
|---|---|
| 2026-11-13 | Median 56.2% success rate with mitigations on, 80% range 36.3% to 76.1% success rate with mitigations on |

## Sources Prolepsia read

- [ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?](https://www.cybergym.io/exploitgym)
- [exploitgym/docs/defenses.md at main · sunblaze-ucb/exploitgym · GitHub](https://github.com/sunblaze-ucb/exploitgym/blob/main/docs/defenses.md)
- [ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?](https://arxiv.org/html/2605.11086v1)
- [GPT-6 Cyber System Card](https://deploymentsafety.openai.com/gpt-6-cyber/gpt-6-cyber.pdf)
- [Center for Responsible, Decentralized Intelligence at Berkeley](https://rdi.berkeley.edu/blog/exploitgym)
- [Gemini 4 Carbon Safety & Security Report](https://storage.googleapis.com/deepmind-media/gemini/gemini_4_carbon_safety_report.pdf)
- [developer report](https://varas-onemind.github.io/posts/exploitgym)
- [Brief notes on the OpenAI/Hugging Face incident](https://abstatisticalconsulting.substack.com/p/brief-notes-on-the-openaihugging)
- [OpenAI – Hugging Face Incident Technical Report](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf)
- [ExploitGym · measurement-db](https://aims.stanford.edu/measurement-db/exploitgym)

## The question

ExploitGym is a benchmark from UC Berkeley and partners (https://www.cybergym.io/exploitgym/, paper https://arxiv.org/abs/2605.11086). Each of its 869 tasks gives an AI agent vulnerable source code, an input that triggers the bug, and a test environment. The agent succeeds when it writes a working exploit that uses the intended vulnerability to read a secret flag. The suite has 502 userspace tasks, 181 Chrome V8 tasks and 186 Linux kernel tasks. As of 13 November 2026 the official leaderboard shows: GPT-6 Astra 368 of 869 (42.35%, 3 September 2026), the highest result of any model; GPT-6.1 Sol 305; Claude Opus 5.5 300; Claude Mythos 5.1 264. The best full-suite open-weight result on the leaderboard is GLM-5.3 with Claude Code at 130 of 869 (15.0%, rescaled six-hour budget, 18 August 2026). Developers report 20.7% for Kimi-K3.1-Pro and 17.8% for MiMo-V2.6-Pro-RL, but their full-suite coverage is not established. The agent system VARAS-OneMind reports 206 successes with DeepSeek-V4.1-Flash, but on a 314-task subset. ExploitGym can run every task with standard defences switched on: address space layout randomisation (ASLR), stack canaries, the V8 heap sandbox and the kernel's usual protections. The leaderboard shows these results as a lower bar. As of 13 November 2026 the best is GPT-5.6 Sol with Codex CLI at 136 of 869 (15.65%), against 293 for the same run with mitigations off; GPT-5.5 scored 22 against 129, and GPT-5.4 scored 3 against 61. This question asks whether those defences keep holding back stronger models.

### How it resolves

The question resolves to the highest mitigations-enabled success rate of any qualifying result publicly disclosed on or before 30 June 2027 at 23:59 UTC, including results already public. A qualifying result must use ExploitGym's own mitigations-enabled configuration on all 869 tasks, as shown by the leaderboard's lower bar, the results repository, or a developer report that states it used that configuration on the full suite. Any model or single named agent system counts. The success rate is the number of tasks solved through the intended vulnerability, as ExploitGym's method counts them, divided by 869 and multiplied by 100. Only results on the original frozen 869-task suite count; results on a subset, on a different benchmark version, or with a changed denominator do not count, and unsolved tasks stay in the denominator. Where a source gives exact counts, the counts decide; a percentage-only result counts when the source presents it as an official aggregate for the full 869-task suite. Any time budget, number of attempts, tool use and agent scaffolding is allowed. Sources, in order: the official ExploitGym website and its public results repository (https://github.com/sunblaze-ucb/exploitgym-results); then original developer technical reports, model cards, system cards and papers that identify the model, the configuration and full-suite coverage. Results later shown to be fabricated, miscalculated or on a materially different benchmark are excluded.

### Fine print

Disclosure dates are in UTC. A result counts by the date it was first published, not the date a leaderboard later lists it. The benchmark stays frozen at the original 869-task suite even if ExploitGym adds tasks or publishes a successor. If no source can establish any qualifying result, the question resolves to the best qualifying result already public on 13 November 2026.

---

Prolepsia’s forecasting engine made this forecast: it researched the question, weighed the evidence and wrote the report on this page. A probability is not a promise; this one changes as the evidence does.
