Prolepsia forecastNumber

What will be the highest success rate on ExploitGym's full 869-task suite reported for any AI model or agent system by 30 June 2027?

As of 13 November 2026, Prolepsia’s median forecast is 70.3, with an 80% chance it falls between 57.7 and 88.3.

70.3median

Prolepsia’s forecast distributionHow likely each value is for: What will be the highest success rate on ExploitGym's full 869-task suite reported for any AI model or agent system by 30 June 2027?. Median: 70.3. 80% range: 57.7 to 88.3.
Prolepsia forecast made

How Prolepsia reasons

The report Prolepsia wrote with this forecast.

How forecasts are made · Track record

TL;DR

The forecast for the highest qualifying success rate on ExploitGym’s full 869-task suite by 30 June 2027 is a median of 70.3%, with a 10th percentile of 57.7% and a 90th percentile of 88.3%. The evidence supports continued improvement, not a plateau at GPT-6 Astra, but the remaining tasks could impose a substantial ceiling.

Key evidence

  • The starting point is higher than the leaderboard suggests. Google’s Gemini 4 Carbon report gives 54.8% at ten hours and 53.5% at six hours, explicitly covering 869 tasks with intended-vulnerability grading and mitigations disabled. The rules allow both longer budgets and percentage-only full-suite aggregates, so neither feature is grounds for exclusion (Google safety report).
  • The historical record shows sustained gains: Claude Mythos 5 solved 247 tasks in June, GPT-5.6 Sol solved 293 in July, and GPT-6 Astra solved 368 in September (official leaderboard). OpenAI’s subsequent GPT-6 Cyber report gives 48.45%, providing another sign of progress beyond Astra (system card).
  • More runtime helps, but gains diminish. Carbon’s six-to-ten-hour increase is only 1.3 percentage points (Google safety report). Better agent organization offers another route: DoGNAVY solved 30 of 40 matched tasks with its complete system, versus 18 without parallel execution. That small experiment does not establish a full-suite multiplier (paper).
  • Solvability is the strongest counterweight. A firsthand account of correspondence with the benchmark authors reports an informal estimate that 60–70% of tasks are solvable. This is not a verified ceiling (account).

Uncertainties

The central uncertainty is how many remaining tasks are genuinely exploitable, rather than merely difficult. The informal ceiling estimate pulls against the forecast’s upper range, but no task-by-task census establishes it as a hard bound.

A reported keynote preview of an upcoming OpenAI cyber model at 77.0% is an upside signal, not an accepted record: model identity, full-suite coverage and grading remain unverified (contemporaneous report). Future capability must also become a qualifying public disclosure before the deadline. Selected-subset results, successor benchmarks and unions of separately reported runs do not count.

Read the full report

TL;DR

The forecast for the highest qualifying ExploitGym success rate by 30 June 2027 is a median of 70.3%, with a 10th percentile of 57.7% and a 90th percentile of 88.3%. Google’s documented full-suite result of 54.8% has already surpassed the question’s GPT-6 Astra baseline (safety report). The central expectation is further substantial progress, with the attainable task ceiling as the main constraint.

Context

ExploitGym tests whether an agent can turn a known vulnerability into a working exploit. Each task supplies vulnerable source code, a bug-triggering input and a test environment. Success requires reading a secret flag through the intended vulnerability, not simply finding another way to capture it. The frozen suite contains 502 userspace tasks, 181 Chrome V8 tasks and 186 Linux kernel tasks (benchmark).

The question concerns the best qualifying single submission publicly disclosed by 30 June 2027 at 23:59 UTC. Any model or named agent system can count, with any time budget or number of attempts. But the submission must cover the original 869-task suite in the main, mitigations-off configuration. Separate runs cannot be combined after the fact. These rules make both technical progress and public documentation decisive.

Evidence

The historical backbone is a rising record on the frozen suite. There is no broad empirical base rate that establishes how quickly this particular benchmark must improve. Its own history is the most relevant evidence, but it mixes different models, budgets and agent designs. It shows progress, not a controlled learning curve.

The dated official results and subsequent developer disclosures give the following history. Not every row set a new record; retaining the lower results shows that newer releases do not automatically outperform the leader.

Date Model Reported full-suite result Source
28 May 2026 Claude Opus 4.8 120 successes Leaderboard
9 June 2026 Claude Mythos 5 247 successes Leaderboard
13 July 2026 GPT-5.6 Sol 293 successes Leaderboard
1 September 2026 Claude Mythos 5.1 264 successes Leaderboard
3 September 2026 GPT-6 Astra 368 successes; 42.35% Leaderboard
22 September 2026 Claude Opus 5.5 300 successes Leaderboard
29 September 2026 GPT-6.1 Sol 305 successes Leaderboard
28 October 2026 announcement Gemini 4 Carbon 54.8% Announcement
5 November 2026 GPT-6 Cyber 48.45% System card
9 November 2026 Claude Mythos 5.2 331 successes at two hours; 341 at six hours System card

The apparent plateau is partly a reporting lag. Google’s November safety report explicitly identifies the full denominator, intended-vulnerability grading and mitigations-off setting. It reports Carbon at 53.5% with a six-hour limit and 54.8% with a ten-hour limit. These are different budgets, not conflicting measurements of the same run. The higher aggregate meets the question’s stated reporting provisions; no unpublished numerator needs to be inferred (Google safety report).

The suite’s history also creates a comparability trap. The original paper studied 898 tasks. The June public release reduced that to 869 by filtering non-exploitable cases. Earlier paper aggregates therefore cannot simply be placed alongside frozen-suite results or used to extend the same trend backward (official changelog). The forecast concerns the frozen denominator even if a later benchmark version appears.

Runtime evidence supports continued gains, but not effortless saturation. Carbon’s improvement from six to ten hours is 1.3 percentage points. In the same report, Gemini 4 Argon scores 39.5% at six hours and 42.0% at ten hours. Both models benefit from more time, but neither comparison supports assuming that much longer runs solve nearly everything (Google safety report).

A separate full-suite AIOS-Cyber run provides a complete set of cumulative checkpoints: 152 intended successes at two hours, 200 at four hours and 218 at six hours. The successive increments shrink from 48 to 18. These are checkpoints within one run, not independent trials, but they show both the value of persistence and diminishing returns (evaluation documentation). The inference is that large further gains need better models or better systems, not just more runtime under an unchanged approach.

Agent design is a credible source of those gains. In DoGNAVY’s matched 40-task ablation, the complete system solved 30 tasks. Removing stages produced 24 successes; removing expertise also produced 24; removing parallel execution produced 18; removing memory produced 28. This supports a real role for organization and parallel work. The sample is small and selected, so applying the same improvement factor to the full frontier result would overstate the evidence (DoGNAVY paper).

Subset results need similar discipline. VARAS-OneMind reports 206 intended successes using DeepSeek-V4.1-Flash on 314 tasks. That demonstrates useful capability, but it does not establish performance over all 869 tasks. The omitted tasks cannot be assumed equally easy, and the subset’s headline success rate is not a qualifying full-suite record (VARAS report).

Scoring matters as much as coverage. The AIOS-Cyber full-suite submission reports 261 flag captures but only 218 intended-vulnerability successes, with 43 off-target captures. An alternate judging exercise produced 213 successes rather than 218. This is evidence of correction risk, not an additional capability gain (submission; judging documentation). A result can look stronger under raw flag capture while making no corresponding progress on the metric that resolves this question.

The strongest evidence against a high outcome concerns exploitability. The paper warns that a reproducible bug does not guarantee a usable exploit and that ground-truth exploits are unavailable for every task (benchmark paper). A July account of correspondence with the authors reports their informal estimate that 60–70% of tasks are solvable with mitigations disabled (firsthand account). That estimate is a serious constraint on the optimistic reading. It is not a demonstrated hard cap: no complete task-level solvability census establishes it.

There is also an upside signal beyond established results. A contemporaneous report describes an OpenAI keynote slide showing 77.0% for an upcoming cyber model. Its model identity, task population and grading are not established by an original methods statement. The same report gives a GPT-6 Cyber comparator that differs from its published system-card result, adding a comparability warning. The preview is evidence of a possible development pipeline, not a qualifying record or a floor (keynote coverage).

What's non-obvious

The obvious reading asks whether a model will finally beat Astra. The better reading starts from the fact that a documented developer result already has. The question accepts more than the official leaderboard, so leaderboard inactivity is not evidence of capability stagnation. Conversely, a dramatic preview or selected-subset score is not enough. Documentation can turn a real technical advance into an eligible result—or leave it outside the outcome entirely.

The other distinction is between difficult tasks and impossible tasks. More capable agents can overcome the former, but not the latter under a frozen environment and intended-vulnerability rule. A missing known exploit does not prove impossibility. Yet repeated improvements in model strength do not prove that the whole suite is exploitable either. The wide forecast range reflects this unresolved distinction, rather than assuming either a firm low ceiling or inevitable saturation.

Uncertainties

The largest gap is a task-level account of what remains solvable. A validated set of intended exploits, or rigorous explanations of why particular tasks cannot be exploited under the benchmark settings, would narrow the uncertainty more than another aggregate headline. The informal authors’ estimate weighs against the forecast’s upper range without replacing it with a hard bound.

The leading developer result is documented, but public task-level replication of Carbon is not established. Exact success logs, reproducible configurations and independent review would reduce the risk of grading or reporting corrections. Percentage-only reporting and a longer budget are allowed; the remaining concern is whether the reported aggregate survives validation.

Finally, technical capability and qualifying disclosure are separate events. A stronger system must publish a sufficiently identified full-suite result before the deadline. There is no verified release roadmap establishing when that will happen, and progress on a successor suite would not resolve this question. Those publication constraints, together with the unknown solvability ceiling and uncertain transfer of scaffold gains, remain the main reasons for the forecast’s breadth.

Forecast history

How Prolepsia’s forecast has movedOne forecast so far: 70.3 on 13 Nov 2026.
DateForecast
Median 70.3, 80% range 57.7 to 88.3

Sources Prolepsia read

  1. Gemini 4 Carbon Safety & Security Reportstorage.googleapis.com
  2. https://www.cybergym.io/exploitgym/www.cybergym.io
  3. Introducing Gemini 4 Carbonblog.google
  4. GPT-6 Cyber System Carddeploymentsafety.openai.com
  5. System cardwww.anthropic.com
  6. official changelograw.githubusercontent.com
  7. AIOS-Cyber/README.zh-CN.md at main · AIOS-Cyber/AIOS-Cyber · GitHubgithub.com
  8. DoGNAVY-Exploit: Harnessing Parallel Open-Weight Agents for Stage-Aware Vulnerability Exploitationarxiv.org
  9. VARAS-OneMind on ExploitGym: Capability-Led Exploitation at Scale | VARASvaras-onemind.github.io
  10. Brief notes on the OpenAI/Hugging Face incidentabstatisticalconsulting.substack.com

The question

ExploitGym is a benchmark from UC Berkeley and partners (https://www.cybergym.io/exploitgym/, paper https://arxiv.org/abs/2605.11086). Each of its 869 tasks gives an AI agent vulnerable source code, an input that triggers the bug, and a test environment. The agent succeeds when it writes a working exploit that uses the intended vulnerability to read a secret flag. The suite has 502 userspace tasks, 181 Chrome V8 tasks and 186 Linux kernel tasks. As of 13 November 2026 the official leaderboard shows: GPT-6 Astra 368 of 869 (42.35%, 3 September 2026), the highest result of any model; GPT-6.1 Sol 305; Claude Opus 5.5 300; Claude Mythos 5.1 264. The best full-suite open-weight result on the leaderboard is GLM-5.3 with Claude Code at 130 of 869 (15.0%, rescaled six-hour budget, 18 August 2026). Developers report 20.7% for Kimi-K3.1-Pro and 17.8% for MiMo-V2.6-Pro-RL, but their full-suite coverage is not established. The agent system VARAS-OneMind reports 206 successes with DeepSeek-V4.1-Flash, but on a 314-task subset. This question asks whether the best result keeps rising after GPT-6 Astra, which no model has beaten since September 2026.

How it resolves

The question resolves to the highest success rate of any qualifying result publicly disclosed on or before 30 June 2027 at 23:59 UTC, including GPT-6 Astra's 368 of 869. Any model counts, open or closed, and any single named agent system counts, including one that uses several models, as long as the result is reported as one submission. Combining the best tasks from separately reported runs does not count. A qualifying result uses the benchmark's main configuration, which is the main bar on the official leaderboard, not the configuration with exploit mitigations switched on. The success rate is the number of tasks solved through the intended vulnerability, as ExploitGym's method counts them, divided by 869 and multiplied by 100. Only results on the original frozen 869-task suite count; results on a subset, on a different benchmark version, or with a changed denominator do not count, and unsolved tasks stay in the denominator. Where a source gives exact counts, the counts decide; a percentage-only result counts when the source presents it as an official aggregate for the full 869-task suite. Any time budget, number of attempts, tool use and agent scaffolding is allowed. Sources, in order: the official ExploitGym website and its public results repository (https://github.com/sunblaze-ucb/exploitgym-results); then original developer technical reports, model cards, system cards and papers that identify the model, the configuration and full-suite coverage. Results later shown to be fabricated, miscalculated or on a materially different benchmark are excluded.

Fine print

Disclosure dates are in UTC. A result counts by the date it was first published, not the date a leaderboard later lists it. The benchmark stays frozen at the original 869-task suite even if ExploitGym adds tasks or publishes a successor. If no source can establish any qualifying result, the question resolves to the best qualifying result already public on 13 November 2026.