Prolepsia forecastNumber
What will be the highest success rate on ExploitGym's full 869-task suite reported for an open-weight model by 30 June 2027?
As of 13 November 2026, Prolepsia’s median forecast is 49.5, with an 80% chance it falls between 26.8 and 70.9.
49.5median
How Prolepsia reasons
The report Prolepsia wrote with this forecast.
How forecasts are made · Track record
TL;DR
The forecast for the highest qualifying open-weight success rate by 30 June 2027 is a median of 49.55%, a 10th percentile of 26.8% and a 90th percentile of 70.9%. The outlook supports substantial improvement, but the outcome depends on public checkpoint releases and valid full-suite disclosures—not capability alone.
Key evidence
- The starting point is stronger than the older leaderboard suggests. AIOS-Cyber’s 6 November 2026 submission reports 218 intended-vulnerability successes across all 869 tasks, or 25.0863%, with mitigations disabled. Public disclosure, rather than acceptance onto the leaderboard, determines timing under this question’s rules (submission). The named DeepSeek-V4.1-Flash-1017 checkpoint has publicly available weights (release announcement). This is a provisional qualifying baseline, not an independently reproduced result.
- Post-training and additional execution time have delivered gains. GLM-5.2 recorded 29 and 39 successes at two- and six-hour budgets; GLM-5.3 recorded 105 and 130 under the corresponding rescaled budgets (model card). AIOS reports 152, 200 and 218 cumulative successes by two, four and six hours, showing both further gains and diminishing returns (developer report).
- Specialist agents offer upside, but headline subset rates overstate what is established. VARAS reports 206 successes on 314 tasks with DeepSeek, not a qualifying full-suite result (report). Google’s reported 54.8% for closed-weight Gemini 4 Carbon is a broader capability signal, not an eligible open-weight score (announcement).
Uncertainties
The largest uncertainty is how much progress transfers to the difficult tasks omitted from specialist evaluations. More attempts can help, but repeated failures can share the same underlying cause. The benchmark paper also acknowledges incomplete ground-truth exploit coverage and potentially non-exploitable tasks (paper).
Release policy is a separate constraint. Moonshot describes reducing cyber capability before releasing public weights, so private improvements need not become eligible results (technical blog).
Finally, intended-vulnerability judging can revise reported totals. AIOS’s published judge-agreement figures imply 213 positives under the alternative judge, versus the reported 218; agreement does not establish correctness (report). These uncertainties explain the wide forecast range without changing the saved forecast.
Read the full report
TL;DR
The forecast for the highest qualifying open-weight ExploitGym success rate by 30 June 2027 is a median of 49.55%, with a 10th percentile of 26.8% and a 90th percentile of 70.9%. Use the median as the central planning case and the percentile endpoints to represent the wide range of outcomes. The decisive distinction is between strong exploit capability and a publicly documented result that satisfies the full-suite and exact-weight requirements.
Context
ExploitGym tests whether an agent can turn a known vulnerability into a working exploit that reads a secret flag through the intended vulnerability. The frozen suite contains 869 tasks: 502 userspace, 181 Chrome V8 and 186 Linux kernel tasks (benchmark website). The question concerns the highest eligible result disclosed by 30 June 2027 at 23:59 UTC, not an average model’s performance or a standardized evaluation budget. Unlimited attempts, time and agent scaffolding are allowed.
The official leaderboard’s older open-weight reference is GLM-5.3 with Claude Code at 130 successes, dated 18 August 2026 (leaderboard). A newer public submission changes the starting point: AIOS-Cyber reports 218 intended-vulnerability successes across the complete suite (submission). That report supports a provisional qualifying baseline, while leaving validation and checkpoint-provenance questions open.
Evidence
The historical backbone is short and uneven. There is no long, standardized series from which to infer a dependable rate of progress. The clearest open-weight comparison comes from the same developer’s full-suite evaluations: GLM-5.2 solved 29 tasks at the two-hour budget and 39 at six hours; GLM-5.3 solved 105 and 130 at the corresponding budgets. The developer attributes the improvement to post-training on the same base model. These budgets are throughput-rescaled, not ordinary equal wall-clock comparisons (GLM-5.3 model card).
This comparison supports two conclusions. Open-weight exploit performance can improve substantially without a new base model. Additional inference effort also raises the score. It does not establish that the same improvement repeats with every release, or that execution time has a constant return. The forecast expects substantial further progress without treating this short history as an exponential law.
The broader proprietary-frontier history also points upward. The reported sequence is 129 successes out of 869 on 16 June, 293 on 13 July and 368 on 3 September 2026 (leaderboard), followed by Google’s 54.8% announcement for Gemini 4 Carbon on 28 October 2026 (announcement). These are different models and evaluation settings, not a controlled time series. They show that the older proprietary leaderboard record is not a fixed technical ceiling. They do not establish how quickly open-weight systems will catch up.
The strongest current open-weight evidence is AIOS-Cyber’s 6 November 2026 submission. It reports 119 successes among 502 userspace tasks, 63 among 181 V8 tasks and 36 among 186 kernel tasks, totaling 218 of 869, or 25.0863%. The submission specifies disabled mitigations and a six-hour budget (submission). The relevant starting point is therefore stronger than the older 130-success leaderboard entry, subject to the newer report surviving validation.
Eligibility has meaningful support. The submission names DeepSeek-V4.1-Flash-1017 as the exploit-working model, and the exact named checkpoint was released on 17 October with public weights (release announcement). The proprietary model in the submission acts as a post-hoc judge, rather than an exploit worker (submission metadata). Under the question’s rules, an open pull request is not disqualified merely because it has not been accepted: first public disclosure determines timing. None of this substitutes for independent experimental validation.
AIOS also provides a useful execution-time history. Its same-run cumulative intended-vulnerability successes were 152 by two hours, 200 by four hours and 218 over the six-hour-budget run. Flag captures at those checkpoints were 176, 237 and 261. The extra time therefore produced real intended-path gains, not just additional flag captures, but the increments declined from 48 successes to 18 (developer report). These are checkpoints within one run, not independent retry experiments.
Reported inference expenditure for that evaluation was $3,528.18, or approximately $4.06 per task, with extensive caching. Those figures exclude infrastructure, engineering and audit costs (developer report). I read this as evidence that further full-suite experimentation is financially plausible. It is not evidence that much larger budgets will solve all remaining tasks.
Specialized agents supply another route to improvement. VARAS reports 159 intended successes on a 264-task subset with GLM-5.3 and 206 on a 314-task subset with DeepSeek-V4.1-Flash (VARAS report). DoGNAVY reports 79 successes on 225 tasks with GLM-5.2 and 157 on 485 tasks with GLM-5.3; security experts selected its subsets (paper). These are evidence of useful agent engineering, but none qualifies as a full-suite result.
The more controlled scaffold evidence comes from DoGNAVY’s selected 40-task ablation. The complete system solved 30 tasks, compared with 24 without stage decomposition, 24 without expert knowledge, 18 without parallel execution and 28 without memory (paper). This supports the claim that agent design matters beyond the underlying model. The small, selected sample does not establish a full-suite uplift of the same size.
Other advertised open-weight results are weaker anchors. Kimi-K3.1-Pro reports 20.7%, while MiMo-V2.6-Pro-RL reports 17.8%; their documentation does not establish original full-suite coverage sufficiently to treat either as a separate qualifying baseline (Kimi model card; MiMo model card). Public weights and a benchmark percentage are not enough when the evaluated population or exact checkpoint remains unclear.
The strongest counterweights are release policy and task solvability. Moonshot describes removing cyber-related training material before public release and increasing released cyber capability gradually (technical blog). Separately, correspondence reported by Alexander Barry gives an author estimate that 60–70% of standard-configuration tasks are solvable (account). That estimate is not a task-by-task audit. The benchmark paper acknowledges that ground-truth exploits are unavailable for every task (paper). This is a material uncertainty for the upper end of the forecast, not a verified cap that replaces it.
What's non-obvious
Public disclosure matters more than leaderboard placement. The newer AIOS submission can affect this question before its pull request is accepted. Conversely, an impressive percentage on a leaderboard can be irrelevant if it comes from a subset. Dividing subset successes by 869 does not turn an incomplete evaluation into a qualifying full-suite run. The useful distinction is between a disclosed eligible aggregate and evidence that an agent has promising capabilities.
The target is also a best-result record, not ordinary model performance. New weights are only one source of progress. Better orchestration, parallel search, additional attempts and broader evaluation can all improve the eventual maximum. But those gains cannot simply be added together: systems can solve the same easy tasks, and omitted tasks can be systematically harder. The evidence supports substantial upside from engineering while rejecting direct extrapolation of attractive subset rates.
Uncertainties
The first gap is validation. AIOS reports 261 flag captures, including 43 judged off-target. Its judges agreed on 242 captures and disagreed on 19; the reported agreement counts imply 213 alternative-judge positives, compared with the primary total of 218 (report). Neither agreement nor disagreement establishes ground truth. Independent replay and intended-vulnerability review would tighten the provisional baseline.
The second gap is transfer to hard tasks. There is no complete evidence showing how specialist subset performance carries over to every omitted task, or how repeated attempts scale once failures become correlated. Full-suite task-level results, matched-budget comparisons and repeated-attempt curves would distinguish durable capability gains from selection and resource effects.
The third gap is eligibility and publication. Stronger private checkpoints do not count unless the exact exploit-working weights become publicly downloadable by the deadline. A fine-tuned derivative needs its own public weights. Complete model-role inventories and immutable checkpoint identifiers would strengthen provenance. Missing documentation is uncertainty, not proof of ineligibility.
Finally, benchmark identity must remain fixed. The original manuscript describes 898 tasks, while the released suite relevant here contains 869; the release changelog records filtering of non-exploitable cases (paper; changelog). Future results must preserve the question’s frozen population and main configuration. The wide forecast range reflects these unresolved constraints alongside the demonstrated room for progress.
Forecast history
| Date | Forecast |
|---|---|
| Median 49.5, 80% range 26.8 to 70.9 |
Sources Prolepsia read
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?www.cybergym.io
- Add AIOS-Cyber ExploitGym evaluation results(DeepSeek-V4.1-Flash-1017, no-mitigation) by AIOS-Cyber · Pull Request #7 · sunblaze-ucb/exploitgym-results · GitHubgithub.com
- zai-org/GLM-5.3 · Hugging Facehuggingface.co
- Introducing Gemini 4 Carbonblog.google
- DeepSeek-V4.1-Flash-1017: Wish-Driven Programming Made Real | DeepSeek API Docsapi-docs.deepseek.com
- DoGNAVY-Exploit: Harnessing Parallel Open-Weight Agents for Stage-Aware Vulnerability Exploitationarxiv.org
- Kimi K3.1 Pro Tech Blog: Advancing Open Intelligencewww.kimi.ai
- Brief notes on the OpenAI/Hugging Face incidentabstatisticalconsulting.substack.com
- reportraw.githubusercontent.com
- VARAS-OneMind on ExploitGym: Capability-Led Exploitation at Scale | VARASvaras-onemind.github.io
The question
ExploitGym is a benchmark from UC Berkeley and partners (https://www.cybergym.io/exploitgym/, paper https://arxiv.org/abs/2605.11086). Each of its 869 tasks gives an AI agent vulnerable source code, an input that triggers the bug, and a test environment. The agent succeeds when it writes a working exploit that uses the intended vulnerability to read a secret flag. The suite has 502 userspace tasks, 181 Chrome V8 tasks and 186 Linux kernel tasks. As of 13 November 2026 the official leaderboard shows: GPT-6 Astra 368 of 869 (42.35%, 3 September 2026), the highest result of any model; GPT-6.1 Sol 305; Claude Opus 5.5 300; Claude Mythos 5.1 264. The best full-suite open-weight result on the leaderboard is GLM-5.3 with Claude Code at 130 of 869 (15.0%, rescaled six-hour budget, 18 August 2026). Developers report 20.7% for Kimi-K3.1-Pro and 17.8% for MiMo-V2.6-Pro-RL, but their full-suite coverage is not established. The agent system VARAS-OneMind reports 206 successes with DeepSeek-V4.1-Flash, but on a 314-task subset. This question asks how far open-weight models will have climbed by mid-2027. A companion public question asks for the date an open-weight model first reaches 42.4%.
How it resolves
The question resolves to the highest success rate of any qualifying open-weight result publicly disclosed on or before 30 June 2027 at 23:59 UTC, including results already public. A qualifying result uses the benchmark's main configuration, which is the main bar on the official leaderboard, not the configuration with exploit mitigations switched on. The exact evaluated model checkpoint must have its inference weights downloadable by the general public, without individual approval, on or before 30 June 2027; a smaller released checkpoint does not make a larger private one eligible. Licences that restrict commercial use do not disqualify a model whose weights are publicly downloadable. Fine-tuned versions count if their own weights are public. Agent systems count only when every model doing substantive exploit work in them is an eligible open-weight model. The success rate is the number of tasks solved through the intended vulnerability, as ExploitGym's method counts them, divided by 869 and multiplied by 100. Only results on the original frozen 869-task suite count; results on a subset, on a different benchmark version, or with a changed denominator do not count, and unsolved tasks stay in the denominator. Where a source gives exact counts, the counts decide; a percentage-only result counts when the source presents it as an official aggregate for the full 869-task suite. Any time budget, number of attempts, tool use and agent scaffolding is allowed. Sources, in order: the official ExploitGym website and its public results repository (https://github.com/sunblaze-ucb/exploitgym-results); then original developer technical reports, model cards, system cards and papers that identify the model, the configuration and full-suite coverage. Results later shown to be fabricated, miscalculated or on a materially different benchmark are excluded.
Fine print
Disclosure dates are in UTC. A result counts by the date it was first published, not the date a leaderboard later lists it. The benchmark stays frozen at the original 869-task suite even if ExploitGym adds tasks or publishes a successor. If no source can establish any qualifying result, the question resolves to the best qualifying result already public on 13 November 2026.
More public forecasts
What will be the highest success rate on ExploitGym's full 869-task suite reported for any AI model or agent system by 30 June 2027?
70.3median
Prolepsia forecast
When will an open-weight model first reach at least 42.4% success on ExploitGym's 869-task exploit-development suite?
12 Aug 2027median date
Prolepsia forecast
What will be the highest success rate on ExploitGym's full 869-task suite with exploit mitigations switched on, reported by 30 June 2027?
56.2median
Prolepsia forecast
On 31 December 2027, will the highest Epoch Capabilities Index score of any open-weight model be at least as high as the highest score of any model on 1 January 2027?
76%chance of yes
Prolepsia forecast
Will a binding US federal measure restricting the public release of AI model weights be in effect before 1 January 2028?
29%chance of yes
Prolepsia forecast
Will Eli Lilly's reported revenue for full-year 2027 be at least 25% higher than its revenue for full-year 2026?
15%chance of yes
Prolepsia forecast