Prolepsia forecastDate

When will an open-weight model first reach at least 42.4% success on ExploitGym's 869-task exploit-development suite?

As of 5 November 2026, Prolepsia’s median date is 13 Jul 2027, with an 80% chance it falls between 27 Jan 2027 and 9 Nov 2028.

13 Jul 2027median date

Prolepsia’s forecast distributionHow likely each value is for: When will an open-weight model first reach at least 42.4% success on ExploitGym's 869-task exploit-development suite?. Median: 13 Jul 2027. 80% range: 27 Jan 2027 to 9 Nov 2028.
Prolepsia forecast made

How Prolepsia reasons

The report Prolepsia wrote with this forecast.

How forecasts are made · Track record

TL;DR

The forecast is: 10th percentile: Jan 27, 2027; Median (50th percentile): Jul 13, 2027; 90th percentile: Nov 09, 2028. The central expectation is a qualifying disclosure in mid-2027, with substantial room for either rapid gains from better agent systems or a longer delay caused by hard remaining tasks and release restrictions.

Key evidence

  • The target requires a complete evaluation. The released benchmark contains 869 tasks, and its website warns that leaderboard submissions can cover subsets. Under the question’s rules, exact-count reports need 369 successes; a percentage-only official aggregate of at least 42.4% can qualify under the stated exception. (ExploitGym)
  • The open-weight starting point is stronger than the supplied chart suggests. MiMo-V2.6 Pro reports 17.8%, while Kimi-K3.1-Pro reports 20.7% with a six-hour per-task budget. These are useful capability indicators, but the evaluated checkpoint identities and complete task coverage are not fully established. (MiMo model card; Kimi model card)
  • Post-training and additional inference produce real gains. On an explicitly full-suite comparison, GLM-5.2 records 29 and 39 successes at normalized two- and six-hour budgets; GLM-5.3 records 105 and 130. The models share a base model, making this evidence that progress need not wait for a new base-model generation. (GLM-5.3 model card)
  • Specialist systems show both promise and a measurement trap. VARAS reports 206 intended-vulnerability successes, but evaluates only 314 tasks. That is evidence of useful agent capability, not a qualifying full-suite result. (VARAS report)

Uncertainties

The largest unknown is performance on the original tasks omitted from strong subset evaluations. Those subsets are not established as representative, and task-level comparisons show substantial concentration in tasks already solved by other systems.

More attempts and longer budgets are allowed, but no verified full-suite open-weight retry curve establishes how much additional coverage they deliver. Failed attempts can remain correlated.

Eligibility also matters. The exact evaluated weights must be publicly downloadable, and Moonshot describes deliberate limits on released cyber capability. Internal capability therefore need not translate immediately into an eligible public result. (Kimi technical blog)

Read the full report

TL;DR

The forecast is: 10th percentile: Jan 27, 2027; Median (50th percentile): Jul 13, 2027; 90th percentile: Nov 09, 2028. Mid-2027 is the central expectation for the first qualifying public disclosure, not a deadline. The decisive requirement is a documented result on all 869 tasks from the exact publicly downloadable checkpoint, not a high success rate on a selected subset.

Context

ExploitGym measures whether agents can turn known vulnerabilities and proof-of-vulnerability inputs into working exploits. The frozen released suite contains 502 userspace tasks, 181 Chrome V8 tasks and 186 Linux-kernel tasks, for 869 in total. The earlier paper described 898 instances; the public-release changelog explains that filtering non-exploitable cases produced the smaller released suite. These are different benchmark populations, not interchangeable denominators. (ExploitGym; paper; changelog)

The forecast concerns the first verifiable public disclosure that meets the question’s rules. The exact evaluated inference weights must be publicly downloadable by the qualifying date. Larger inference budgets, repeated attempts and agent scaffolding are permitted, but substantive exploit generation by proprietary models is not. Exact-count reports require 369 successful tasks; a reported percentage of at least 42.4% can qualify without exact counts under the question’s explicit exception. The target stays fixed even as the wider model frontier advances.

Evidence

There is no reliable historical base rate for this exact event. The available history is short and mixes model releases, agent designs, evaluation budgets and task coverage. It supports a direction of progress, but not a precise monthly growth law. The supplied chart’s move from GLM-5.1 at 0.46% to MiMo at 17.8% therefore cannot be read as a clean time series. GLM-5.1’s underlying record is four intended-vulnerability successes, and its leaderboard date differs from its submission metadata date. (official leaderboard; GLM-5.1 metadata)

The clearest controlled historical comparison comes from the GLM reports. GLM-5.2 and GLM-5.3 share a base model, yet the later post-training produces much stronger exploit-development results. The report also supplies full-suite comparisons for Kimi K3 and Qwen3.8-Max. The complete set of paired observations is below; each cell is a successful-task count out of 869. (GLM-5.3 model card)

Model Normalized two-hour budget Normalized six-hour budget
GLM-5.2 29 39
GLM-5.3 105 130
Kimi K3 36 70
Qwen3.8-Max 14 26

These observations support two conclusions. Post-training can make a large difference without replacing the base model. More inference also helps, but its effect varies across models. The budgets normalize API inference time using model-specific token throughput, so they are not equal-cost or ordinary wall-clock comparisons. Even the strongest count in this comparison remains well below the required 369. (evaluation methodology)

Developer-reported model-family results show a broader rise in capability. Xiaomi reports ExploitGym scores of 0.2% for MiMo-V2.5 Pro, 6.0% for MiMo-V2.6 Flash and 17.8% for MiMo-V2.6 Pro. DeepSeek reports 1.8% for V4-Flash, 5.4% for V4-Pro and 15.3% for V4.1-Flash. These are family comparisons, not standardized observations taken at regular intervals. They support progress across several developers, but do not establish a common growth rate. (MiMo technical report; DeepSeek model card)

Kimi-K3.1-Pro reports a stronger 20.7% result using Kimi Code with six hours per task. Public weight files exist, making it a relevant candidate beyond the chart’s claimed MiMo lead. But the report does not establish a complete task manifest, exact success count and immutable mapping between the evaluated model and downloadable checkpoint. MiMo’s result has similar documentation gaps. These figures are provisional capability indicators, not fully audited full-suite records. (Kimi model card; Kimi weight files; MiMo model card)

Specialized agents provide the strongest reason not to treat ordinary model-card scores as a ceiling. VARAS-OneMind reports 159 intended-vulnerability successes on 264 evaluated tasks with GLM-5.3, and 206 on 314 with DeepSeek-V4.1-Flash. Both use six-hour budgets. These results demonstrate useful system-level capability, but neither evaluates the full suite. Holding the 206 successes fixed, the DeepSeek system would need another 163 successes among the 555 untested tasks to reach the exact-count threshold. (VARAS developer report)

The selection problem is more serious than an unknown denominator. A task-level comparison of the official result files found that all 309 distinct tasks solved by five audited baseline systems lay inside VARAS’s 314-task subset. Adding two other audited agent submissions produced a success-set union of exactly 314 tasks. This does not prove how the subset was chosen, but it shows that it is concentrated in tasks already known to be solvable. Its success rate cannot be transferred to the omitted tasks. (official results repository; VARAS task results)

DoGNAVY supplies a second line of scaffold evidence. It reports 79 successes on 225 selected tasks with GLM-5.2 and 157 on 485 with GLM-5.3. On a matched 40-task ablation, the complete system solves 30 tasks, compared with 24 without stage decomposition, 24 without expert knowledge, 18 without parallel execution and 28 without memory. This is direct evidence that agent design matters. It is not evidence that the same gain applies across the full suite. The paper’s eight-hour configuration and the leaderboard’s ten-hour label also prevent a clean standardized-budget interpretation. (DoGNAVY paper; submission metadata)

The fixed target is below some reported proprietary-model performance. Google reports 54.8% for Gemini 4 Carbon, although full-suite equivalence was not established in the supplied research. This is a signal that the capability level is attainable, not an eligible open-weight result. Meanwhile, Moonshot explicitly describes removing cyber-related training material and avoiding open-weight releases that extend the absolute cyber frontier. Capability growth and public weight release are separate gates. (Google report; Moonshot release policy)

What's non-obvious

The obvious reading is that open models must improve until their headline score reaches 42.4%. The less obvious point is that the question permits substantial improvements around the model: better planning, parallel exploration, retries and extended inference. A qualifying result need not wait for a new base-model generation. The controlled post-training comparison and scaffold ablations make this a credible route, rather than a purely speculative shortcut. (GLM comparison; DoGNAVY ablations)

The opposite mistake is to treat a selected-subset result above the threshold as evidence that success is nearly complete. The omitted tasks are the central capability uncertainty. The task-level overlap shows why evaluating the remaining tasks is not just paperwork. At the same time, release policy need not block the target forever: the question freezes the threshold, while Moonshot’s stated restriction concerns extending the moving absolute frontier. These forces explain both the mid-2027 central expectation and the substantial later tail. (official result files; Moonshot policy)

Uncertainties

The largest missing observation is a reproducible full-869 evaluation showing how open-weight performance changes with retries and larger budgets. Independent-attempt calculations would overstate gains when failures share the same underlying cause. A complete task-level retry curve would show whether extra compute unlocks new vulnerabilities or mostly repeats existing successes.

Checkpoint identity is another gap. Public weight repositories establish downloadability, but not always that the exact evaluated tensors are the released ones. Full task manifests, benchmark revisions, configuration details and immutable checkpoint identifiers would close that gap. Reporting inconsistencies in some agent results also make exact counts and original records more useful than headline percentages. (Kimi model card; MiMo model card; DoGNAVY README)

Finally, achievement and disclosure need not coincide. A capable checkpoint can remain private, and a successful evaluation can remain unpublished. No qualifying open-weight result was established in the reviewed evidence; that is not exhaustive proof that none existed. The forecast therefore remains about the first verifiable eligible public report, with no artificial deadline imposed by the chart’s August projection.

Forecast history

How Prolepsia’s forecast has movedOne forecast so far: 13 Jul 2027 on 5 Nov 2026.
DateForecast
Median 13 Jul 2027, 80% range 27 Jan 2027 to 9 Nov 2028

Sources Prolepsia read

  1. ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?www.cybergym.io
  2. https://arxiv.org/abs/2605.11086arxiv.org
  3. changeloggithub.com
  4. https://huggingface.co/zai-org/GLM-5.3huggingface.co
  5. VARAS developer reportraw.githubusercontent.com
  6. https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/blog.google
  7. Kimi K3.1 Pro Tech Blog: Advancing Open Intelligencewww.kimi.ai
  8. https://www.kimi.com/en/blog/kimi-k3-1-prowww.kimi.com
  9. https://mimo.mi.com/docs/zh-CN/news/latest/v2-6mimo.mi.com

The question

As of November 5, 2026, this question asks for the first calendar date on which a publicly released open-weight AI model is reported to achieve a success rate of at least 42.4% on the original, fixed 869-task ExploitGym exploit-development benchmark. ExploitGym, introduced in May 2026, evaluates whether AI agents can transform known software vulnerabilities and proof-of-vulnerability inputs into working exploits. Its 869 tasks comprise 502 userspace vulnerabilities, 181 Chrome V8 vulnerabilities, and 186 Linux-kernel vulnerabilities. The benchmark's official website and research paper describe its methodology and evaluation criteria. Sources: https://www.cybergym.io/exploitgym/ and https://arxiv.org/abs/2605.11086. GPT-6 Astra established the reference threshold of 42.4%, reported in September 2026. This is approximately 368 successful exploits out of 869 tasks, as documented in OpenAI's GPT-6 Astra reporting and the ExploitGym leaderboard. The question specifically concerns open-weight models rather than the overall model frontier. The supplied forecasting chart identifies MiMo-V2.6-Pro-RL as the leading open-weight model at 17.8% as of November 2026, although this particular score and its evaluation configuration have not been independently established from primary sources. The chart also cites GLM-5.1 at 0.46% in June 2026 and projects that open-weight models could reach the 42.4% threshold approximately nine months after November 2026, around August 2027. These figures provide forecasting context rather than binding resolution evidence. The target is the first qualifying publicly reported benchmark result, not the date a model was trained, internally evaluated, or initially made available. A model must have its weights publicly released no later than the date of its qualifying report. The threshold remains fixed at 42.4% regardless of subsequent improvements in proprietary or open-weight models. The question covers models developed anywhere in the world and has no fixed resolution deadline.

How it resolves

The question resolves to the earliest calendar date, in UTC, on which a qualifying evaluation result is first publicly disclosed for an eligible open-weight AI model. An eligible model must have its trained inference weights publicly available for download by the general public, without individualized approval, at or before the qualifying disclosure. Source code alone, an API endpoint, a hosted demonstration, or a promise to release weights does not qualify. Licenses restricting commercial use or redistribution do not automatically disqualify a model if its actual inference weights are publicly downloadable. A qualifying result must report successful exploit development on the original 869-task ExploitGym suite, with an aggregate success rate of at least 42.4%. For reports specifying exact successful-task counts, at least 369 of the 869 tasks must be successfully solved, because 368/869 is approximately 42.35% and the threshold is interpreted as an unrounded percentage of at least 42.4%. However, an explicitly reported score of 42.4% or higher will qualify when the source reports percentages without exact counts, provided the score is presented as an official aggregate result for the 869-task suite. The evaluation must cover all 869 tasks, and tasks not successfully solved must remain in the denominator. Results based on a subset of tasks, a different benchmark version, or a changed task denominator do not qualify. The success metric must measure successful exploit development under ExploitGym's published evaluation methodology, rather than vulnerability identification, partial exploitation, or another cybersecurity benchmark. The evaluation must be attributable to a specific eligible open-weight model, although the model may use an agent framework, tools, scaffolding, or extended inference budgets. Different agent configurations and computational budgets are permitted, consistent with the original question's emphasis on best reported performance rather than standardized inference costs. Results from independently evaluated fine-tuned versions of open-weight models qualify if the particular version being evaluated also has publicly released weights. Ensembles or systems relying on proprietary language models for substantive exploit generation do not qualify. The primary resolution source will be the official ExploitGym website and its associated public evaluation-results repository. Additional acceptable sources include original developer technical reports, model cards, research papers, and independently published benchmark evaluations that provide enough methodological detail to establish the model's identity, eligibility, benchmark coverage, and reported success rate. Secondary leaderboards may be used to discover results but should be checked against original reports whenever possible. The resolution date is the earliest verifiable public publication date of a qualifying result, provided the evaluated weights were already publicly available. A later leaderboard update recording an earlier public qualifying result does not replace the original publication date. If a report predates the release of the evaluated weights, the qualifying date is the later of the public result announcement and the public weight release. If multiple results qualify, the earliest qualifying date wins. Results subsequently demonstrated to be fabricated, incorrectly calculated, or based on a materially different benchmark must be excluded. If no qualifying result has appeared, the question remains unresolved rather than resolving to a negative outcome. If the original benchmark becomes unavailable or stops being maintained, historical or independently reproducible evaluations of the original 869-task suite remain eligible. If the evidence needed to establish any qualifying result becomes permanently unavailable, the question may be annulled rather than assigned an invented date.

Fine print

The benchmark is frozen to the original 869-task ExploitGym suite referenced by the question, even if ExploitGym later adds tasks, changes its leaderboard, or publishes a successor benchmark. Open-weight status applies to the exact evaluated model checkpoint, not merely to its model family. Public release of a smaller or less capable checkpoint does not make a larger private checkpoint eligible. A previously closed-weight model becomes eligible only when the weights corresponding to its qualifying result are publicly released. Inference-time compute scaling, multiple attempts, extended reasoning, tool use, and agent scaffolding are allowed, but the reported success rate must be attributable to a clearly identified evaluation configuration. The question concerns the best reported result for an individual qualifying model and its associated agent system, not the combined performance of several independently selected models. The 42.4% threshold is fixed and is not adjusted upward as frontier models improve. Where reports give success counts and percentages that disagree because of rounding or presentation conventions, exact task counts take precedence, except when the report's explicitly defined metric uses a different documented aggregation method that is demonstrably equivalent to the original full-suite success rate. Dates are recorded in UTC. If an original publication is revised, the earliest publicly verifiable version containing a qualifying result determines the date, provided the result remains valid. The forecast concerns a publicly reported achievement, not the potentially earlier, unobservable date on which a model first acquired the underlying capability. There is no assumption that the chart's projected August 2027 milestone will occur.

Analytics cookies

Prolepsia counts visits anonymously unless you accept analytics cookies, which link your visits together.