Nine AI agents sent blind into 100 labs built from real bug bounties

Marius du Preez@mdp_sec

Most AI cybersecurity benchmarks answer the wrong question for bug-bounty hunters.

They give the model source code, a known CVE, a vulnerability category, or a tightly framed objective.

That is useful for measuring security knowledge and white-box reasoning. But it is not how web and API bug-bounty hunting starts.

A hunter gets a deployed target and a scope.

No repository. No answer key. No known vulnerable component. No guarantee that the first interesting response leads anywhere.

So I built the benchmark I wanted to see.

I took 100 vulnerabilities I had personally found and submitted across 71 real bug-bounty programs, reduced each one to its underlying security mechanism, and rebuilt it as an isolated synthetic web or API lab.

Each lab had real state, generated identities, and a four-rung proof chain. The target itself verified whether the agent performed the required operations. A confident explanation earned nothing without the exact target-issued proof.

Then I sent nine AI agents in blind.

Each model received the same broad target brief and shell access.

No source code. No vulnerability category. No answer key.

Pass one ran across all 100 labs. Pass two gave each miss one fresh attempt with the same information and a new seed. Each attempt had up to 80 turns and 25 minutes.

Across the final standalone runs, the benchmark consumed 5.14 billion priced tokens.

The results:

Opus 5: 63 solves / 303 rungs
Grok 4.6: 62 / 295
DeepSeek V4 Flash: 53 / 270
Qwen3.8 Flash: 53 / 270
Qwen3.8 27B: 51 / 252
DeepSeek V4 Pro: 48 / 258
Sol 5.6: 45 / 244
GLM-5.3 Flash: 31 / 190
Luna 5.6: 18 / 150

The cost differences were even more surprising.

Using one frozen OpenRouter base-rate sheet for every model:

Opus cost $569.08.
Grok cost $327.28.
DeepSeek Flash cost $36.04.
Qwen Flash cost $9.97.

Qwen Flash reached 84% of Opus's solve count for less than 2% of the normalized cost.

But cheap does not automatically mean better.

GLM had the lowest cost per solve at $0.11, but it solved only 31 labs. Luna cost $0.44 per solve and finished last at 18.

Cost efficiency means nothing if you ignore what the model failed to find.

Product tiers did not predict the ordering either.

DeepSeek V4 Pro solved 48 labs for $92.94.
DeepSeek V4 Flash solved 53 for $36.04.

Sol, running at high effort in its own vendor's harness, solved 45 for $106.71. Four cheaper models finished ahead of it.

The retry also changed the story.

Grok led pass one with 58 solves. Opus started at 52, then added 11 on the retry and finished first at 63.

Qwen Flash moved from 42 to 53 and erased DeepSeek Flash's four-solve first-pass lead.

Luna added 13 rungs on the retry but no new solve at all.

The lesson is not simply "run every model twice." Retry value is model-specific. It helps when failures are recoverable, not when the model is repeatedly hitting the same wall.

This is still a synthetic benchmark. It does not reproduce production rate limits, WAFs, duplicate risk, scope ambiguity, triage, or the legal judgment required on a live program.

But it directly tests the part most public cyber benchmarks miss:

Can the agent start from a black-box target, discover the relevant surface, build the required state, carry the vulnerability to impact, and prove it?

The full article includes the methodology, every model result, pass-one and retry performance, partial rung progress, timing, token volume, normalized cost, and all of the data visuals.

.

#BugBounty #CyberSecurity #TogetherWeHitHarder #ItTakesACrowd #Bugcrowd #Hackerone