GLM 5.2 Beats Claude on Cyber Benchmarks

While U.S. policymakers spend their energy trying to gatekeep frontier models like Anthropic's Mythos and OpenAI's GPT-5.6 behind strict licensing and export controls, the open-weight ecosystem has quietly moved the goalposts.

Zhipu AI recently released GLM 5.2, a 753-billion-parameter Mixture-of-Experts (MoE) model, under an MIT license. Almost immediately, security researchers began putting it through its paces. The most striking evaluation comes from static analysis firm Semgrep, which ran the model against its internal Insecure Direct Object Reference (IDOR) benchmark.

Running in a basic, un-scaffolded prompt harness, GLM 5.2 scored a 39% F1 on IDOR detection. That score did not just beat Claude Code running Claude Opus 4.8 (which scored 28%); it did so at roughly $0.17 per vulnerability found, which is about one-sixth the cost of running Anthropic's frontier model. For teams building security tooling, automated code-review pipelines, or internal static analysis agents, this is empirical data that justifies a serious look at open-weight alternatives.

xychart-beta
    title "IDOR Detection F1 Scores by Model & Harness"
    x-axis ["Semgrep (GPT 5.5)", "Semgrep (Opus 4.8)", "GLM 5.2 (Pydantic AI)", "Claude Code (Opus 4.6)", "Claude Code (Opus 4.8)"]
    y-axis "F1 Score (%)" 0 --> 70
    bar [61, 53, 39, 37, 28]

The Benchmark Breakdown

An IDOR vulnerability is a common access-control flaw where an application exposes a direct reference to an internal database object (like a user ID or order number) without validating that the requesting user has authorization to access it. Spotting them is notoriously difficult for traditional static application security testing (SAST) tools because it requires semantic understanding of the application's business logic, routing, and authorization framework across multiple files.

To test how raw models handle this, Semgrep ran several LLMs through a basic Pydantic AI harness. They provided the models with the codebase, a search strategy, and a description of what IDORs look like, but no advanced scaffolding like endpoint discovery or guided navigation.

The results revealed a stark performance gap:

• GLM 5.2 achieved a 39% F1 score.
• Claude Code (Opus 4.6) followed at 37%.
• Claude Code (Opus 4.8) dropped significantly to 28%.

While Semgrep's proprietary, heavily engineered multimodal pipeline still leads the pack (scoring up to 61% F1 using GPT 5.5), that system relies on custom pre-processing, endpoint enumeration, and code filtering. When comparing raw, prompt-only performance, GLM 5.2 took the crown from the Western frontier models.

The "Refusal Tax" and the Guardrail Problem

One of the most telling details in the benchmark data is the performance degradation of Claude Opus from version 4.6 to 4.8. In a standard coding task, newer iterations typically show incremental improvements. In a security auditing task, however, Opus 4.8's performance plummeted by 9 percentage points.

This is the "refusal tax" in action. As commercial AI labs face intense pressure to prevent their models from being used for malicious purposes, they continuously tighten safety guardrails. Unfortunately, the prompt patterns used to find vulnerabilities in a codebase look highly similar to the prompt patterns used to write exploits.

When a model is over-aligned, it defaults to safety-related refusals or over-cautious reasoning, rendering it far less useful for defensive security engineering. Because GLM 5.2 is an open-weight model released under an MIT license, developers can run it without these restrictive, cloud-enforced safety filters. It does not refuse to analyze code for flaws, making it a more reliable workhorse for automated auditing.

The Developer Angle: Architecture and Hardware Realities

For developers looking to integrate GLM 5.2 into their CI/CD pipelines or security workflows, there are some major architectural trade-offs to consider.

First, GLM 5.2 is massive. It features 753 billion total parameters, though its MoE architecture means only about 40 billion are active per token. This active parameter count keeps inference speeds high and API costs low, but hosting the model locally is a massive undertaking. Running a 753B model at reasonable quantization requires serious enterprise hardware, typically an 8x RTX 6000 Ada Generation setup costing upwards of $80,000.

For most development teams, running the model locally is cost-prohibitive. The pragmatic path is using API endpoints from providers like OpenRouter or Hugging Face, which offer access to the weights without the hardware overhead. However, if your organization handles highly sensitive, proprietary codebases where data privacy is non-negotiable, the ability to run GLM 5.2 entirely on-premises (even with the steep hardware cost) is an option that gated APIs like Claude or OpenAI simply cannot provide.

Second, GLM 5.2 boasts a 1-million-token context window. Zhipu AI designed this window specifically to maintain reliability across long, complex agent trajectories. In security auditing, this means you can feed entire repositories, routing tables, and dependency trees into the context window at once, allowing the model to trace data flow across the entire application architecture.

Harnessing the Model

What this benchmark ultimately proves is that while raw model capability is rising, the real ROI still lies in the engineering around the model. Semgrep's custom pipeline outperforming the raw models by over 20 points shows that scaffolding (endpoint discovery, AST parsing, and targeted context feeding) is still necessary for production-grade security tools.

But for developers building those custom harnesses, the choice of the underlying engine is no longer limited to expensive, heavily restricted APIs. GLM 5.2 provides a highly capable, cost-effective, and unaligned alternative that can be integrated directly into private infrastructure. If you are building security tooling, it is time to add an open-weight model to your evaluation stack.