Built with world-class reverse engineer Michał “Redford” Kowalczyk, this open-source benchmark has sparked excitement among security experts, opening a new frontier in binary analysis.
Quesma, Inc. announced BinaryAudit, the independent benchmark testing whether AI can find hidden threats in software binaries before they cause damage. The results show both promise and limitations: while AI can detect some threats, even the best-performing model, Claude Opus 4.6, succeeded only 49% of the time and frequently flagged safe software as dangerous.
AI binary analysis could be a new layer of defence in supply-chain security. BinaryAudit helps to track and encourage progress in this field.
Supply-chain attacks are already causing real-world damage. State-sponsored actors recently hijacked Notepad++, replacing legitimate binaries with infected ones. Shai Hulud 2.0 compromised thousands of organizations, including Fortune 500 companies and governments, stealing credentials. In the XZ Utils case, a long-term contributor legitimately gained ownership access using it to insert malicious code. Security weaknesses can also originate from vendors, including manufacturer-planted code to disable trains and hardcoded credentials in Cisco devices. These public cases are only a fraction of what exists.
Read More: SalesTechStar Interview with Matt Price, CEO of Crescendo
Traditional binary reverse engineering is a last-resort method. It’s performed by a small pool of specialists, typically only after a breach or major incident. AI has the potential to transform this reactive approach into a proactive layer of defense, making it feasible to inspect software at any point – before deployment, during updates, before the purchase, or years after release. This could change how organizations approach supply-chain security, turning what was once an emergency response tool into a preventive safeguard.
Read More: Salestech for Network-Led Growth: Turning Internal Relationships into Pipeline
“We were genuinely surprised that LLMs can detect malicious code at all. At current performance levels, it’s an assistant, not a solution,” said Jacek Migdał, CEO of Quesma. “AI binary analysis could be a new layer of defence in supply-chain security. We hope new AI models released in the next 1-2 years will make binary analysis go mainstream. BinaryAudit helps to track and encourage progress in this field.”
