It provides a standardized, quantitative evaluation of AI agents' ability to recover obfuscated indicators, with ground truth and scoring; includes results from 7 models across 12 tasks.
A static reverse-engineering benchmark. Each task gives an agent one malware sample and a shell, and asks it to recover the indicators of compromise a human analyst would extract - C2 endpoints, file paths, registry keys, mutexes, config IDs - including values hidden behind encoding, encryption, or obfuscation.
12 tasks, 121 ground-truth indicators. The agent writes its findings to /workspace/iocs.json; tests/score.py matches them against tests/ground_truth.json and scores recall. environment/samples/ carries each sample's SHA-256.
runs/ holds 252 runs (7 models × 3 passes × 12 tasks), laid out as runs////. Four runs have no iocs.json — the agent never wrote the file, so there was nothing to score and the run rewards 0.0: BlackByte_Go/qwen3.7-max/pass1, CrystalX_RAT/qwen3.7-max/pass2, SHELBYC2/qwen3.7-max/pass2, Embargo_Encryptor_Rust/qwen3.7-max/pass3.
Read the full article: http://arimlabs.ai/writing/malware-reverse-engineering
Code is Apache-2.0; task data, ground truth, and trajectories are CC-BY-4.0
Similar projects matched by category, topics, and programming language.
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.
