Detection benchmark

What Socair catches, measured on reproductions of published attacks, with the misses pinned and other scanners run over the same files.

Socair's numbers come from internal/benchmark, which runs in go test ./... in CI. Each case's outcome is pinned, so a regression fails the build, and so does an improvement until the case is updated. The other scanners' numbers come from one manual run of scripts/benchmark-compare.py over the same generated corpus, on 2026-10-04; CI does not re-run them.

Read this first

• The corpus is ours. Every case reproduces a published technique (each is cited in internal/benchmark/corpus.go). But Socair's authors chose the cases knowing what Socair checks, so the corpus leans toward Socair's coverage: some cases, such as a remapped tokenizer, need a reference only Socair supports. Read the results as a reproducible floor and a regression gate, not as an independent evaluation. New cases are welcome, especially ones Socair misses.
• Four cases need an input only Socair takes. gguf-tokenizer-remapped is scanned with a canonical tokenizer table (SOCAIR_TOKENIZER_REFERENCE), gguf-template-reviewed-near-miss with the reviewed template it departs from (SOCAIR_TEMPLATE_REFERENCE), dir-known-bad-hash with a denylist that names the file (SOCAIR_DENYLIST), and dir-weights-edited with the reviewed model's tensor manifest (SOCAIR_TENSOR_REFERENCE). No other scanner here was given any of them. With Socair's defaults (none of the four), the remapped tokenizer and the near-miss template pass, the known-bad file is matched against no list (with none supplied, that check does not run), and the edited weights have no row that compares them, so Socair detects 63 of 73. Scores with the cases that need an input left out follow the comparison table.
• The attacks are defanged. Each case keeps the attack's structure (the gadget, the opcodes, the archive trick, the template construct) and swaps its payload for echo socair-benchmark, or, for a template that injects an instruction, a benign instruction in its place. Every scanner here, Socair included, reads structure and never loads the file, so a defanged reproduction is detected as the original would be.
• Live samples are not here yet. These exist only as real malicious files and need an isolated machine:
  • in-the-wild malicious uploads;
  • the 7z-wrapped nullifAI samples;
  • the ShadowPickle corpus (arXiv 2607.17503).
• Few benign controls. There are 17 controls. False positives on real models are measured separately, on real corpora, in false-positive-baseline.md.

Socair (check set tier1/0.11, 2026-10-15)

Benign controls flagged (FAIL or LEAD on any row): 0 of 17.

Four of these detections depend on a reference input (see "Read this first"); with Socair's defaults it detects 63 of 73.

Check set tier1/0.8 replaced the pickle row's import allowlist with a typed grammar (socair-wo/1, see check-set.md) and added three pickle cases. Under tier1/0.7, pickle-ordereddict-code-string PASSed (a miss: the allowlist accepted collections.OrderedDict with any argument), pickle-nested-loader was a LEAD (an unreviewed global), and pickle-memo-reput already FAILed (the import scan modelled the memo as CPython does). Every earlier case kept its outcome and severity. The control-torch-zip control now holds a tensor and its storage record: it used to hold a plain OrderedDict beside a data/0 record nothing referenced, which the grammar reports as bytes torch.load never reads.

A model directory now finds a pickle by its bytes as well as its name (#186, also tier1/0.8): dir-pickle-renamed (a gadget named notes.txt) and dir-torch-zip-renamed (a torch zip named weights.dat) were never opened before, and are detected now. dir-pickle-protocol0-renamed is pinned as a miss: a protocol 0 pickle is ordinary text with no header, so in a directory it is found by its extension or not at all. control-dir-torchscript pins that a TorchScript archive (rust_model.ot, as sentence-transformers ships) stays a named, unscanned archive and does not become a LEAD.

Check set tier1/0.9 adds the control-torch-zip-untyped control (#233): a torch.save checkpoint holding a uint16 and a float8_e4m3fn tensor beside a float32 one. torch.save writes a tensor of a dtype in torch.storage._new_dtypes() with _rebuild_tensor_v3 over an untyped storage, and names that class torch.storage.UntypedStorage; the grammar knew only torch.UntypedStorage, which torch.save does not write, so the control was a LEAD before this change. No attack case changed outcome.

Check set tier1/0.9 also adds gguf-template-tuple-trigger (#246): the conditional backdoor with its trigger phrases in a tuple, m['content'] in ('Summarize the invoice', 'Total the invoice'). The analysis took a list of constants as a content condition's literal and a tuple as not, so this was a miss (PASS) before; a tuple is now a sequence like a list, and the case is a LEAD. No other case changed outcome. Check set tier1/0.9 also narrows the Remote code row to the Python a loader would import (#254): modules a config names, files at a path a loader imports by convention, and the repository files they import. It adds the control-dir-training-script control, a directory with the train_script.py that sentence-transformers' most-downloaded repositories ship, which was a Remote code LEAD before this change. It adds seven cases the row detects: dir-remote-code-imported (a payload reached only through a relative import of the module auto_map names), dir-remote-code-other-repo and dir-remote-code-outside-path (auto_map naming code in another repository, or by an absolute path), dir-st-custom-module (a sentence-transformers module in modules.json, with no auto_map), dir-custom-generate (transformers' custom_generate/generate.py), dir-inference-handler (an Inference Endpoints handler.py), and dir-diffusers-component-shadow (a standard-looking model_index.json entry whose unet/diffusers.py diffusers loads in place of the library). Before this change each was a LEAD as well, through its auto_map or its .py file. And it pins dir-script-caller-named as a miss: a custom_interface.py that only the caller's code names (SpeechBrain's foreign_class takes the file name as an argument) is listed in the row's notes and not judged, where it was a LEAD. No earlier case changed outcome.

Check set tier1/0.10 adds the Tensor reference row (#262), which compares a model's tensors with the reviewed model's tensor manifest (SOCAIR_TENSOR_REFERENCE, tensors.md). It adds dir-weights-edited, PoisonGPT's shape (Mithril Security, 2023: GPT-J-6B edited with ROME and uploaded under a lookalike of EleutherAI's name, within 0.1% of the original on a benchmark): a model directory whose first weight has one bit flipped, scanned with the unedited model's manifest. No other row can see an edit like this, since every byte is still a well-formed weight. It also adds the control control-dir-tensor-reference, the same directory unedited, which PASSes. No earlier case changed outcome.

Check set tier1/0.10 also reads ONNX models (#264) and adds four cases and two controls. onnx-external-data-traversal points an initializer's external data at ../../../../etc/hosts, the directory traversal of CVE-2022-25882 and its bypass CVE-2024-27318; onnx-external-data-memory uses onnxruntime's in-memory marker */_ORT_MEM_ADDR_/* as a location, which onnxruntime 1.20.0 reads as a memory address; onnx-initializer-size gives an initializer 4096 bytes of raw_data where its dims need 64; and onnx-metadata-script puts a shell script in metadata_props. The first three are Format and structure FAILs, the fourth a File inventory FAIL. Before this change an ONNX file was an unknown format, NOT_TESTED on every row that reads one. The control-onnx control is a clean ONNX model, and control-dir-onnx a directory with an onnx/ export as sentence-transformers ships one: external data beside the model, and an onnxruntime contrib operator (com.microsoft::FastGelu), as optimum's O2 to O4 exports have. No earlier case changed outcome.

Check set tier1/0.10 also compares an artifact with its uploader's own SHA-256 lists (#263, provenance-bundle.md). dir-checksum-swapped is a directory whose model.safetensors was replaced, by a well-formed and benign file, after its SHA256SUMS was written: the provenance row is a LEAD naming the file, where it was NOT_TESTED (no manifest), so the directory was withheld only as a gap an acceptance could clear. The control-dir-checksums control carries a SHA256SUMS that matches every file, which is noted and flags nothing. No earlier case changed outcome.

Check set tier1/0.10 also reads OpenVINO IR models (#268, openvino.md) and adds five cases and two controls. openvino-const-size gives a Const a size of 4096 bytes where its element type and shape take 16 (a Format and structure FAIL), and openvino-rt-info-script puts a shell script in the IR's rt_info (a File inventory FAIL). Two are withheld as gaps by design, the benchmark's first: openvino-const-past-bin points a Const past the end of its .bin, which OpenVINO refuses and a truncated download also produces, and openvino-bin-tail appends an ELF executable to the .bin past every Const, bytes OpenVINO never reads; both are Format and structure NOT_TESTED, naming the range, so the model is withheld and an acceptor who accepts the gap accepts those bytes unread. openvino-bin-pickle names a protocol 0 pickle gadget pytorch_model.bin and writes pytorch_model.xml beside it, an IR whose u8 Const covers the file: the IR reads as well formed, and the .bin still reaches the Pickle opcode scan, which FAILs it, because a loader reads that file by its name as a torch checkpoint. Before this change an IR .xml was an unknown format, not read in a directory, and its .bin reached the Pickle opcode scan by its extension (NOT_TESTED, "not a pickle opcode stream"). So the two single-file cases were NOT_TESTED on their rows, the two directory cases' Format and structure rows PASSed on the directory's safetensors (misses, withheld only by the Pickle opcode scan's gap), and the fifth was already a FAIL. The control-openvino control is a clean IR .xml on its own, and control-dir-openvino a directory with an openvino/ export as sentence-transformers ships one: an f32 IR and an int8 one whose Consts share a range, as OpenVINO's writer gives Consts of identical bytes. No earlier case changed outcome.

Check set tier1/0.10 also reads Keras models (#269, keras.md): an HDF5 .h5 and a .keras archive. It adds seven cases and three controls. keras-lambda-bytecode is CVE-2024-3660's Lambda layer in a full-model .h5, its function marshalled bytecode that only prints the payload (kerastest.Bytecode; no test loads or runs it); keras-safe-mode-off is CVE-2025-9906's config, a Lambda that calls keras.config.enable_unsafe_deserialization ahead of a Lambda of bytecode; keras-config-module names subprocess.run as a layer's class, the shape of CVE-2025-1550; keras-storage-past-eof is a weights dataset whose contiguous storage runs past the end of the HDF5 data, the contiguous form of the shape bombs of CVE-2026-0897 and CVE-2026-12570; keras-external-storage is CVE-2026-1669's dataset whose data is /etc/passwd; and keras-metadata-script puts a shell script in an archive's metadata.json. Each is a FAIL, the last on File inventory and the rest on Format and structure. keras-custom-object names a layer class from a package outside Keras: the code is the caller's, not the file's, so the row is NOT_TESTED, a gap an acceptance can clear, and not a detection. The controls are a weights-only tf_model.h5 as transformers' TensorFlow save_pretrained writes it, a .keras archive of Keras' own layers, and a directory with a tf_model.h5 beside its safetensors weights. Before this change a Keras file was an unknown format, NOT_TESTED on every row that reads one, and in a directory a weight file named unparsed. No earlier case changed outcome.

Check set tier1/0.11 changes no case's outcome. It stops a GGUF's special-token id of -1 (0xFFFFFFFF, the "no token" sentinel llama.cpp skips) from failing the Tokenizer row as an id outside the vocabulary; every case and control here writes its special-token ids inside the vocabulary.

There are three misses. The first is the published known miss: a default system prompt that steers answers as ordinary guidance, with no override phrase, URL, obfuscation, or condition on the user's message. It is on the detection ceiling. Since tier1/0.8 the row quotes that prompt as text the template adds, for a reviewer to read, but does not judge it.

The second is dir-pickle-protocol0-renamed, a protocol 0 pickle under a name that is not a pickle's. It is stated in check-set.md and in every directory report that has no pickle, whose File inventory row says no file is named as a pickle or begins as one "(a protocol 2 to 5 stream, or a zip holding a pickle)". It is not a ceiling bullet: something must still load the file by name. Standard loaders do not, and a repository whose own loader code a config names, or sits where a loader imports it by its path, is a Remote code LEAD; a loader already in the serving stack, or a script a person runs, is outside what a scan of the artifact can see.

The third is dir-script-caller-named, repository Python that no config names and no loader imports by its path: here SpeechBrain's foreign_class, which imports the file its caller names. Since #254 the Remote code row lists such a file in its notes and does not judge it, because Socair cannot tell whether anything runs it, and its PASS statement says so. Before, every .py file was a LEAD, which also withheld every repository that ships a training or evaluation script.

Other scanners on the same files (2026-10-04)

Versions:

• picklescan 1.0.5
• ModelScan 0.8.8 (under Python 3.12; it does not install on 3.13+)
• ModelAudit 0.2.37 (promptfoo)
• Fickling 0.1.12 (Trail of Bits)

Each ran with default settings, and its verdict was read as its documentation describes. This comparison predates dir-executable-named-script, added with check set tier1/0.7, and the five cases added with tier1/0.8 (the rendered-template cases gguf-template-macro-conditional and gguf-template-reviewed-near-miss, and the pickle cases pickle-ordereddict-code-string, pickle-nested-loader, and pickle-memo-reput), and the four added with #186 (dir-pickle-renamed, dir-torch-zip-renamed, dir-pickle-protocol0-renamed, and the control-dir-torchscript control), the control-torch-zip-untyped control added with #233, gguf-template-tuple-trigger added with #246, and the nine added with #254 (the seven Remote code cases, dir-script-caller-named, and the control-dir-training-script control), and the two added with #262 (dir-weights-edited and the control-dir-tensor-reference control), the six ONNX cases and controls added with #264, the two added with #263 (dir-checksum-swapped and the control-dir-checksums control), the seven OpenVINO cases and controls added with #268, and the ten Keras cases and controls added with #269, so its counts cover the cases before them; the other scanners have not been run on those forty-eight. A suspicious result counts as a finding, as Socair's LEAD does: picklescan's suspicious globals and Fickling's SUSPICIOUS. "Not scanned" means the tool skipped the file, could not parse it, or does not support the format.

How to read it:

• Pickle is the common ground. All five tools scan pickles, and the main differences are:
  • ModelScan routes by file extension. It read the plain PyTorch zip checkpoint (pytorch_model.bin) and found its gadget, but could not scan the raw pickles named .bin or .ckpt, or the .tar checkpoint. The harness counts any ModelScan run without a finding whose output mentions a skipped file as could not scan, even when it scanned other files in the same artifact; that is how both altered zip checkpoints are counted. It also missed the gadgets outside its denylist: pip, asyncio, importlib, numpy.load, and an unknown module.
  • picklescan missed the legacy tar checkpoint, and flagged both benign pickles as suspicious. Both controls hold a Python set (__builtin__.set), which a typical state dict does not, so that flag may come from the control rather than from anything a real checkpoint carries.
  • ModelAudit missed the gadget placed in an archive entry not named data.pkl.
  • Fickling could not read the zip checkpoints, and flagged the benign pickle.
• On GGUF, ModelAudit found the appended tensor bytes but none of the chat-template attacks. That covers the four SSTI reproductions (including CVE-2024-34359's) and the conditional and hidden-instruction backdoors. Picklescan and ModelScan do not read GGUF.
• Fickling's GGUF and safetensors "findings" are not detections. Fickling is a pickle analyzer. It reports any file that is not a pickle as "invalid opcodes, likely unsafe", the benign GGUF included.
• Model directories: ModelAudit read the directories and found the pickle weights and the template SSTI; picklescan and ModelScan found the pickle weights. Only Socair flagged the remote code, the native executable, and the tokenizer normalizer; the known-bad hash case needs a denylist, which only Socair was given.

With the four cases that need a Socair-only input left out (gguf-tokenizer-remapped, gguf-template-reviewed-near-miss, dir-known-bad-hash, and dir-weights-edited), Socair detects 63 of 69 attacks with check set tier1/0.10 (34 of 35 in the 2026-10-04 run, which predates the near-miss case), and the two formats they belong to read, from that run:

Per case

Reproduce

SOCAIR_BENCHMARK_OUT=results.md SOCAIR_BENCHMARK_CORPUS=/tmp/corpus \
  go test ./internal/benchmark -count=1 -v
scripts/benchmark-compare.py /tmp/corpus > compare.json   # scanners on PATH, or --picklescan PATH etc.

The corpus is generated and defanged, so it is safe to write anywhere.