Skip to content
Get started

Blog

Model intake for on-prem AI: signed, offline, and honest about what it can't see

Christopher Maher,

Last month, Pluto Security reported a malware campaign on Hugging Face: 31 malicious repositories across seven accounts, published in two waves since September, using models that look like small Qwen2 test checkpoints.

There were no poisoned weights and no pickle exploit. In Pluto's words, trust_remote_code=True "imports the repository's Python with the caller's privileges before any weights load." The most-downloaded repository's model.safetensors held no tensors and was 42 bytes. Its Python installs a crypto miner disguised as a kernel worker, and some builds also open a reverse shell. With linked models, the repositories passed 32,000 downloads, which Pluto counts as downloads, not confirmed infections.

Hugging Face and ModelScope removed the first wave's accounts after Pluto reported them, and a second wave appeared within days under new accounts. The Hub does scan uploads: Pluto notes that it runs every file through antivirus and that partner scanners inspect weight formats, "but nothing analyzes what the Python does." And: "The second-wave flagship scanned clean while it collected 28,412 downloads."

Find the code a loader would run

I've been building Socair since late September: an open-source intake gate for open-weight models headed for on-prem and air-gapped GPUs. It doesn't try to decide whether a repository's code is malicious. It finds the Python a loader would import from a model repository, names every file, and holds the model. That covers:

  • modules a config names: an auto_map in any JSON config, custom_pipelines, sentence-transformers modules, and diffusers' model_index.json;
  • files imported by their path: custom_generate/generate.py for transformers, and handler.py, pipeline.py, and code/inference.py at the root, which the Hugging Face and SageMaker inference toolkits import without trust_remote_code;
  • every repository file those import, transitively.

Each one is a LEAD: a named, unresolved signal. Any other Python in the repository is listed in the report, not judged.

It also reads two config fields that make transformers load code when the caller didn't enable trust_remote_code: an attention field naming a Hub kernel (CVE-2026-4372), and trust_remote_code written into a LightGlue config (CVE-2026-5241).

A LEAD withholds promotion, and no acceptance clears it. Socair has no override, and its airlock never promotes a withheld model. The report says what happens next: "it needs a person's review outside Socair."

The campaign, defanged, in the benchmark

Socair's detection benchmark carries six cases built from the public write-ups. They are defanged: their payloads only build a string.

  • The flagship's shape: a Qwen2-looking config whose auto_map names a module that runs its payload at import, beside a 42-byte model.safetensors with no tensors.
  • The same repository with no weight file.
  • The payload in modeling_qwen2.py beside configuration_qwen2.py, named after the library's own Qwen2 modules, as one first-wave build did.
  • The payload moved into a configuration module, and into a tokenizer module.
  • A native executable shipped as a .bin data file, for the loader to start.

The five repositories are Remote code LEADs, and the executable is a File inventory FAIL, so every case is withheld. The benchmark's test pins each outcome.

The rest of the checks

Remote code is one row of a set of checks:

  • Format and structure for GGUF, safetensors, ONNX, OpenVINO IR, and Keras: malformed containers, and tensor data that doesn't tile the file exactly, which is where payloads hide.
  • Pickle checkpoints, held to a typed grammar: a PASS means the pickle builds only tensors and plain containers.
  • Chat templates, parsed and rendered by Socair's own bounded evaluator. A branch that tests a message's content and adds a system turn or its own instructions is a LEAD. In a GGUF repository, a quant whose template renders like another's on every standard probe but adds such a branch is a LEAD too.
  • Tokenizer configs, publisher signatures (OpenSSF Model Signing), and provenance bound to the artifact's hash at an immutable commit.

More than a scanner

  • The output is a signed attestation, not a log: an in-toto statement in a DSSE envelope, signed with your own key, which anyone holding your public key can verify offline.
  • It enforces. The airlock promotes a model into its clean store only with an attestation from a key you trust, for those exact bytes, and never a withheld one.
    • socair airlock serve-hub serves the clean store over the Hugging Face Hub API. An inference server pointed at it with HF_ENDPOINT downloads the promoted bytes of the models the store admits, and nothing else the store holds.
    • On Kubernetes, for models packaged as OCI artifacts, Kyverno policies refuse a pod whose model lacks an authorized binding signed with your key.
  • It says what it can't see. Every report carries the detection ceiling and the sentence "This attestation does not certify the absence of unknown backdoors." A check that couldn't test something says NOT_TESTED, with the reason, and that gap withholds promotion until someone signs an acceptance of it. A check whose input wasn't supplied is listed as not run, with the reason. The static checks read the weights' layout, never their behavior.
  • Opt-in Tier 2 runs the model on your own GPUs, through the engine that will serve it. Trigger confirmation runs each trigger a flagged chat template tests, with and without the text the template's branch adds. Refusal regression compares the model's refusals with those of the base model its card declares. A measurement can withhold promotion, and it can never authorize it.

Try it

It's Apache-2.0, every check. It's written in Go with vendored dependencies, so it builds and runs air-gapped. Install it with Homebrew, the Helm chart, or a release binary, and check each with gh attestation verify.

You can also preview a public Hugging Face model in your browser. Socair's engine runs there on the model's headers and small files, read straight from Hugging Face. Nothing is uploaded, and it's a preview, not a signed report.

The code is at github.com/defilantech/socair.

I'll be at KubeCon + CloudNativeCon North America in Salt Lake City, November 9 to 12. If you run inference on your own GPUs and have opinions about how models should get there, I'd like to hear them: hello@socair.ai.

Sources

All posts