Skip to content
Get started

What Socair checks

Tier 1 is static and offline. It reads an artifact’s bytes and metadata and runs no inference, so it checks the file, not how the model behaves.

The four results

Each check is a row in a report, and every row comes back one of four ways.

PASS
The check inspected what it names and found no indicator within its stated scope. Each row states exactly what its PASS means, and where it stops.
FAIL
Positive evidence, which the row names. A FAIL is never a guess. It withholds promotion, and no acceptance clears it.
LEAD
A suspicious but inconclusive signal that needs a person's review outside Socair. It is not a finding of malice, and not a gap: no acceptance clears it.
NOT_TESTED
Socair did not look, and the row names why. It is never a pass: it withholds promotion until the input it needs is supplied and the row passes, or a named person signs an acceptance of that gap.

The two fixed statements

A Tier 1 report carries one of two fixed sentences, chosen from its rows; a report whose sentence does not match its rows does not validate. A report that carries Tier 2 measurements takes one of a second fixed pair instead, which says where Tier 2 ran the model: all four are in the template.

When no row is FAIL or LEAD
For the artifact identified by its hash, the Tier 1 checks that returned PASS found no indicators within their stated scope. Rows marked NOT_TESTED were not examined, for the reason each row gives. Tier 1 runs no inference, so no model behavior was tested. Every surface outside that scope is listed under Out of scope.
When any row is FAIL or LEAD
For the artifact identified by its hash, the Tier 1 checks found the indicators this report lists: each FAIL is positive evidence, and each LEAD is a suspicious signal that needs review. Rows marked NOT_TESTED were not examined, for the reason each row gives. Tier 1 runs no inference, so no model behavior was tested. Every surface outside that scope is listed under Out of scope.

The Tier 1 checks

Each row is in the words a report uses.

On every report

  • Format and structure

    Malformed container structure, unexpected tensors

    PASS means: The container parsed, and its tensor data tiles the data section exactly: no unaccounted bytes, overlapping tensors, or duplicate names. For an ONNX model: every tensor holds exactly the data its type and shape need, no two initializers of a graph share a name, external data stays inside the model's directory and fits its file, and every operator is ONNX's own, one of onnxruntime's contrib operators, or a function the model defines. For an OpenVINO IR: every Const lies inside the .bin and is exactly the size its element type and shape take, the Consts cover the .bin with no byte outside them, and every operation OpenVINO builds is in its own opsets. Tensor values are not judged, and an operator library a deployment registers is outside the model. For a Keras model (HDF5 or a .keras archive): every dataset's storage lies inside the HDF5 data, sized by its type and shape, with none overlapping and no data or link in another file, the archive holds only what Keras reads, and the config names no Python bytecode, pickled module, or class or function from outside Keras and TensorFlow (KerasHub, KerasCV, and KerasNLP classes are named). Weight values are not judged.

  • File inventory and payloads

    Hidden files, embedded payloads, unexpected executables

    PASS means: No script, shell command, long encoded blob, or executable signature was found in string metadata values, including every string-array element of 64 bytes or more, and no native executable among the files. Shorter array elements, such as vocabulary tokens, are too short to carry one and are not scanned. An archive among the files counts only when the Pickle opcode scan PASSed it and read every entry in it: a torch.save checkpoint with nothing outside torch.save's layout.

  • Hash, provenance, lineage

    Traceable origin: a manifest bound to this hash, from an immutable upstream commit; files that disagree with the uploader's checksum list

    PASS means: A trusted publisher signature verified over this artifact, or an operator-supplied manifest binds this hash to a repository at an immutable commit. The notes say which; a manifest is the operator's claim, not a signature. No file the scan compared with a SHA-256 list its uploader ships (SHA256SUMS or a .sha256 file) disagrees with it, save Markdown, which is named and not graded; agreement is noted, and the PASS does not rest on it.

By format

  • Chat template

    GGUF, model directory, when it carries a chat template

    Code reach, hidden or obfuscated text, override or content-triggered instructions, and the text the rendered template adds to the prompt

    PASS means: The template parsed and reaches no Python internals or process execution, and carries no override or concealment language, URL, hidden or obfuscated text, or condition on message content other than a standard content-part type such as text or image. Instructions written as ordinary guidance, or added for a content-part type, are not judged: the notes list the text a template adds for a content-part type, and, when the template rendered, the text it adds to the prompt and whether it matches a reviewed template.

  • Tokenizer config

    GGUF, model directory

    Tampered tokenizer tables: special-token ids, token types, control tokens

    PASS means: The tokenizer tables are internally consistent, and no control token carries instructions. When a canonical reference table for its family is configured (from a feed or SOCAIR_TOKENIZER_REFERENCE), every token was compared with it and no ordinary token differs; the notes say which reference, or that none was configured or matched.

  • Quant match

    GGUF

    Declared quantization (file name) against the tensor types in the file

    PASS means: At least one tensor has the base type the declared quantization requires. This checks labeling, not safety.

  • Remote code

    Model directory

    Python a loader would import: modules a config names (auto_map, custom_pipelines, sentence-transformers and diffusers modules), files loaders import by their path, and the repository files those import

    PASS means: No config a loader reads names Python in this directory or in another repository (through an auto_map entry, custom_pipelines, a sentence-transformers module, or a diffusers custom pipeline or component), and no file sits where a loader imports one by its path (custom_generate/generate.py, or an inference toolkit's handler.py, pipeline.py, or code/inference.py). Other Python and script files are listed in the notes and not judged: a loader Socair does not model, or a person, can still run them.

  • Pickle opcode scan

    Pickle checkpoint

    Imports in pickle-based model files that reach code execution, and pickles that build more than tensors and plain containers

    PASS means: Every pickle conforms to grammar socair-wo/1: protocol 2 or 3 as CPython writes it, building only tensors and plain containers through calls whose arguments were checked, and every storage it references matches a record of exactly its size. A NumPy array with no object dtype holds no pickle, and its data matches its shape. Not covered: a loader bug, a reader that parses the bytes differently, code carried as data, or a tampered Python environment.

With an input you supply

Each of these rows runs only when you supply what it checks against: your known-bad list, a policy, or the reviewed model’s tensors. Without it, a report has no such row and nothing is withheld for it; the report names Known-bad hash match under Out of scope as not run, and the license is still named in its identity section.

  • Known-bad hash match

    With your known-bad list (SOCAIR_DENYLIST, or your organization’s signed feed)

    Match against the known-bad artifact denylist

    PASS means: This exact hash is not on the configured denylist. Any changed byte gives a new hash, so it catches only known files.

  • License policy

    With your license policy (SOCAIR_LICENSE_POLICY)

    The license the artifact states, against the operator's allowed list (SOCAIR_LICENSE_POLICY)

    PASS means: The license the artifact states (model card, LICENSE file, or GGUF metadata) is on your allowed list. This is not legal advice: usage-policy obligations, such as an acceptable-use policy or a user cap, are listed in the notes and not marked met.

  • Intake policy

    With your intake policy (SOCAIR_INTAKE_POLICY)

    The artifact against your intake policy (SOCAIR_INTAKE_POLICY): allowed formats, publisher allow and deny lists, and publishers whose models need a verified signature

    PASS means: Every rule your intake policy sets holds: each model file's format, identified from its bytes, is allowed; the publisher, read from a provenance manifest bound to this hash, is on your allowed list and not on your denied list; and a publisher whose models need a signature has one that verified against a key or root you trust. A rule the policy does not set is not checked. It is your organization's rule, not a finding about the model's safety.

  • Tensor reference

    With a reviewed model’s tensor manifest (SOCAIR_TENSOR_REFERENCE)

    Weights that are not, tensor for tensor, the reference model's

    PASS means: Every tensor is the reference model's tensor of the same name, by its bytes or by exact values in another float dtype, and neither side has a tensor the other lacks: the weights are the weights of the model you named. It says nothing about whether that model is safe, and a model compared with its own manifest always matches.

A report where Tier 2 did not run lists the two Tier 2 checks, Forward-pass trigger probes (Tier 2) and Serving-stack differential (Tier 2), under Out of scope as not run, as it lists Known-bad hash match without your list and the Chat template row for a model that carries no template. Tier 2 is opt-in: its foundation is in the release, with no real Tier 2 check yet, and its checks are planned. What no row can see is published as the ceiling. How the checks fare against defanged reproductions of published attacks, misses included, is in the detection benchmark. The attestation, section by section, is in the template, and which rows are evidence for which framework controls (MITRE ATLAS, OWASP LLM, NIST, AI supply-chain guidance) is in the control mapping.

Being built in the open

Not in any release today. Each item is planned: an open issue in the repository, where the design and the limits are worked out in public.

Tier 1 and trust

More formats and static checks, to shrink the ceiling and close the gaps a report still leaves NOT_TESTED.

Tier 2: opt-in checks that run the model on your own GPUs

Tier 1 runs no inference. Tier 2 runs the model through your own serving engine, on your own hardware, and records what it measured. Its foundation is in the release, opt-in: the report section and the protocol for a probe helper, with no real Tier 2 check yet. These are the checks it is for. A measurement can raise a LEAD, or a FAIL on deterministic evidence, and can never authorize a model.