Nullgap.dev Docs

Pull request review

Supply-chain checks on pull requests

What happens when a pull request changes a manifest or lockfile, what is reported, and how package rules interact with the severity gate.

When a pull request adds a package or moves one to a new version, the review checks that package: is it known to be malicious, does it look suspicious, and do your organisation's package rules allow it.

This page covers the pull request side. For how verdicts are reached and what the signals mean, see malicious and suspicious packages. For the rules, see package policy.

#When the check runs

The check is part of every review. It has something to do when the pull request adds or modifies one of these files:

EcosystemFiles
npm (npm, pnpm, Yarn)package.json, package-lock.json, npm-shrinkwrap.json, pnpm-lock.yaml, yarn.lock
PyPI (pip, Poetry, uv, Pipenv)requirements*.txt, pyproject.toml, poetry.lock, uv.lock, Pipfile, Pipfile.lock
crates.io (Cargo)Cargo.toml, Cargo.lock

Other ecosystems are not checked on pull requests.

#Which packages are checked

Only what the change brings:

  • a lockfile entry counts when one of its lines was added or modified by the pull request;
  • a manifest line that names a new dependency counts when no lockfile in the repository resolves it.

This is read from the pull request's head commit and the lines its diff touched. A lockfile edit is checked even when no manifest changed, because a quiet change to a lockfile is exactly how a package can be swapped unnoticed.

Up to 400 packages are checked per pull request, direct dependencies first. A change that adds or moves more is reported as incomplete; it is not silently cut.

#How a package is checked

Nothing is installed and nothing from the package is run. The check uses fixed rules, not Nullgap AI, over three sources:

  1. published advisories for that exact version;
  2. the registry's own information about the package;
  3. the files of the package's published archive, read as text.

Then your organisation's package rules are applied.

#What is reported

VerdictOn the pull requestSeverityAssurance level
Malicious, listed in a published malware advisoryA finding, with a comment on the changed lineCriticalVerified
Malicious, judged from the package's published files, no advisory yetA findingCriticalUnverified
SuspiciousA finding listing the signalsUp to high, never criticalUnverified, always
Blocked by one of your package rulesA finding titled Blocked by organisation policyHighVerified
Vulnerable: a vulnerability advisory lists the versionListed on the review. Not a finding.n/an/a
Unknown: it could not be checkedListed on the review, and the review is marked incompleten/an/a
CleanNothing, unless a rule blocks itn/an/a

Notes:

  • A suspicious package is a signal for a person to weigh. It is never called verified or proven. The finding says what to look at before trusting the package.
  • A package on your allowed packages list is not raised as suspicious. Being on that list never clears a malicious verdict.
  • A package can produce two findings: one about what it is (malicious or suspicious) and one because a rule blocks it.
  • A known vulnerability in a dependency is listed, not raised as a finding, because nothing showed that your application can be attacked through it. See advisories and findings.
  • A lockfile or requirements file that points package downloads at a risky source can also be reported, as an unverified finding titled Risky package source setting.

Package findings are located on the manifest or lockfile line that brought the package in, so they get an inline comment there like any other finding. They are triaged the same way; see comments and commands.

#Package rules and the severity gate

There are two different ways a package can fail the check.

The finding isDoes it fail the check?
A package blocked by one of your package rulesAlways, whatever failCheckAt is set to, including off. A rule is a decision your organisation already made.
A malicious package (critical)When the gate is critical, high, medium or low. Not when the gate is off.
A suspicious packageOnly when its severity is at or above the gate. With the default gate of high, a suspicious package rated medium or low leaves the check neutral.

So if you want a malicious package to fail the check even with the gate switched off, the gate is the wrong tool: there is no setting for that today. Keep the gate at critical or lower.

The rules that can block a package are the blocked packages list, a minimum age for a version, a ban on install scripts with its exceptions, and the list of allowed registries. A rule that could not be evaluated, for example because a publication date is not known, does not block; it is noted. Setting the rules is on package policy.

The exact rules for the three conclusions are on checks and gating.

#When the check cannot finish

The supply-chain check never fails a review by breaking. If it cannot finish, the review still completes, but it is marked incomplete and cannot end in success:

SituationWhat the review says
The check itself failedThe packages this change adds could not be checked.
Some packages could not be checked against the advisory database, or there were more than 400Some packages this change adds could not be checked, with the reason when there is one

In both cases:

  • the check's conclusion is neutral unless a finding at or above the gate, or a blocked package, makes it failure;
  • the review carries the sentence in supplyChainNote;
  • package findings from earlier reviews of the same pull request stay open, because nothing can be concluded from a check that did not run.

Unknown is not clean. Push a new commit to run the check again. See incomplete reviews.

#How package findings close

A package finding stays open while reviews of the pull request keep reporting it. When a later review, whose supply-chain check finished, no longer reports the package (because you removed it or moved to another version), the finding is closed as patched.

#Reading the result in the API

GET on a review returns the packages in supplyChain, and the list route gives supplyChainCount.

curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/reviews/rev_7a8b9c0d1e2f"

Abridged answer:

{
  "id": "rev_7a8b9c0d1e2f",
  "status": "completed",
  "checkConclusion": "failure",
  "supplyChainNote": null,
  "supplyChain": [
    {
      "ecosystem": "npm",
      "name": "fast-csv-utils",
      "version": "1.0.2",
      "verdict": "clean",
      "reasons": [],
      "advisories": [],
      "manifest": "package-lock.json",
      "direct": true,
      "policy": {
        "action": "block",
        "rule": "min-age",
        "detail": "fast-csv-utils@1.0.2 was published 2 day(s) ago; the organisation requires at least 7."
      }
    }
  ]
}

Each row has the verdict, the reasons found, the advisories, the policy rule that matched if any, and findingId when a finding about the package itself was raised. A clean package appears only when a rule blocks it.

To check packages without a pull request, for a pipeline step or a registry proxy, use the package check API. The full field lists are in the pull request reviews reference and the dependencies reference.

Nullgap documentation