Pull request review
Supply-chain checks on pull requests
What happens when a pull request changes a manifest or lockfile, what is reported, and how package rules interact with the severity gate.
When a pull request adds a package or moves one to a new version, the review checks that package: is it known to be malicious, does it look suspicious, and do your organisation's package rules allow it.
This page covers the pull request side. For how verdicts are reached and what the signals mean, see malicious and suspicious packages. For the rules, see package policy.
#When the check runs
The check is part of every review. It has something to do when the pull request adds or modifies one of these files:
| Ecosystem | Files |
|---|---|
| npm (npm, pnpm, Yarn) | package.json, package-lock.json, npm-shrinkwrap.json, pnpm-lock.yaml, yarn.lock |
| PyPI (pip, Poetry, uv, Pipenv) | requirements*.txt, pyproject.toml, poetry.lock, uv.lock, Pipfile, Pipfile.lock |
| crates.io (Cargo) | Cargo.toml, Cargo.lock |
Other ecosystems are not checked on pull requests.
#Which packages are checked
Only what the change brings:
- a lockfile entry counts when one of its lines was added or modified by the pull request;
- a manifest line that names a new dependency counts when no lockfile in the repository resolves it.
This is read from the pull request's head commit and the lines its diff touched. A lockfile edit is checked even when no manifest changed, because a quiet change to a lockfile is exactly how a package can be swapped unnoticed.
Up to 400 packages are checked per pull request, direct dependencies first. A change that adds or moves more is reported as incomplete; it is not silently cut.
#How a package is checked
Nothing is installed and nothing from the package is run. The check uses fixed rules, not Nullgap AI, over three sources:
- published advisories for that exact version;
- the registry's own information about the package;
- the files of the package's published archive, read as text.
Then your organisation's package rules are applied.
#What is reported
| Verdict | On the pull request | Severity | Assurance level |
|---|---|---|---|
| Malicious, listed in a published malware advisory | A finding, with a comment on the changed line | Critical | Verified |
| Malicious, judged from the package's published files, no advisory yet | A finding | Critical | Unverified |
| Suspicious | A finding listing the signals | Up to high, never critical | Unverified, always |
| Blocked by one of your package rules | A finding titled Blocked by organisation policy | High | Verified |
| Vulnerable: a vulnerability advisory lists the version | Listed on the review. Not a finding. | n/a | n/a |
| Unknown: it could not be checked | Listed on the review, and the review is marked incomplete | n/a | n/a |
| Clean | Nothing, unless a rule blocks it | n/a | n/a |
Notes:
- A suspicious package is a signal for a person to weigh. It is never called verified or proven. The finding says what to look at before trusting the package.
- A package on your allowed packages list is not raised as suspicious. Being on that list never clears a malicious verdict.
- A package can produce two findings: one about what it is (malicious or suspicious) and one because a rule blocks it.
- A known vulnerability in a dependency is listed, not raised as a finding, because nothing showed that your application can be attacked through it. See advisories and findings.
- A lockfile or requirements file that points package downloads at a risky source can also be reported, as an unverified finding titled Risky package source setting.
Package findings are located on the manifest or lockfile line that brought the package in, so they get an inline comment there like any other finding. They are triaged the same way; see comments and commands.
#Package rules and the severity gate
There are two different ways a package can fail the check.
| The finding is | Does it fail the check? |
|---|---|
| A package blocked by one of your package rules | Always, whatever failCheckAt is set to, including off. A rule is a decision your organisation already made. |
| A malicious package (critical) | When the gate is critical, high, medium or low. Not when the gate is off. |
| A suspicious package | Only when its severity is at or above the gate. With the default gate of high, a suspicious package rated medium or low leaves the check neutral. |
So if you want a malicious package to fail the check even with the gate switched off, the gate is the wrong tool: there is no setting for that today. Keep the gate at critical or lower.
The rules that can block a package are the blocked packages list, a minimum age for a version, a ban on install scripts with its exceptions, and the list of allowed registries. A rule that could not be evaluated, for example because a publication date is not known, does not block; it is noted. Setting the rules is on package policy.
The exact rules for the three conclusions are on checks and gating.
#When the check cannot finish
The supply-chain check never fails a review by breaking. If it cannot finish, the review still completes, but it is marked incomplete and cannot end in success:
| Situation | What the review says |
|---|---|
| The check itself failed | The packages this change adds could not be checked. |
| Some packages could not be checked against the advisory database, or there were more than 400 | Some packages this change adds could not be checked, with the reason when there is one |
In both cases:
- the check's conclusion is
neutralunless a finding at or above the gate, or a blocked package, makes itfailure; - the review carries the sentence in
supplyChainNote; - package findings from earlier reviews of the same pull request stay open, because nothing can be concluded from a check that did not run.
Unknown is not clean. Push a new commit to run the check again. See incomplete reviews.
#How package findings close
A package finding stays open while reviews of the pull request keep reporting it. When a later review, whose supply-chain check finished, no longer reports the package (because you removed it or moved to another version), the finding is closed as patched.
#Reading the result in the API
GET on a review returns the packages in supplyChain, and the list route gives supplyChainCount.
curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/reviews/rev_7a8b9c0d1e2f"
Abridged answer:
{
"id": "rev_7a8b9c0d1e2f",
"status": "completed",
"checkConclusion": "failure",
"supplyChainNote": null,
"supplyChain": [
{
"ecosystem": "npm",
"name": "fast-csv-utils",
"version": "1.0.2",
"verdict": "clean",
"reasons": [],
"advisories": [],
"manifest": "package-lock.json",
"direct": true,
"policy": {
"action": "block",
"rule": "min-age",
"detail": "fast-csv-utils@1.0.2 was published 2 day(s) ago; the organisation requires at least 7."
}
}
]
}
Each row has the verdict, the reasons found, the advisories, the policy rule that matched if any, and findingId when a finding about the package itself was raised. A clean package appears only when a rule blocks it.
To check packages without a pull request, for a pipeline step or a registry proxy, use the package check API. The full field lists are in the pull request reviews reference and the dependencies reference.
Nullgap documentation