Pull request review
What is reviewed
What starts a pull request review, what it looks at, why a review is skipped, and how its findings close.
A review is one security review of one pull request at one commit. This page says exactly when one runs and what it covers.
#What starts a review
| Trigger | trigger in the API | Notes |
|---|---|---|
| A pull request is opened or reopened | opened | In a repository with review switched on. |
| New commits are pushed to it | synchronize | Each new head commit gets its own review. |
Someone comments @nullgap review | comment | Works on drafts and on repositories with review switched off. |
| Someone asks in the console or through the API | manual | See manual review. |
Closing or merging a pull request starts nothing.
There is one review per head commit. A second event for a commit that is already queued, running or reviewed returns the existing review. When a new commit arrives while the review of the previous one is still queued, the queued one is cancelled.
On GitHub and GitLab, marking a draft as ready does not start a review by itself, because no commit changed. Push a commit or comment @nullgap review.
#What a review looks at
Nullgap fetches the repository at the pull request's head commit and reviews what the change brings. Flaws in code the pull request does not touch are a pentest's job, not a review's.
It reads:
- The changed files, except removed ones, with the diff.
- The code around the change: the functions the change touches, their callers and callees, the entry points near them, and files linked to the changed ones.
- Your context: the threat model, your notes and the project rules of the repository's application, lessons from earlier triage decisions, and deployment files found in the repository. See project rules and custom checks.
- The pull request's title and description.
It then does these things, in this order:
- Static analysis of the changed files. Only results on or right next to changed lines are kept as leads.
- Discovery: the changed files are examined in groups, more than once, each pass looking from a different angle.
- An access-control pass over the whole change.
- Your custom checks that apply to a changed file, in one pass.
- An independent review of every candidate. A candidate that is refuted is dropped and never reaches you.
- Reproduction in the sandbox of a limited number of the verified findings of high or critical severity. See assurance levels.
- A supply-chain check when the change touches a manifest or lockfile.
The skills that are switched on and apply to the changed files also take part, at most four in one review. What a skill raises is a candidate like any other and goes through the independent review in step 5.
#Limits on what is covered
- Application folder. If the repository's main application has a folder other than the whole repository, only changed files inside that folder are reviewed.
- Large changes. Changed files are grouped by folder, up to four files to a group, and at most twelve groups get the in-depth discovery pass: the ones with the most added lines. Files beyond that are still seen by the access-control pass, your custom checks and the supply-chain check, but not examined in depth, and the review is not marked incomplete for this reason. Keep pull requests small if you want every file examined.
- More than 500 changed files. The pull request is not reviewed at all. See below.
- Running out of the workspace's usage limit mid-review. The review finishes with what it has and is marked incomplete. See incomplete reviews.
#Why a review is skipped
A pull request that will not be reviewed is still recorded, under PR reviews with the status skipped and the reason. The checks are made in this order, and the first that applies decides.
| Order | Reason recorded | When | Shown on the pull request | Bypassed by @nullgap review or a manual review |
|---|---|---|---|---|
| 1 | PR reviews are not enabled for this repository | Review is switched off for the repository. | No | Yes |
| 2 | This workspace is suspended. Contact Nullgap support. | Nullgap has suspended the workspace. | Yes | No |
| 3 | No active trial or subscription. Start a trial or subscribe to resume reviews. | There is no plan, or the trial has ended. | Yes | No |
| 4 | This workspace has reached its monthly usage limit. Contact Nullgap support to raise it. | A usage ceiling set for the workspace is used up for the month. | Yes | No |
| 5 | Draft pull request | The pull request is a draft. | No | Yes |
| 6 | Excluded by ... filter, or Does not match any ... include filter | A review filter leaves it out. | No | Yes |
| 7 | This change touches N files; reviews cover up to 500. Split it into smaller pull requests. | More than 500 changed files. | Yes | No |
| 8 | The pooled review quota for this month is used up. Turn on overage or give more members a seat to resume reviews. | The organisation's monthly review pool is used up and overage is off. | Yes | No |
"Shown on the pull request" means a neutral check titled Review skipped, and a comment that starts Nullgap: this pull request was not reviewed. The check needs postChecks, the comment needs both postComments and postSystemComments; see checks and gating. The silent skips post nothing, so that a draft or a filtered bot does not fill a pull request with notices.
#Seats and the review pool
Between checks 7 and 8, if the author of the pull request is a member of the workspace with a linked code-host login and holds no seat, they are given one. Only members take a seat: a pull request from a bot or an outside contributor is reviewed from the shared pool and takes no seat. A pull request that was skipped earlier in the list, for example by an author filter, uses no seat.
Each seat adds reviews to a monthly pool shared by the whole organisation. Reviews that are queued, running or completed in the current month count against it; skipped, failed and cancelled ones do not. Seats and overage are on People and Billing; see usage.
#Pull requests from forks
Pull requests from forks are reviewed like any other. Two things differ: Nullgap does not open fix pull requests for a finding that came from a fork, and automations do not start a fix for one.
#Re-review on new commits
Every push gets a review of the new head commit. For the pull request as a whole:
- A flaw found again is the same finding. It gains a sighting and gets no second inline comment.
- A new flaw becomes a new finding with its own inline comment.
- The summary comment is edited in place and lists the current findings, with the ones this push resolved marked Resolved in followed by the commit.
#Manual review
Use a manual review for a pull request opened before you connected the repository, for a draft, or for a repository with review switched off.
- On the pull request: comment
@nullgap review. The comment must start with the command. It is acted on for someone with write access to the repository, or a member of your Nullgap organisation the code host identifies; a comment from anyone else is ignored. See who may use commands. - In the console: on PR reviews, press Review a pull request, choose the repository and enter the pull request number.
- With the API (a key with the
writescope):
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"repoId": "repo_3f9a1c2b7d4e", "prNumber": 7}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/reviews"
Abridged answer:
{
"id": "rev_7a8b9c0d1e2f",
"repoFullName": "acme/shop",
"prNumber": 7,
"headSha": "5d41402abc4b2a76b9719d911017c592a1b2c3d4",
"trigger": "manual",
"status": "queued",
"skipReason": null,
"checkConclusion": null
}
A manual review still needs an active plan, room under the usage limit and the review pool, and no more than 500 changed files. If it cannot run, the answer has status: "skipped" and the skipReason.
A manual review does not review the same commit twice. If the head commit already has a queued, running or completed review, that review is returned. To review again, push a commit.
Read a review, its findings and its log:
curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/reviews/rev_7a8b9c0d1e2f"
curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/reviews/rev_7a8b9c0d1e2f/logs"
The list route takes status, repoId, limit and offset. The full field list is in the pull request reviews reference.
#How findings close
Each review of a pull request looks again at that pull request's open findings.
| What the new review sees | What happens to the finding |
|---|---|
| The flagged code is no longer in the head commit | Closed as patched by Nullgap, with the commit recorded. The summary shows Resolved in that commit and the inline comment gets the same note. |
| The flagged code is unchanged, and this review looked at the same place and refuted it | Closed as a false positive by Nullgap, with the reviewer's reason. |
| The flagged code is unchanged, and this review did not report it again | Stays open. A review that did not find a flaw again has not shown it fixed. |
| A package finding, and the supply-chain check did not finish | Stays open. Nothing can be concluded from a check that did not run. |
"The flagged code" is the flagged line as stored on the finding, compared with the file at the new head commit.
Other things to know:
- A finding you dismissed that is found again is not raised again. It is listed on the review as held back, with the decision it rests on, and the check says how many were held back. See dismissal memory.
- A patched finding that is found again is reopened.
- Merging or closing the pull request closes nothing. Reviews of other pull requests do not close it either: each review closes only its own pull request's findings. A finding that was open when the pull request merged stays open until someone triages it, so mark it patched once its fix has merged.
- Every change of state by Nullgap is recorded with the reason, and your linked Jira issue follows it. See triage.
Nullgap documentation