Pull request review
Checks and gating
How a review's result appears on the commit, when it passes, fails or stays neutral, and the settings that control what is posted.
Every review reports its result on the head commit of the pull request, as a check named Nullgap Security Review. This page says exactly which result is given when, and how to use it to block a merge.
#The check on the commit
| Moment | Status | Title |
|---|---|---|
| The review starts | In progress | Security review in progress |
| The review finishes | Completed, with a conclusion | No issues found, N findings, N findings (review incomplete) or Review incomplete |
| The pull request is skipped with a notice | Completed, neutral | Review skipped, with the reason |
| The review fails or is cancelled after it started | Completed, neutral | Review failed or Review cancelled |
The finished check's summary lists each finding as severity and title, and links to the review in the console. If findings were held back because of earlier triage decisions, the summary says how many.
#Check conclusions
A finished review has one of three conclusions. It is also on the review in the API as checkConclusion.
| Conclusion | Given when |
|---|---|
failure | At least one finding of this review is at or above the gate severity (failCheckAt), or a package in the change is blocked by one of your organisation's package rules. |
neutral | Not a failure, and either the review has at least one finding (all below the gate) or the review is incomplete. |
success | No findings, and the review is complete. |
What counts as "a finding of this review":
- Every finding this review itself reported at that commit. A flaw that an earlier review of the same pull request found counts again when this review finds it again, which is the usual case because each review looks at the whole change.
- Not an open finding from an earlier review that this review did not find again. Such a finding stays open (see how findings close), but it does not fail this commit's check. Look at the pull request's open findings, not only at the latest check, before merging.
- Not findings that were held back because you dismissed them, and not findings this push resolved.
- The assurance level does not matter. An unverified finding at or above the gate fails the check like a proven one.
A review never ends in success when part of the change could not be looked at. See incomplete reviews.
#Severity gating
failCheckAt is the lowest severity that fails the check. The default is high.
failCheckAt | In the console | The check fails on |
|---|---|---|
critical | Critical and above | Critical |
high | High and above | Critical, high |
medium | Medium and above | Critical, high, medium |
low | Low and above | Critical, high, medium, low |
off | Never fail the check | Nothing, with one exception below |
Findings of severity info never fail the check.
The exception: a package that one of your package rules blocks fails the check whatever the gate is, including off. A blocked package is a decision your organisation already made. See supply-chain checks.
The gate is one setting for the whole organisation.
#Blocking a merge
Nullgap sets the result; your code host decides what a result means for merging. To block merges on a failing review, make Nullgap Security Review a required check (or required status) for the branch on your code host.
Only failure blocks. A neutral result, which includes skipped, failed and incomplete reviews, is treated as passing: GitHub counts a neutral check as passing for a required check, and the other code hosts are given a passing status for it (see the next section).
#How the result appears on each code host
GitHub has a check with a neutral outcome, and Azure DevOps has a state for a status that does not count. GitLab and Bitbucket Cloud have neither: every finished state other than a pass blocks merging there, so a neutral result is written as a pass and marked in its text.
| Code host | Shown as | In progress | success | failure | neutral |
|---|---|---|---|---|---|
| GitHub, GitHub Enterprise Server | Check run | In progress | Success | Failure | Neutral |
| GitLab | Commit status | Running | Success | Failed | Success, with a description that starts "Neutral, not a pass:" |
| Bitbucket Cloud | Build status | In progress | Successful | Failed | Successful, with "(neutral)" after the name and a description that starts "Neutral, not a pass:" |
| Azure DevOps | Pull request status | Pending | Succeeded | Failed | Not applicable |
On GitLab, the status may be missing on a merge request from a fork, because the commit is on no branch of the project.
#Incomplete reviews
A review is incomplete when part of its work did not happen:
- a reviewer failed, or the review ran into the workspace's usage limit before it had looked at its share of the change;
- the supply-chain check could not finish, or could not check some of the packages the change adds.
What you see:
- The conclusion is
neutralwhen there are no gate-level findings, neversuccess. If a finding at or above the gate was found before the review ran short, the conclusion is stillfailure. - The check title says Review incomplete, or adds (review incomplete) after the number of findings.
- The check summary and the review's log carry a line that starts This review is incomplete: and says what could not be reviewed.
- For the supply-chain part, the review has
supplyChainNotein the API.
What to do: the same commit is not reviewed twice, so push a new commit to get a full review. If reviews keep ending incomplete, see getting help.
#Settings that control posting
These are organisation settings. In the console they are under Settings, on the PR reviews tab. Only admins and owners can change them.
| Setting | In the console | Default | What it does |
|---|---|---|---|
postChecks | Update check statuses | true | Show progress and the result as the check on each commit. When off, nothing is written to the commit; the conclusion is still recorded on the review. Remove the check from your required checks before switching this off. |
postComments | Post comments | true | Post the summary comment and inline comments when there is a finding. When off, findings appear only in the console, the API and your notifications, and a fix is always delivered as a pull request, never as a suggested change. |
postSystemComments | Post notices | true | Comment on the pull request when a review was skipped for a reason you should know about. Needs postComments to be on as well. |
failCheckAt | Fail the check at | high | The gate severity; see above. |
commentStyle | Comment style | standard | How much each comment says: concise, standard or detailed. See comments and commands. |
With both postChecks and postComments off, reviews run and post nothing on the code host.
Read the settings (any key):
curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/settings"
Change them (a key with the admin scope). Only the fields you send are changed.
curl -s -X PATCH -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"failCheckAt": "medium", "postSystemComments": false}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/settings"
A value that is not one of those listed is refused with 400.
#Using the result outside the code host
- The
review.completedwebhook carries the review'scheckConclusionand its findings. GETon a review returnsstatus,checkConclusion,findingsCountand the findings; see what is reviewed.- For gating a pipeline directly, see CI.
The full field list is in the organisation reference.
Nullgap documentation