Nullgap.dev Docs

Pull request review

Checks and gating

How a review's result appears on the commit, when it passes, fails or stays neutral, and the settings that control what is posted.

Every review reports its result on the head commit of the pull request, as a check named Nullgap Security Review. This page says exactly which result is given when, and how to use it to block a merge.

#The check on the commit

MomentStatusTitle
The review startsIn progressSecurity review in progress
The review finishesCompleted, with a conclusionNo issues found, N findings, N findings (review incomplete) or Review incomplete
The pull request is skipped with a noticeCompleted, neutralReview skipped, with the reason
The review fails or is cancelled after it startedCompleted, neutralReview failed or Review cancelled

The finished check's summary lists each finding as severity and title, and links to the review in the console. If findings were held back because of earlier triage decisions, the summary says how many.

#Check conclusions

A finished review has one of three conclusions. It is also on the review in the API as checkConclusion.

ConclusionGiven when
failureAt least one finding of this review is at or above the gate severity (failCheckAt), or a package in the change is blocked by one of your organisation's package rules.
neutralNot a failure, and either the review has at least one finding (all below the gate) or the review is incomplete.
successNo findings, and the review is complete.

What counts as "a finding of this review":

  • Every finding this review itself reported at that commit. A flaw that an earlier review of the same pull request found counts again when this review finds it again, which is the usual case because each review looks at the whole change.
  • Not an open finding from an earlier review that this review did not find again. Such a finding stays open (see how findings close), but it does not fail this commit's check. Look at the pull request's open findings, not only at the latest check, before merging.
  • Not findings that were held back because you dismissed them, and not findings this push resolved.
  • The assurance level does not matter. An unverified finding at or above the gate fails the check like a proven one.

A review never ends in success when part of the change could not be looked at. See incomplete reviews.

#Severity gating

failCheckAt is the lowest severity that fails the check. The default is high.

failCheckAtIn the consoleThe check fails on
criticalCritical and aboveCritical
highHigh and aboveCritical, high
mediumMedium and aboveCritical, high, medium
lowLow and aboveCritical, high, medium, low
offNever fail the checkNothing, with one exception below

Findings of severity info never fail the check.

The exception: a package that one of your package rules blocks fails the check whatever the gate is, including off. A blocked package is a decision your organisation already made. See supply-chain checks.

The gate is one setting for the whole organisation.

#Blocking a merge

Nullgap sets the result; your code host decides what a result means for merging. To block merges on a failing review, make Nullgap Security Review a required check (or required status) for the branch on your code host.

Only failure blocks. A neutral result, which includes skipped, failed and incomplete reviews, is treated as passing: GitHub counts a neutral check as passing for a required check, and the other code hosts are given a passing status for it (see the next section).

#How the result appears on each code host

GitHub has a check with a neutral outcome, and Azure DevOps has a state for a status that does not count. GitLab and Bitbucket Cloud have neither: every finished state other than a pass blocks merging there, so a neutral result is written as a pass and marked in its text.

Code hostShown asIn progresssuccessfailureneutral
GitHub, GitHub Enterprise ServerCheck runIn progressSuccessFailureNeutral
GitLabCommit statusRunningSuccessFailedSuccess, with a description that starts "Neutral, not a pass:"
Bitbucket CloudBuild statusIn progressSuccessfulFailedSuccessful, with "(neutral)" after the name and a description that starts "Neutral, not a pass:"
Azure DevOpsPull request statusPendingSucceededFailedNot applicable

On GitLab, the status may be missing on a merge request from a fork, because the commit is on no branch of the project.

#Incomplete reviews

A review is incomplete when part of its work did not happen:

  • a reviewer failed, or the review ran into the workspace's usage limit before it had looked at its share of the change;
  • the supply-chain check could not finish, or could not check some of the packages the change adds.

What you see:

  • The conclusion is neutral when there are no gate-level findings, never success. If a finding at or above the gate was found before the review ran short, the conclusion is still failure.
  • The check title says Review incomplete, or adds (review incomplete) after the number of findings.
  • The check summary and the review's log carry a line that starts This review is incomplete: and says what could not be reviewed.
  • For the supply-chain part, the review has supplyChainNote in the API.

What to do: the same commit is not reviewed twice, so push a new commit to get a full review. If reviews keep ending incomplete, see getting help.

#Settings that control posting

These are organisation settings. In the console they are under Settings, on the PR reviews tab. Only admins and owners can change them.

SettingIn the consoleDefaultWhat it does
postChecksUpdate check statusestrueShow progress and the result as the check on each commit. When off, nothing is written to the commit; the conclusion is still recorded on the review. Remove the check from your required checks before switching this off.
postCommentsPost commentstruePost the summary comment and inline comments when there is a finding. When off, findings appear only in the console, the API and your notifications, and a fix is always delivered as a pull request, never as a suggested change.
postSystemCommentsPost noticestrueComment on the pull request when a review was skipped for a reason you should know about. Needs postComments to be on as well.
failCheckAtFail the check athighThe gate severity; see above.
commentStyleComment stylestandardHow much each comment says: concise, standard or detailed. See comments and commands.

With both postChecks and postComments off, reviews run and post nothing on the code host.

Read the settings (any key):

curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/settings"

Change them (a key with the admin scope). Only the fields you send are changed.

curl -s -X PATCH -H "Authorization: Bearer $NULLGAP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"failCheckAt": "medium", "postSystemComments": false}' \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/settings"

A value that is not one of those listed is refused with 400.

#Using the result outside the code host

  • The review.completed webhook carries the review's checkConclusion and its findings.
  • GET on a review returns status, checkConclusion, findingsCount and the findings; see what is reviewed.
  • For gating a pipeline directly, see CI.

The full field list is in the organisation reference.

Nullgap documentation