Nullgap.dev Docs

Pull request review

Comments and commands

What Nullgap posts on a pull request, the reply commands that triage a finding from its thread, and when a fix arrives as a suggested change.

A review that has something to report posts a summary comment and inline comments. You can answer them with short commands to triage a finding without opening the console.

Comments are posted only when postComments is on; see settings that control posting.

#The summary comment

There is one summary comment per pull request, headed Nullgap Security Review. Later reviews edit it in place.

It is posted when a review reports at least one finding. A review with nothing to report posts no comment, unless an earlier review of the same pull request already posted one; then the comment is updated, for example to All clear once every finding is resolved.

It contains:

  • a line with the number of open findings by severity, and the commit that was reviewed;
  • a note when any finding is unverified;
  • a table with one row per finding (its columns are below);
  • in the standard and detailed styles, a section for each finding;
  • a line reminding reviewers of the reply commands.
ColumnContent
SeverityCritical, high, medium, low or info.
FindingThe title, linked to the finding in the console. Marked (unverified) when it is.
WhereFile and line.
StateOpen, or Resolved in followed by the commit, or how it was triaged.
RefThe finding's reference, starting with fnd_. Use it in reply commands on the conversation.

When a pull request has more findings than fit in one comment, the comment says how many are not shown and the full list is in the console.

#Inline comments

A finding gets an inline comment when it is new in this review and sits on a line the pull request changed, or within fifteen lines of one in the same file; the comment is then placed on the nearest changed line. A finding elsewhere in the code appears in the summary only. A finding that was already reported by an earlier review does not get a second inline comment.

An inline comment gives the severity and title, what an attacker can do, how to fix it, and a collapsed Fix with your coding agent section with a hand-off text for fixing it in your own checkout.

When a later push removes the flagged code, the inline comment is edited to add Resolved in and the commit.

#Comment style

commentStyle is an organisation setting, changed under Settings, PR reviews, Comment style, where a preview shows the effect.

StyleSummary commentInline comment
conciseThe table only.Title, impact and fix.
standard (default)The table, then a section per finding with the description and the fix.Adds a short description.
detailedAdds the evidence list, the probe and its output when there is one, and a patch when there is one.A longer description, and the patch when there is one.

A patch shown in a comment is labelled Verified patch or Unverified patch (review before use).

#Reply commands

Reply to a finding with one of these. The command must be the first thing in your reply. Commands are not case-sensitive, and quoted lines (starting with >) are ignored.

ReplyEffectNotes
!fp <reason>Marks the finding as a false positive.A reason of at least three characters is required. !false-positive also works.
!accept <reason>Marks the finding as an accepted risk.A reason is required. !accepted and !accept-risk also work.
!validConfirms the finding. If it was closed, it is reopened.Text after the command is kept as a note.
!reopenReopens a closed finding.
@nullgap fixStarts a verified fix for the finding.The finding must be open. Uses automation credits. See suggested changes.
Anything elseSaved as a note on the finding.Up to 2,000 characters. Nothing else changes.

Examples:

!fp the value is validated in the caller, see validateOrderId()
!accept internal tool, not reachable from outside the VPN

@nullgap review is a different kind of command: it is a comment on the pull request, not a reply to a finding, and asks for a review.

#Where to reply

  • In the finding's inline thread. Nullgap knows which finding the thread is about. This is the simplest way.
  • On the pull request conversation, naming the finding's Ref from the summary table:
!fp fnd_ab12cd34ef56 the value is validated in the caller

On the conversation, a command without a Ref is answered with a request to add one, and nothing changes. A plain comment without a Ref is ordinary discussion and is left alone. Quoting a part of Nullgap's comment that names exactly one finding also identifies it.

A Ref only works for a finding of the same repository and, for a finding from a review, on its own pull request.

#What happens next

Nullgap answers in the same thread: Nullgap: done. with what it did, or Nullgap: nothing was changed. with the reason.

A !fp or !accept goes through the same path as a decision in the console: the state changes, the reason is kept, the audit log records who decided, as the code-host login followed by (pull request reply), and the finding is not raised again. See triage and dismissal memory.

#Who may use them

A reply command is acted on when its author is one of:

  • a person the code host says has write or admin access to the repository; or
  • a member of your Nullgap organisation whose identity on that code host Nullgap knows (the table says how).

Anyone else gets the answer that only members and collaborators with write access can triage findings, and nothing changes. A plain note from someone who may not triage is ignored without an answer. If the code host cannot be asked about access, or the question fails, nothing changes.

Code hostHow write access is checkedHow a member is recognised
GitHubAsked of GitHub for that repository.By the GitHub login on their Nullgap profile.
GitHub Enterprise ServerAsked of your GitHub server for that repository.Not recognised separately: write access decides.
GitLabAsked of GitLab for that project.Not recognised separately: write access decides.
Bitbucket CloudAsked of Bitbucket, which needs a token that may read the workspace's repository permissions.Not recognised separately: write access decides.
Azure DevOpsCannot be asked.By their Azure DevOps sign-in address, when it is the e-mail address of their Nullgap account.

A name is only compared with the host it belongs to: someone on GitLab or Bitbucket who has chosen the same user name as a member's GitHub login is not treated as that member.

The same rule decides who may start a review with an @nullgap review comment. A comment from anyone else is ignored without an answer, so that a stranger on a public repository cannot make Nullgap review, or post, at will. At most three such comments start a review on one pull request in ten minutes.

Other limits:

  • Comments written by bots, and by Nullgap's own account, are never acted on.
  • At most twelve replies per pull request are acted on per minute. Further replies in that minute are ignored without an answer.
  • A finding keeps its latest 100 notes.

#Suggested changes

When a fix has been prepared and checked, for example after @nullgap fix, it reaches the author in one of two forms.

#As a suggested change on the reviewed lines

The fix is posted as a suggested change in the pull request under review when all of these hold:

  • the finding came from a review of a pull request that is still open and is not from a fork;
  • the fix is for that one finding (not one fix for the same flaw in several places, and not a dependency fix);
  • nobody asked for a pull request instead (as: "pull_request" when starting the fix through the API);
  • postComments is on;
  • the fix is one block of change in one file, and every line it replaces is a line the pull request added or changed;
  • the block replaces at most 10 lines and puts at most 20 in their place;
  • the pull request's head commit is still the one the fix was checked against. If the author pushed in the meantime, no suggestion is posted.

What you get depends on the code host:

Code hostForm
GitHub, GitHub Enterprise Server, GitLabA suggestion the author accepts with one click. No pull request is opened.
Bitbucket Cloud, Azure DevOpsA fix pull request is opened, and the same change is shown as a patch in a comment on the reviewed line, with a link to that pull request.

If the code host refuses the suggestion comment, the fix is delivered as a pull request instead.

Only a fix that was run and passed its checks is ever posted as a suggestion.

#As a pull request

In every other case the fix is opened as a pull request:

  • on a new branch named nullgap/fix-..., based on the branch of the pull request the finding came from when that pull request is still open, otherwise on the repository's default branch;
  • with one commit per file when it touches several files;
  • with the author of the pull request under review asked to review it;
  • never merged and never force-pushed by Nullgap. An existing branch is never reused.

A fix is not delivered at all, and the fix run says why, when it would change build, pipeline or Nullgap configuration files, or when the base branch has moved so that the patch no longer applies.

What a fix is checked against, and what a fix run reports, is on verified fixes. Starting a fix through the API is in the fixes reference.

Nullgap documentation