Nullgap.dev Docs

Start here

Your first reviewed pull request

Connect GitHub, switch review on for a repository, open a pull request and read the result.

This walks through the shortest path from a new organisation to a reviewed pull request, using GitHub. For another code host, connect it first (which way to connect) and continue from step 3.

#What you need first

  • A Nullgap organisation in which you are an admin or owner.
  • An active trial or plan. Without one, pull requests are recorded as skipped and not reviewed.
  • On GitHub: an account that owns the repositories, or that administers every repository you will let Nullgap read. The reason is the verified link step.

#Step 1: start the trial

When you create an organisation, tick Start the free trial now. I accept the services agreement. If the organisation already exists without a plan, open Billing and press Start trial. Where workspaces are set up by invitation, Nullgap has already chosen the plan with you.

#Step 2: connect GitHub

  1. Open Integrations. Under Source code, in the GitHub row, press Install on GitHub.
  2. On GitHub, choose the account and the repositories the app may read, and install it.
  3. GitHub sends you back to the console. Nullgap then asks GitHub whether your GitHub account administers that installation. GitHub may ask you to confirm who you are.
  4. When GitHub says yes, the installation is linked to your organisation and you land on Repositories.

If GitHub does not send you back, return to Integrations, press Link installation and paste the installation number: the number at the end of the address of the page GitHub shows after installing, after /installations/.

If you end on a page titled GitHub was not connected, it says why. The causes and what to do are listed under the verified link step.

#Step 3: check that review is on for the repository

Open Repositories. Each repository from the installation has a PR review switch. New repositories start with it on. Switch off the ones you do not want reviewed.

With the API, list the repositories (any key), then switch review on or off for one (a key with the admin scope):

curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/repositories"
curl -s -X PATCH -H "Authorization: Bearer $NULLGAP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"reviewEnabled": true}' \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/repositories/repo_3f9a1c2b7d4e"

Use the id of your repository from the first call in place of repo_3f9a1c2b7d4e.

#Step 4: open a pull request

Open a pull request in that repository that is not a draft. A review is queued when the pull request opens, and again for each push to it.

A review runs only when all of these hold. The full list, with what you see for each, is under why a review is skipped.

ConditionIf it does not hold
Review is on for the repositorySkipped silently
The workspace is not suspended and has an active trial or planSkipped, with a notice on the pull request
The workspace has not reached its monthly usage limitSkipped, with a notice
The pull request is not a draftSkipped silently
Your review filters do not exclude itSkipped silently
The change touches no more than 500 filesSkipped, with a notice
The pooled review quota for the month is not used up, or overage is onSkipped, with a notice

Seats need no action from you at this point: the author of a reviewed pull request is given a review seat automatically the first time, bots included. Seats are managed under People; see usage.

#What appears on the pull request

A check. The check Nullgap Security Review shows Security review in progress while the review runs, then its result:

  • success, titled No issues found;
  • failure, when a finding is at or above the severity your organisation gates on (the default is high);
  • neutral, when there are findings below that severity, or the review could not look at everything.

The exact rules are on checks and gating.

A summary comment. Posted when the review reports at least one finding: a table of the findings with their severity, location, state and a reference (fnd_...), followed, in the default comment style, by a section for each. It is one comment, edited in place by later reviews. A review of a pull request with nothing to report posts no comment.

Inline comments. Each new finding that sits on a line the pull request changed gets a comment on that line, saying what an attacker can do and how to fix it.

In the console, the review is listed under PR reviews with its findings and its log.

#What to do with the first finding

  1. Read it. The comment gives the impact and the fix; the summary table links to the finding in the console, where the full evidence is. Note its assurance level: proven, verified or unverified.
  2. If it is real, fix it. Either push a fix yourself, or reply @nullgap fix in the finding's thread to have a fix prepared and checked. When your next push removes the flagged code, the next review closes the finding as patched and marks it Resolved in the summary.
  3. If it is not a flaw, say why. Reply !fp <reason> in the thread. If it is real and you accept it, reply !accept <reason>. The reason is kept and the finding is not raised again.

All reply commands, and who may use them, are on comments and commands. You can make the same decisions in the console; see triage.

#Review a pull request by hand

To review a pull request that was opened before you connected, or one that was skipped as a draft, comment @nullgap review on it, or press Review a pull request on the PR reviews page. With the API (a key with the write scope):

curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"repoId": "repo_3f9a1c2b7d4e", "prNumber": 7}' \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/reviews"

Then read the result:

curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/reviews?limit=5"

#If nothing happens

  • No check and no comment. Look at PR reviews. Every skipped review is recorded there with its reason, including the ones that post nothing on the pull request. If the pull request is not listed at all, the repository is not connected: press Sync on Repositories, and check on GitHub that the app can read the repository.
  • A check but no comment. That is the normal result for a review with no findings.
  • The check says the review failed. Push a new commit or comment @nullgap review to run it again.

#Next

Nullgap documentation