Nullgap.dev Docs

Start here

Core ideas

The words Nullgap uses everywhere, with exactly what each one means.

These terms are used the same way in the console, the API, reports and the rest of this site.

#Finding

A finding is one flaw in one place: a title, a severity, a location in your code (or a package), a description, and the evidence that was gathered about it.

One flaw is one finding, however often it is found. When a later review or pentest finds the same flaw again, the existing finding gains a sighting and keeps its id and its first-seen date. See duplicates.

A candidate that the independent review refuted is not a finding. It is not stored and never reaches you.

#Proof

Proof is a demonstration that the flaw can be used, run by a program and checked by a second party. An explanation of why code looks vulnerable is evidence. It is not proof.

There are two ways a finding can be proven: in the sandbox, against a copy of your code, or against a running application you put in scope. The standard for each is set out below. More on proof.

#Assurance levels

Every finding has exactly one assurance level. In the API it is derived from finding.status; exports and the MCP tools give it directly as assurance. Outbound webhook events also have a field named assurance, but it carries the status value from the middle column below, not the level: see outbound webhooks.

Levelfinding.statusWhat has happened
Provenproven, or fixed after a verified fixThe flaw was reproduced, a benign control did not show it, a repeat gave the same result, and an independent review accepted the demonstration.
VerifiedverifiedAn independent review tried to refute the finding and could not. It was not reproduced.
UnverifiedcandidateThe independent review did not complete, or the finding is a signal by nature. It may be a false positive.

#Proven in the sandbox

A finding is proven in the sandbox only when all of the following happened, in this order:

  1. A probe was written that calls your project's own code and exits one way when the flaw shows and another way when it does not.
  2. Nullgap ran the probe itself. The claim of whoever wrote the probe is not taken on trust.
  3. The probe showed the flaw.
  4. A control run, the same call with benign input, behaved safely. A probe without a control, or whose control did not hold, proves nothing and is discarded.
  5. The probe was checked to exercise your project's code. A self-contained probe that never touches the project is discarded.
  6. The probe was run a second time and gave the same result.
  7. An independent reviewer read the probe and its output and accepted that it shows the flaw the finding describes. A reviewer that gives no clear acceptance counts as a refusal.

If any step fails, the finding keeps the level it had, and the evidence records what happened: tried and not reproduced, could not run in the sandbox as it is, no working probe, or the limit for one proof was reached.

Only findings that are already verified, are of critical or high severity and have a location in your code are sent for reproduction, and each run reproduces a limited number of them, most severe first. A verified finding that was never sent for reproduction is not weaker for it. It was not tried.

#Proven against a running application

Live testing is optional and only happens in a pentest that names a running deployment. A live result counts as proven only when all four of these hold:

  1. The test supplied a recipe: the request that shows the flaw, a benign control request, and a signal that a program can look for in a response. The signal must be real and the control must differ from the attack.
  2. Without the tester, a program sent the attack again twice and the control again twice, through the same scope check as every other request.
  3. The program, not Nullgap AI, decided: the signal was in every attack response and in no control response.
  4. An independent reviewer read the requests, the responses and the signal, and accepted that together they show the flaw.

A live confirmation by the tester on its own never counts as proof, however convincing the response looked. It is attached to the finding as evidence and the level stays as it was. A live result never lowers a finding's level.

#Verified

Each candidate is handed to an independent reviewer that starts fresh, is told to assume the candidate is a false positive, and looks for a reason it cannot be used: no route from an entry point, a check in the way, input an attacker does not control. Verified means that review ran and could not refute the finding. The finding then carries the chain the reviewer confirmed, from entry point to the flawed statement.

Verified does not mean reproduced. Treat a verified finding as real and read its evidence.

Two kinds of package finding are verified in a different sense, because there is no code path to argue about: a package version that a published malware advisory lists, and a package that one of your organisation's own package rules blocks.

#Unverified

Unverified means the independent review did not complete for that finding, for example because the reviewer failed or the run reached its limit. Nullgap keeps a critical or high finding in that situation and marks it, so it is never silently dropped. A medium or low candidate whose review did not complete is not reported.

Findings about a suspicious package, and about a package judged malicious from its published files with no advisory yet, are always unverified: they are signals for a person to look at.

Comments and reports label an unverified finding as such. Confirm it before acting on it.

#What a level does not change

  • A later run of a different kind that could not reproduce a proven finding does not take the proof away.
  • Dependency advisories are not findings and never carry a level. See advisories and findings.

#Severity

critical, high, medium, low or info. Severity says how bad the flaw would be if used. It is separate from the assurance level and from priority.

#Priority

Priority is severity moved by what is known about exposure: P0 (most urgent) to P3. A finding reachable from an internet-facing entry point with no sign-in moves up one step; test-only code, or code reached only through an internal, administrative or command-line entry point, moves down. A fact that could not be determined moves nothing, and a proven finding is never placed below what its severity alone gives. Each finding lists the reasons for its priority. See priority and exposure.

#Review

A review is one security review of one pull request at one commit. It looks at the change and the code around it and reports what the change brings. A new commit on the pull request gets a new review. See what is reviewed.

#Pentest

A pentest is a whitebox assessment of one repository at a branch, tag or commit you choose, optionally narrowed to folders, optionally with live testing. It ends with findings and three reports. See scoping a pentest.

#Retest

A retest runs a finished pentest's scope again at a newer version, to check your fixes. Each pentest includes one. A finding is marked fixed only when its flagged code is gone and the retest did not find the flaw again. A finding the retest did not find, but whose code is unchanged, stays open. See retest.

#Scope

Scope is what Nullgap is allowed to look at or touch. There are two kinds.

#The scope list

The scope list is your organisation's list of hosts that live testing may send requests to. A pentest that names a running deployment is refused unless the deployment's host is on the list.

  • Entries are public host names, optionally with a port, such as staging.shop.acme.example. An entry starting with *. covers every host under that name, but not the name itself.
  • Private, loopback and internal addresses are never accepted, on the list or as a target.
  • Only put hosts on the list that you own or are authorised to test.

How to read and set the list, in the console or by API, is on live testing.

#Path scope

Path scope narrows what code is examined.

  • A pentest takes paths: up to 50 folders, relative to the repository root. Empty means the whole repository.
  • An application (see project rules and custom checks) has a folder. Pull request review looks at changed files inside the folder of the repository's main application, which is the whole repository unless you changed it.

#Triage states

Triage is your decision about a finding. A finding is in one of four states.

StateAPI valueMeaning
OpenopenNeeds attention.
PatchedpatchedFixed. Set by you, or by Nullgap when the flagged code is gone from the pull request or a retest shows it fixed.
Accepted riskaccepted_riskReal, and you decided to live with it. A reason is required. It can carry a date to look at it again.
False positivefalse_positiveNot a flaw. A reason is required.

A dismissal with its reason is remembered: the same finding, and code that is the same as code you dismissed, is not raised again, and each run lists what it held back and why. See triage and dismissal memory.

Nullgap documentation