Nullgap.dev Docs

Pentests

Proof

How a finding is reproduced in the sandbox or against a running deployment, what has to hold before it is called proven, and what each outcome means.

A finding is proven when the flaw was made to happen and the demonstration passed a fixed set of checks. This page describes those checks, in the sandbox and against a running deployment, and what is recorded when a finding could not be proven.

The three assurance levels are defined on Core ideas. In short:

LevelMeaning
ProvenA probe reproduced the flaw and the demonstration passed the checks on this page.
VerifiedAn independent review read the code path and could not refute the finding. It was not reproduced by running it.
UnverifiedThe independent review did not complete. Treat it as a lead to confirm. Only critical and high findings are ever reported at this level.

#Which findings are sent for proof

Reproducing a finding takes time, so only the most serious are tried.

  • The finding must be verified and have a code location. In a pull request review it must be of critical or high severity. In a pentest a medium finding is sent too, after every critical and high one.
  • A finding whose title starts with Latent: (a real flaw in code a pull request adds that no entry point reaches yet) is not sent.
  • Up to 8 findings per pentest and up to 6 per pull request review, most serious first. While a pentest is still examining the code, medium findings take at most half of its 8, so a critical or high finding confirmed later still has a place.

A low finding, and a medium one in a pull request review, is therefore verified at most, unless a fix run later reproduces it.

#The proof environment

A probe can only reproduce a flaw if your code can be built and started in the sandbox. A pentest finds that out in its first minutes, right after the checkout and while the code is being read, not at the first proof: it installs your dependencies from the lockfile, compiles the code in scope with no network, and reads from the repository which services the application needs.

The result is the Proof environment section on the pentest page. It appears while the pentest is still running, is written to its log, and is printed in the full report with coverage. On GET /pentests/{id} it is proofEnvironment.

verdictConsole labelMeaning
readyReadyDependencies are installed and the code in scope compiles. Findings can be reproduced by building and running it.
not-neededReadyThe code has no dependencies to install. It runs as it is.
limitedLimitedPart of it works. The rows below the summary say what does not.
unavailableCannot buildThe code cannot be built in the sandbox. Findings are still found and independently reviewed; none can be reproduced by running the code.
not-checkedNot checkedDependency installation is not switched on for the service, or the check could not be made.

Under the summary there is one row per kind of dependency and folder, per service and per language version:

RowWhat it says
A kind of dependency (parts): "Go modules", "npm packages in services/web"ready (it builds), not-checked (installed; no compile check was made, or it ran out of time), dependencies-failed with the reason (for example a dependency hosted outside the public registries, or a private registry), or build-failed with the compiler's own first line.
A service (services): "PostgreSQL"Where the repository says the application needs it, and whether the sandbox can start a throw-away one for a proof. See Services a proof needs.
A language version (toolchains): "Node.js 22 (.nvmrc)"The version your project asks for and the one the sandbox has, when they differ, and whether the sandbox's is older (the code may not build) or newer (it usually does).

What it changes:

  • Nothing stops. A pentest whose code cannot be built runs to the end. Discovery and independent review do not depend on the sandbox.
  • Dependencies are installed once. Every proof of the run reuses what this step installed and compiled; none installs again.
  • Proofs start from the facts. Each probe is written knowing what is installed, what compiles, what is missing and how to start the database, so no proof spends its time finding that out.
  • You know early. If the section says "Cannot build", the findings of this run will be verified at most. The reason is on the same line, and usually names what to change: a dependency that is not on a public registry, a language version, a service.

A pull request review installs dependencies once for its proofs as well, but does not show this section.

#Proof in the sandbox

#What a probe is

A probe is a small program that makes your project's own code misbehave and observes it. Nullgap AI writes it into a .nullgap-probe/ folder inside a disposable copy of the project. Your repository is not modified.

A probe either imports the vulnerable function and calls it the way the entry point would, or starts the application's own server on the loopback address, sends it the request and shuts it down. It runs in an isolated sandbox with no network access: nothing can be downloaded and nothing outside the sandbox can be reached. See The sandbox.

Every probe follows one exit-code contract:

Exit codeMeaning
0The unsafe behaviour happened.
1The code behaved safely: the input was rejected or neutralised.
2 or higherThe probe itself could not run: an import failed, a file was missing, the setup broke.

#What has to hold

The probe's author is not believed. Nullgap runs the probe itself, and then requires all of the following before the word proven is used.

  1. The probe reproduces. Run by Nullgap, it exits 0.
  2. The control run holds. The same probe is run in control mode, which makes the same call with an ordinary, benign value. It must exit 1. A probe whose control does not exit 1 would "succeed" on safe code too, so it proves nothing.
  3. It runs your code. Something in the probe must import, execute or send a request to the project's own code. A probe that only demonstrates the idea in isolation is rejected.
  4. The repeat run agrees. The probe is run a second time and must exit 0 again. A result that depends on timing or on leftovers from the first run is rejected.
  5. An independent review accepts the probe. A separate reviewer, which did not write the probe, reads the probe's files and the output of the runs, and must state that the probe exercises the project's code and observes the unsafe effect itself. Anything short of a clear yes counts as no.

A probe also carries a second payload: a different input for the same flaw, through the same code path. If it reproduces as well, that is recorded. A proof stands without it. A verified fix needs it.

If the first probe does not work, its author gets one more attempt, with the real output of the failed run as feedback. There are at most two attempts. By default one proof may take up to 12 minutes, and each command is stopped after 60 seconds; the service may run with other limits.

#The outcomes

Every attempt at proof leaves an evidence entry of kind probe on the finding, with a proof record. proof.outcome is one of six values.

outcomeConsole labelWhat happenedWhat it means for the finding
reproducedReproducedThe probe showed the flaw and all the checks above held.It becomes proven. The command, its output and the probe's files are kept as evidence.
not_reproducedTried, did not reproduceA working probe ran and the code behaved safely.It stays verified, with the attempt on record. It stands on the independent review alone. In a fix run, this outcome closes the finding as a false positive.
probe_brokenProbe did not runNo trustworthy probe: it errored or timed out, its control did not hold, the repeat run disagreed, or the independent review rejected it.It stays as it was. This outcome says nothing either way.
no_probeNo probe writtenNo runnable probe was produced.It stays as it was. It was not tested.
environment_missingCould not run in the sandboxThe code cannot run in the sandbox as it is. proof.missing says what was lacking, for example a database server or a toolchain.It stays as it was. This is not evidence that the finding is wrong.
budget_exhaustedStopped at the limitThe time or effort limit for one proof was reached first.It stays as it was.

Only not_reproduced is a statement about the code. The other four that are not reproduced mean the question was not answered.

#The proof record

FieldMeaning
outcomeOne of the six values above.
attemptsProbe attempts made: 1 or 2.
missingFor environment_missing: what the sandbox lacked.
dependenciesWhether the project's dependencies were available: installed for this run, already present, not_installed, or not_needed.
controlHeldThe benign control behaved safely. Absent when no control was run.
repeatableThe second run gave the same result.
auditedAn independent reviewer accepted the probe. false means no reviewer took part in that run, or the reviewer rejected the probe.
variantHeldThe second payload also reproduced. false: one was supplied and did not. Absent: none was supplied.
modelsAsked, declinedPresent only when writing the probe was declined: how many Nullgap AI models were asked, each with the same request, and how many of them declined.

The same entry carries command (what was run), exitCode, output, and in detail the checks in words followed by the probe's files. In the console this is the finding's Proof section; Show the probe command and output unfolds the rest.

{
  "kind": "probe",
  "summary": "Probe reproduced the vulnerability: the order lookup returns every row when the id carries a quote",
  "command": "node .nullgap-probe/probe.js",
  "exitCode": 0,
  "proof": {
    "outcome": "reproduced",
    "attempts": 1,
    "dependencies": "not_needed",
    "controlHeld": true,
    "repeatable": true,
    "audited": true,
    "variantHeld": true
  }
}

The sample is abridged: output and detail are left out.

#The live proof standard

When a pentest includes live testing, a finding can also be proven against the running deployment. A live confirmation alone is the testing step's reading of the responses it got. It becomes proof only when all four of the following hold.

  1. The recipe is usable. The testing step hands over the attack request, a benign control request, and a signal that a program can check in a response. The signal must be specific: text of at least six characters in the response body, or a response header value of at least three. A status code alone is not a signal. The control must be a different request from the attack.
  2. Each is re-run twice. Without the testing step taking part, the attack and the control are each sent again, twice, in the order attack, control, attack, control, through the same scope checks as every other request.
  3. A program decides. The signal must be in every attack response and in no control response.
  4. An independent review accepts it. A reviewer reads the requests, the responses and the signal, starting from the assumption that the demonstration is not valid. It is rejected if the signal would appear for an ordinary authorised user, if it does not show what the finding claims, if the control differs from the attack in more than the attacking part, if injected markup comes back escaped, or if the identity replaying the request was entitled to the data. The same standard decides every Leaked cell of the access matrix.

A result that passes all four makes the finding proven. The evidence entry of kind dynamic then begins Proven against the running application, and its proof record has outcome: "reproduced", attempts: 2 and the three flags controlHeld, repeatable and audited.

A result that falls short stays evidence. The entry begins Live dynamic test: with the verdict, says which condition was not met, and the assurance level does not change. In the proof record, probe_broken then means the attack reproduced but the control or the review failed, and not_reproduced means the signal did not appear in both re-runs.

A re-run that would change data is only made when the pentest has the state-changing permission.

#When something cannot be proven

An unproven finding is not a dismissed finding. What happens:

  • The finding keeps its level, verified or unverified, and is reported with the outcome and the reason on record.
  • A proof the finding already has is not taken away. If a pentest proved a flaw and a later review of a pull request could not reproduce it, the stored finding stays proven.
  • A finding is never placed lower in priority for being unproven. A proven one is never placed below what its severity gives.
  • A proof outranks an earlier opinion. A finding your team marked as a false positive is raised again if it is later reproduced. See Memory of dismissals.

What you can do:

OutcomeNext step
environment_missingRead proof.missing and the pentest's proof environment: it says whether the dependencies could be installed, whether the code builds in the sandbox and which service it needs. If the flaw needs a running service the sandbox cannot start, add live testing to the next pentest.
probe_broken, no_probe, budget_exhaustedConfirm the finding by hand from its description and evidence, or start a fix run, which makes its own attempt at reproduction first.
not_reproducedRead the independent review's reasoning. If you agree the code is safe, mark the finding as a false positive with the reason.
Not sent for proofA low finding (or a medium one in a pull request review), or one beyond the limit per run. A fix run on it starts with a proof attempt.

Nullgap documentation