Pentests
Found so far, while a pentest runs
What the "Found so far" panel shows during a pentest, what each stage means, why these leads are preliminary and never counted as findings, and what happens to them when the run completes, fails or is cancelled.
A pentest of a large codebase can take hours. You do not have to wait for the end to see what it is coming across. While it runs, the pentest page has a Found so far panel that fills as the run goes: every lead a reviewer raises, and how far it has got towards being a finding.
These are leads, not findings. Most leads are dismissed by the independent review or merged with another write-up of the same flaw before the report. A lead becomes a finding only when the run completes. Until then it is not counted anywhere findings are counted: not on the dashboard, not in the security score, deadlines, reports or exports, not in webhooks, notifications, Slack or Jira, and not in the merge gate.
#The stages
Each lead shows its title, the severity estimated so far, the file and line it points at, and its stage. A lead changes stage in place as the run moves on.
| Stage | What it means |
|---|---|
| Being checked | A reviewer raised it. Nothing has confirmed it yet. Most leads at this stage are dismissed later. |
| Confirmed by independent review | A separate review by Nullgap AI tried to refute it, with the code in front of it, and could not. It may still be merged with another write-up of the same flaw. |
| Proving… | A probe is being written and run against it in the sandbox. |
| Proven | The probe reproduced it. You can open it to read the review and the probe's outcome so far. |
| Tested: not reproduced | Confirmed by review. A working probe ran in the sandbox and the code behaved safely. It can still be reported as a confirmed finding, standing on the review alone. |
| Confirmed, could not be tested | Confirmed by review, but the sandbox could not build or run the project (for example, it needs a database or a dependency that is not available), so no probe ran. The reason is shown. This says nothing either way about the flaw. |
| Confirmed, proof not completed | Confirmed by review, but the proof stopped before a result: it reached its time or token limit, or no working probe could be written. The reason is shown. This says nothing either way about the flaw. |
| Dismissed | The independent review refuted it, or set it aside as hardening advice. The reason is shown. |
| Held for a person | The independent review could not complete. A serious lead in this state is kept for someone to look at, never silently dropped. |
Leads found to be the same flaw as another are folded into it and are not listed separately. That includes one flaw raised from several lines of the same function, such as a rate limit keyed on a client-supplied header written up at each place the header is read: it is reviewed once, its write-up records the other lines ("Same flaw also reported at …"), and the other leads show as Same flaw as another lead as soon as this is seen, even when they were raised later in the run. Two different flaws in one function stay two leads. The counts at the top of the panel (being checked, confirmed, proven, dismissed) leave them out. The confirmed count includes leads at the last three proof stages above; under it, the panel says how many of those were tested and not reproduced, how many could not be tested and how many proofs did not complete.
#Serious leads are checked first
A critical or high lead does not wait for the rest of the run. It goes to the independent review as soon as it is raised, and to proof as soon as the review confirms it, while the run goes on examining other files. So a proven finding often appears in the first minutes of a long run. A medium lead does the same when the pentest runs proofs: it is proven when a proof slot is free and no critical or high lead is waiting for one, and a medium proof that has started is not interrupted. Low leads are reviewed when the examination of the code has finished.
The standard is the same either way: the same independent review, the same sandbox, the same proof. Starting early changes when a lead is checked, not how.
#When the run ends
- Completed. The panel gives way to the findings. Each lead either became a finding, which you can open from the list, or was dismissed or merged and is gone. Nothing preliminary is left on a completed pentest.
- Failed or cancelled. The leads stay, marked from an unfinished run. They will not become findings, because the run that would have reported them did not finish. A proof that was under way when the run stopped is shown as confirmed, not as proven. They are kept so that you can see what the run had come across and decide whether to resume it or start it again. They are never counted, and they are removed with the pentest (retention, erasing findings, deleting the workspace).
#Stopping and resuming a pentest
You can cancel a queued or running pentest from its page. It stops, and its findings and reports are not produced, because the run that would have reported them did not finish.
What a cancelled pentest costs. A pentest you cancel is charged for the share of its work done before it stopped, never more than its price; the rest is given back. The share is measured from the Nullgap AI work the run did against the work its price pays for, so a pentest cancelled in its first minutes costs a small part of its price, and one cancelled near the end costs close to all of it. The cancel confirmation says about how much the work so far comes to, and the cancelled pentest's page says what it was charged (chargedForWork). A pentest that had not started is not charged. A pentest that fails because of an error on Nullgap's side, or that Nullgap stops (for example when a workspace is suspended), is not charged at all. Workspaces that are not billed per pentest are never charged.
What the pentest had finished is not thrown away. For a limited time (seven days, unless your Nullgap operator has set another period) a cancelled pentest can be resumed from where it stopped. The same holds for a pentest that failed part-way and for one an operator stopped. The pentest page and the pentests list show Resume and say until when.
Resuming continues the same pentest:
- The same code. It examines the commit it was examining before, even if the branch has moved on since, with the same scope, context and settings. If that commit can no longer be fetched from your code host (history was rewritten), the pentest cannot be resumed and nothing is charged; run it again to test the current code.
- No work twice. Files it had examined are not examined again, and leads keep the stage they had reached. It ends with one set of findings and one set of reports, as if it had never stopped.
- Unfinished proofs are tried again. A lead whose proof ended could not be tested or proof not completed in the stopped part gets a new attempt, because what stopped it (a sandbox that could not build the project, a limit) may have been put right since. Proven leads stay proven, tested: not reproduced stays as it was, and dismissed leads stay dismissed.
- Time. The days a pentest sat stopped do not count against the time a pentest may take.
What it costs. Resuming charges what is left of the pentest's price: the price, less what a cancel already charged for the work done (a failed pentest was refunded in full, so resuming it charges the price). The amount is shown before you confirm. So a pentest that is stopped and resumed is never charged more than its price in all. If the resumed pentest is cancelled again, it is charged for all the work done by both runs, still never more than its price; if it fails on Nullgap's side, it is refunded. Workspaces that are not billed per pentest are not charged, and a resumed retest stays free.
A pentest cannot be resumed when it stopped before it had recorded any progress, when the time to resume it has passed, or when its repository or code-host connection has been removed. The page says which. Run again starts a new pentest from the beginning in every case.
In the API, GET /api/orgs/{org}/pentests/{id}/resume says whether a pentest can be resumed, how far it got and what it costs, and POST to the same path resumes it. On the command line: nullgap pentest resume <id>.
#Progress
The log of a pentest reports how far the examination of the code has got in every batch of files: files examined of the total, checks run, and, once enough has been timed, an estimate of the time left at the current pace ("about 40 min", "about 2 h"). The estimate follows the pace of the last quarter of an hour, so a slow start does not weigh on it for the rest of the run, and it is shown only once enough checks have finished to measure that pace.
#In the API
GET /api/orgs/{org}/pentests/{id}/live returns the same leads, with preliminary: true on every response. Pass the response's cursor back as after to get only what changed since. While the pentest runs, state is running; after it completes, state is replaced and becameFinding maps each lead to the finding it became; after it fails or is cancelled, state is unfinished. A lead's stage is one of checking, confirmed, proving, proven, not-reproduced (a probe ran and the code behaved safely), could-not-test (the sandbox could not build or run the project), proof-incomplete (the proof stopped before a result), dismissed, held or merged. Leads recorded before could-not-test and proof-incomplete existed may say not-reproduced for either; their reason tells which. Any member of the workspace and any API key that can read pentests can read it. The full description is in the API reference.
The workspace export lists leads under their own name, pentestlive, never among findings.
Nullgap documentation