Pentests
Retest
Use the retest included with each pentest to check your fixes, and read what fixed, still open and added mean in its result.
Each pentest includes one retest. It tests the same code again, on the same branch or tag, at the version that now holds your fixes, marks the original findings that are fixed, and writes updated reports and a new attestation letter.
#What it costs
- The retest is free while the code in scope has grown by no more than a quarter of the lines the original counted.
- Past that, it costs the difference between the price of its size and what the original pentest cost. The retest dialog says the size now, the free allowance and the amount before you start, and the retest never costs more than the amount shown.
- It is used up as soon as it starts, even if you then cancel it. A retest that fails because of an error on Nullgap's side, or that Nullgap stops, does not use it up.
- Workspaces billed under an agreement are not charged.
#What you need first
- The admin or owner role, or an API key with the
adminscope. - The original pentest must have finished: its status is
completed. - The fixes merged into, or pushed to, the branch or tag the original pentest tested.
- The repository still connected.
#Starting the retest
In the console, open the finished pentest and press Request retest. The dialog counts the code and says whether the retest is free; press Start retest. You are taken to the new pentest.
By API, ask the estimate first, then start it with the extra it gave as the ceiling (0 when it is free):
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/pentests/pen_4c1d9e7a2b60/retest/estimate"
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"maxPriceUsd":0}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/pentests/pen_4c1d9e7a2b60/retest"
A ref other than the original's is refused with 400 retest_ref: to test another branch, start a new pentest.
The answer is a new pentest. Its retestOf field holds the id of the original, and the original's retestId now points at the retest.
{
"id": "pen_9b3f0a6c5d17",
"repoFullName": "acme/shop",
"ref": "main",
"paths": ["services/api"],
"status": "queued",
"retestOf": "pen_4c1d9e7a2b60",
"retest": null
}
The sample is abridged.
#Giving a ref
ref is the version to retest.
- A branch means its latest commit at the moment the retest starts.
- A tag or a commit pins the retest to one exact version. Use one when the new attestation letter must name the release you shipped.
- If you leave
refout, the original'srefis used. For a branch that is the branch as it is now, so this works when the fixes were merged into the branch that was tested. For a tag or a commit it is the same code as before, and nothing will have changed.
#What the retest reuses
Everything except the version comes from the original pentest and cannot be changed for the retest:
- the folders in scope;
- the context notes and the application;
- the live-testing targets, with the test accounts stored for the original, and the state-changing permission if it was given.
Because live testing is repeated, the sign-in check runs again before the retest starts. If the test account has since been removed, the retest is refused with access_check_failed.
The retest is a full pentest of that scope, with the same phases and the same limits on one run. It does not re-run the original probes one by one. It tests the scope afresh and then compares.
#What "fixed" means
A finding of the original pentest is marked fixed only when both of these hold:
- The flagged code is gone. The line that was flagged is no longer in that file at the retested version, or the file no longer exists.
- The retest did not find the flaw again.
The finding's state then becomes patched, decided by nullgap, with the reason "Fixed: the flagged code is gone at" the retested version "and the retest did not find the flaw again". A linked Jira issue follows, if status sync is on.
Either condition alone is not enough:
| The retest found it again | The flagged code | Result |
|---|---|---|
| Yes | Changed or not | Still open. A finding your team had already marked patched is reopened. |
| No | Gone | Fixed. The finding becomes patched. |
| No | Unchanged | Still open. A retest that missed a flaw has not shown it fixed. |
A finding with no stored code line to compare is treated as unchanged, so it is never closed by guesswork.
#The result
When the retest finishes, its retest field carries three counts.
{ "retest": { "fixed": 4, "stillOpen": 1, "added": 2 } }
| Count | What it counts |
|---|---|
fixed | Findings of the original that are now patched: those the retest just marked fixed, and those already marked patched before it that the retest did not find again. |
stillOpen | Findings of the original that the retest found again, plus those it did not find but whose code is unchanged. |
added | Findings the retest reported that had no record before. The fixes, or other changes since the original version, introduced or exposed them. |
Findings of the original that your team marked as an accepted risk or a false positive, and that the retest did not find again, are in none of the three counts. If the retest does produce one of them again, it is held back as described in Memory of dismissals and listed on the retest.
The retest's own findings list holds everything it reported together with the original's findings in their current state, so its reports cover the whole history. Its findingsCount is stillOpen plus added.
In the console, the retest shows a notice: "This is the retest of an earlier pentest", with the three counts in words.
#The reports
The retest writes its own three reports, titled Retest report, Executive summary after retest and Attestation letter after retest. The letter names the retested version and counts findings as they stand after the retest. See Reports.
The original pentest's reports stay available and unchanged. A report is a record of the day it was issued, so the original reports still show the findings as they stood then; the retest's reports show them as they stand after the retest.
#When a retest is refused
| Status and code | When |
|---|---|
409 not_completed | The original pentest has not finished. |
409 is_retest | The pentest you named is itself a retest. Start a new pentest to test again. |
409 retest_used | The retest for this pentest has already been used. |
400 retest_ref | ref names a different branch or tag from the original's. |
402 over_cap | The code grew past the free allowance and the retest now costs more than maxPriceUsd. Nothing was charged. |
402 payment_required | It costs an extra amount and the workspace has neither the pentest credits nor a payment method for it. |
409 repository_gone | The repository is no longer connected. |
400 access_check_failed | Live testing is part of the pentest and the sign-in no longer works. |
403 org_suspended, 402 usage_limit | As for any pentest. See What stops a pentest from starting. |
A retest is used up as soon as it starts, even if you cancel it. A retest that failed on Nullgap's side, or that Nullgap stopped, does not use it up: you can request it again.
#After the retest
- There is one retest per pentest. To test again after that, start a new pentest.
- For findings still open, read the retest's evidence: it describes the flaw as it is in the new version.
- For continuous checking instead of a one-off retest, see Scheduled pentests.
Nullgap documentation