Fixes
Verified fixes
How a fix run reproduces a finding, writes a patch and checks it, what the pull request contains, what happens when the finding cannot be reproduced, what automations do, and what is refused.
A fix run takes one finding, reproduces it, writes the smallest change that repairs it, and checks that change before anyone sees it. Only a fix that passed its checks is proposed as verified, as a pull request or as a suggested change on the pull request under review. When the finding cannot be reproduced at all, a fix can still be proposed as a draft marked "Not proven", which is never counted as a verified fix. Nullgap never merges anything.
#What you need first
- Any member can start a fix in the console. By API, a key with the
writeoradminscope. - The finding must be
open, and its repository must come from a connected code host. A public repository added by address, or an uploaded archive, is read-only: no pull request can be opened on it. - Automation credits, except for a dependency fix.
- For the tests check: a test command in the repository's
.nullgap/config.json, astestCommand.
#Starting a fix
| Where | How |
|---|---|
| The console | Open the finding. Under Fix it, press Remediate. |
| A pull request | Reply @nullgap fix in the finding's thread. |
| The API | POST /findings/{id}/remediate |
| Automatically | An automation starts one for each new finding that matches its rules. |
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/findings/fnd_8f3a2c9d1b4e/remediate"
The body is optional. scope: "class" fixes the same flaw in several places. as: "pull_request" always opens a pull request, even where a suggested change would do.
The answer is the fix run. If a run for that finding is already queued or running, that run is returned and no second one starts. Follow it with GET /automation-runs/{id}:
curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/automation-runs/run_0d4b7e2a9c61"
#What a fix run does
- Checks the code is still there. If the flagged file or line is gone, the run ends as
no_fix_needed. - Reproduces the flaw with a probe in the sandbox, to the same standard as proof. If the probe runs and the code behaves safely, the run ends as
false_positiveand the finding is closed as a false positive. If the flaw cannot be reproduced for another reason, nothing was learned about the code, and no fix can be verified. The run then either proposes a fix that is not proven, or, where that is switched off, ends asfailed. - Writes a patch in a disposable copy of the project. Your repository is not touched.
- Verifies the patch with the checks below, on a fresh checkout.
- Delivers it, only if it is verified. A fix that is not proven is delivered only as a draft pull request that says so.
The code the fix is written against depends on where the finding came from. For a finding from a review of a pull request that is still open, it is that pull request's branch. For anything else it is the repository's default branch.
outcome | Meaning |
|---|---|
fixed | A patch exists and passed verification. It was delivered. |
unverified | The flaw could not be reproduced, so nothing was proven. A patch was written anyway, passed the checks that need no probe, and was opened as a draft pull request marked "Not proven". It is not a verified fix. |
no_fix_needed | The flagged code is no longer in this version. |
false_positive | A probe exercised the flagged code and it behaved safely. |
failed | Nothing was proposed: a patch that was refused or did not pass, a flaw that could not be reproduced where unproven fixes are switched off, an error. detail says which. |
The run also carries verified: true only when the outcome is fixed. Use it rather than the name of the outcome to tell a proven fix from everything else.
#The checks
Every check is recorded by name with one of three statuses: passed, failed, or not_run. Each has a detail sentence saying what was seen, or why it could not run.
| Check | Console label | What passes |
|---|---|---|
scope | Stays on the code the finding is about | Reading the diff finds none of the changes listed under What is refused. Checked first, before anything is run. |
applies | Applies to a fresh checkout | The patch applies cleanly to a new checkout of the same version. |
probe | Original payload no longer reproduces | The probe that reproduced the flaw, exactly as it was when it reproduced, now reports safe behaviour. |
variant | A second payload is stopped too | A different input for the same flaw is stopped as well. This is what shows the fix is not a mask that blocks one literal payload. |
control | Ordinary use still works | The same call with ordinary input still gets its normal result. This is what shows the fix did not delete or break the feature. |
tests | Project tests | Your test command exits successfully on the patched code. It may run for up to ten minutes. |
A dependency fix has two checks of its own, lockfile and resolved, in place of the three probe runs. See Dependency fixes.
#not_run is never a pass
A check that is not_run was not made. It is never counted as passed.
A fix is verified only when no check failed, the patch applied, and probe, variant and control all passed. So:
- If the probe has no usable second payload,
variantisnot_runand the fix is not verified. It is not proposed. - If the probe errors instead of reporting safe behaviour,
probeisfailed: the patch may have broken the code the probe runs. - If your tests fail, the fix is not verified, even when they were already failing before the patch. The
detailthen says they also fail without it.
The one check that may be not_run on a verified fix is tests, when the repository has no test command. That is never hidden: it is listed under Not verified in the run's report and in the pull request.
#Attempts
A run makes up to three attempts at the patch. Each attempt after the first is told exactly what the previous one failed on. If nothing failed but a check could not run, no further attempt is made, because another patch would not change that.
Every attempt is recorded in report.attempts: its number, whether it was verified, the reason it was not, and the files it changed.
#When the flaw cannot be reproduced
Sometimes no probe is obtained, for a reason that says nothing about the code:
- Nullgap AI declined to write the probe for this kind of flaw;
- the time or effort limit for one proof was reached;
- no working probe came out of the attempts;
- the code cannot run in the sandbox as it is.
Before giving up on a probe that was declined, the same request is put, unchanged, to a second Nullgap AI model. The request is never reworded to get a different answer. The finding's evidence records that the step was declined and that another model was asked.
If there is still no probe, the run writes the fix anyway and checks it with everything that does not need one:
| Check | Console label | What passes |
|---|---|---|
scope | Stays on the code the finding is about | The same rules as for a verified fix. See What is refused. |
applies | Applies to a fresh checkout | The patch applies cleanly. |
build | Changed files still parse | The changed files are read by their language's own parser. not_run when the sandbox has no parser for that file type. |
tests | Project tests | Your test command passes on the patched code. not_run when the repository has none. |
review | Independent review of the change | A second Nullgap AI model, not the one that wrote the change, reads the diff and the patched code and accepts that it closes the described flaw without changing other behaviour. |
probe, variant and control are recorded as not_run, with the reason no probe exists.
The run ends as unverified only when none of these failed and the review accepted the change. If the review rejects it, or your tests fail, the next attempt is told why. After three attempts the run ends as failed, with the reason, and nothing is opened.
A probe that ran and saw the code behave safely is different: that is evidence against the finding. No patch is written and the run ends as false_positive, as before.
#How it is delivered
- As a draft pull request, whose title starts with
[Not proven]. A draft cannot be merged until someone marks it ready. - The description opens with a notice: that the change was not proven by reproducing the flaw, why the flaw was not reproduced, what was checked instead (tests passed or not run, the review accepted), and that a person must review it before merging.
- Never as a suggested change, and never merged for you.
| Code host | What you get |
|---|---|
| GitHub, GitHub Enterprise Server | A draft pull request. Where the repository has no draft pull requests, an ordinary one with the same title and description. |
| GitLab | A draft merge request: its title starts Draft: [Not proven]. |
| Bitbucket Cloud | A draft pull request. Where the workspace does not take drafts, an ordinary one with the same title and description. |
| Azure DevOps | A draft pull request. |
On the run, prDraft says which you got: true for a draft, false for an ordinary pull request.
#What it is never counted as
- The finding's status does not change. It does not become proven and it does not become fixed.
- The run's
verifiedisfalse, and so isverifiedin theremediation.completedevent. - In the console it reads Fix proposed, not proven, in amber, never in the green of a verified fix.
- In the findings export, the
fix_provencolumn saysno.
The run's report says the same in notProven: the reason, what the review said, and whether the reviewer was a different model from the author (independentReview).
#Switching it off
An admin can switch this off for the whole organisation: Settings, General, Propose a fix when a finding cannot be reproduced. By API it is unverifiedFixes in the settings:
curl -s -X PATCH -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"unverifiedFixes":false}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/settings"
It is on unless you switch it off. With it off, a run that cannot reproduce the flaw ends as failed and only proven fixes are ever opened. The change is written to the audit log as settings.unverified_fixes, with the new value.
An automation does not deliver a fix that is not proven unless it says so, whatever this setting is.
#Reading the run
{
"id": "run_0d4b7e2a9c61",
"findingId": "fnd_8f3a2c9d1b4e",
"status": "completed",
"outcome": "fixed",
"verified": true,
"kind": "finding",
"prUrl": "https://github.com/acme/shop/pull/31",
"report": {
"kind": "finding",
"places": [ { "findingId": "fnd_8f3a2c9d1b4e", "place": "src/orders.js:42", "result": "verified" } ],
"checks": [
{ "name": "scope", "status": "passed" },
{ "name": "applies", "status": "passed" },
{ "name": "probe", "status": "passed" },
{ "name": "variant", "status": "passed" },
{ "name": "control", "status": "passed" },
{ "name": "tests", "status": "not_run" }
],
"attempts": [ { "attempt": 1, "verified": true, "files": ["src/orders.js"] } ],
"tests": { "command": null, "result": "not_run" },
"notVerified": [ "The project's tests were not run: it has no test command configured." ],
"refused": []
}
}
The sample is abridged: each check and place also has its detail. In the console the same report is under What the fix run checked on the finding, and on the Automations page.
#What a fix pull request contains
The pull request is opened from a branch named nullgap/fix- followed by part of the finding id. A later run for the same finding gets a branch of its own: an earlier branch is never reused or overwritten.
The description is written so a reviewer can decide from it alone:
| Section | Content |
|---|---|
| What was wrong | The finding, its severity and location, a link to it, and its description |
| What this changes | What the change does, in a sentence or two |
| Proof before the fix | The probe command that reproduced the flaw, and what it exited with |
| Proof after the fix | The probe, variant and control checks, each with its status and detail |
| Tests | The test command and its result, or that none is configured |
| Not verified | Every check that could not be made, in its own words. Earlier attempts and why each was rejected. |
| How to revert | How to undo the change after merging |
It ends by naming the commit and branch the fix was verified against, and with: "Review before merging. Nullgap never merges, never force-pushes, and will not push to this branch again."
When the fix touches several files, each file is its own commit, so one part can be read and reverted without the others. For a finding from an open pull request, that pull request's author is asked to review.
#Suggested change or pull request
A small fix for a finding on an open pull request can be posted on that pull request as a suggested change, which the author accepts with one click. Everything else becomes a pull request.
A fix is posted as a suggested change only when all of these hold:
- the finding came from the review of a pull request that is still open;
- the run fixes one finding (not a class, not a dependency), and was not started with
as: "pull_request"; - posting comments on pull requests is switched on for the organisation;
- the patch is one change in one file, and every line it replaces is a line that pull request added or changed;
- it replaces at most 10 lines and puts at most 20 in their place;
- nobody has pushed to the pull request since the fix was verified. If the head has moved, the verification belongs to older code, so no suggestion is posted and a pull request is opened.
| Code host | What you get |
|---|---|
| GitHub, GitHub Enterprise Server, GitLab | The suggested change alone. The run shows "Posted as a suggested change" and has no pull request. |
| Bitbucket Cloud, Azure DevOps | A pull request, and the same change shown as a patch in a comment on the reviewed line. |
The comment that carries the suggestion says what was run before it was posted, with the status of each check.
#Automation credits
A fix run uses automation credits for the work Nullgap AI does in it.
- When a run starts it sets aside the most it may be charged: the smaller of what is left and its per-run limit (the automation's limit, or a fixed limit for runs without one). Credits set aside by runs under way cannot be counted on by another run, so fix runs going at the same time never share the same balance. What a run did not use is released when it ends.
- Credits are taken when the run ends, whether or not it produced a fix. A run that reproduced the flaw and then could not verify a patch has still done the work. A run is never charged more than it set aside.
- A run that fails because of an error on Nullgap's side is not charged, and anything it took is given back.
- A run you cancel is charged only for the work done before it stopped. A run stopped by Nullgap (for example by support, or because the workspace was suspended) is not charged.
- A run that did no AI work uses none. A dependency fix never does.
- The balance never goes below zero. With no credits left, or with everything left set aside by runs under way, a fix is refused with
402no_credits. creditsUsedon the run says what it took. Your balance and its history are on the Billing page. See Usage.
#Cancel a fix run
An admin can stop a queued or running fix run with Cancel fix run on its page, or with POST /api/orgs/:org/automation-runs/:id/cancel. The run stops at its next step and is charged for the work done before then, never more than its per-run limit.
#Automations
An automation starts a fix run by itself for each new finding that matches its rules. Admins manage them on the Automations page, or with the automations routes. The page itself, who is told how a run ended and the run history are described on automations.
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name":"Fix serious findings on main","enabled":true,"trigger":{"severities":["critical","high"],"repoIds":[]},"environment":{"branches":["main"],"maxCreditsPerRun":null},"notifier":{"slack":true,"emails":["dana@acme.example"]}}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/automations"
| Field | Meaning |
|---|---|
name | Required. Up to 80 characters. |
enabled | An automation created without it is switched off. |
trigger.severities | Which severities start a run. Default: critical and high. An empty list means every severity. |
trigger.repoIds | Which repositories. An empty list means every repository. |
environment.branches | Up to two branches a fix may target. An empty list means any branch. |
environment.maxCreditsPerRun | A cap on the credits one run may use. null means no cap. |
allowUnverifiedFixes | Whether a run may open a draft pull request for a fix that is not proven. Default false. |
notifier | Who is told when a run finishes: slack, and up to 20 emails, each the address of a member of the workspace. |
How automations behave:
- They fire on new findings only, from pull request reviews and pentests. A finding that a later run finds again does not fire one.
- Each finding gets at most one automatic run. You can still start more by hand.
- If several automations match a finding, one of them runs.
- A finding from a fork's pull request never gets one.
- With no credits left, the run is skipped and the skip is written to the audit log as
automation.skipped. - If the fix would target a branch outside
environment.branches, the run fails and says so. - When the finding cannot be reproduced, the run ends as
failedand opens nothing, unless the automation hasallowUnverifiedFixesset. Nobody asked for that run, so it does not open unproven pull requests on its own. With the field set, and the organisation setting on, it may open one, as a draft. Changing the field is written to the audit log withautomation.updated.
An automatic fix goes through exactly the same checks, and is delivered the same way. Nothing is merged for you.
#What is refused
#Fixes that are not started
| Status and code | When |
|---|---|
409 fork_pr | The finding came from a fork's pull request. Nullgap does not open fix pull requests for forks. |
409 read_only_repository | The code was added by address or upload. Connect its code host to get fix pull requests. |
402 no_credits | No automation credits are left. |
409 no_fix_class | scope: "class" was asked for and no other open finding is the same flaw. |
#Patches that are refused before they are run
The scope check reads the diff. A patch that contains any of the following is refused without being executed, and the reason is recorded in report.refused.
| Refused | Why |
|---|---|
| A change to build, pipeline or Nullgap configuration | See the protected paths below. |
| A test switched off, a test file deleted, or more test cases and assertions removed than added | The suite must not be weakened to make a fix pass. New tests are welcome. |
An error caught and ignored: an empty catch, an except that does nothing, a discarded promise rejection, a command whose failure is hidden | The cause must be fixed, not silenced. |
An allow-list widened to everything: any origin, a wildcard host list, 0.0.0.0/0, an access check removed for everyone | A fix must not make its probe pass by allowing more. |
| A security setting lowered: certificate checking off, CSRF protection off, a cookie protection off, debug mode on, permissions opened to everyone, token verification weakened, a browser protection off, a dangerous construct or a weak algorithm introduced | A fix must not trade one weakness for another. |
| A comment that tells a checker to ignore the code | Suppressing a warning is not a fix. |
| A change to a file that is not where the finding is, not on its data-flow path, and not a helper the fixed code now uses | The change must stay on the code the finding is about. |
| A change to a dependency manifest or a lockfile | A code fix must not add, remove or re-pin dependencies. That is a dependency fix. |
| A source file deleted, or code only removed from a file with none added | Removing the feature is not a fix Nullgap proposes on its own. |
| A patch that is too large: more than 6 files or 400 changed lines for one finding | A fix that size needs a person. |
These are simple pattern checks, and they are strict on purpose. They can refuse a legitimate fix, for example one that really needs a new dependency. Such a fix goes to a person instead of being opened as verified.
#Protected paths
An automated change never touches these, by a pull request or by a suggested change. They run with your repository's secrets or tell Nullgap how to behave.
.github/, .gitlab-ci.yml, .gitlab/, .circleci/, Jenkinsfile, azure-pipelines.yml, .azure-pipelines/, bitbucket-pipelines.yml, .buildkite/, .drone.yml, .travis.yml, .husky/, .nullgap/, .git/, .gitmodules
This is checked again at the moment of delivery, for the whole diff and for each commit. A fix that passed verification and would still touch one of them is not opened, and the run says why.
#Fixes that are verified but not delivered
- The branch moved. If the target branch changed after verification and the patch no longer applies, no pull request is opened. Start the fix again to rebuild it on the current code.
- A protected path, as above.
#What Nullgap never does with a fix
- It never merges.
- It never force-pushes, and never pushes to a fix branch a second time.
- It never proposes a patch as verified that did not pass verification. A patch that failed a check stays on the finding, marked Not verified, for a person to read.
- It never presents a fix that was not proven as a verified one. Such a fix is a draft, labelled in its title and at the top of its description.
#Related
Nullgap documentation