Nullgap.dev Docs

Fixes

Class fixes

Fix the same flaw in several places with one change, see what a fix would cover before you start it, and know what happens when the combined fix cannot be verified.

A team that makes a mistake once has usually made it again a few files away. A class fix repairs the same flaw in several places with one consistent change, checks it against each place, and opens one pull request that lists them all.

#What counts as the same flaw

Two open findings are offered as one fix when they are the same flaw in different places:

  • the same weakness class (for example, both are SQL injection), and
  • the same dangerous call: the function the flagged line hands the value to. db.query(...) in one file and pool.query(...) in another both count as query. The object varies from file to file; the method is what the places share.

They must also:

  • be in the same repository;
  • both be open;
  • come from the same kind of run. Findings from a pull request review are grouped only with findings from that same pull request, because the fix is written against its branch;
  • be different places. Two reports of the same line are one place;
  • not already be covered by a fix run that is queued or running;
  • not come from a fork's pull request.

A finding with no weakness id, or whose flagged line has no recognisable call, is never grouped. A dependency finding is not part of a class: see Dependency fixes.

This grouping is for fixing only. It does not merge, close or reprioritise findings. Each place stays its own finding.

#The cap

One class fix covers at most 6 places, counting the finding you start from. When more places match, the most severe are taken, and the rest are left for another run.

#Seeing what a fix would cover

GET /findings/{id}/fix-scope says what a fix started from a finding would cover. It starts nothing.

curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/findings/fnd_8f3a2c9d1b4e/fix-scope"
{
  "kind": "class",
  "findings": [
    { "id": "fnd_8f3a2c9d1b4e", "title": "SQL injection in order lookup", "severity": "high", "place": "src/orders.js:42" },
    { "id": "fnd_2b7d0e5a9c13", "title": "SQL injection in invoice lookup", "severity": "high", "place": "src/invoices.js:18" },
    { "id": "fnd_c41a6f08d2e7", "title": "SQL injection in report filter", "severity": "medium", "place": "src/reports.js:77" }
  ],
  "max": 6
}
kindMeaning
classThe same flaw in several places. findings lists them, the finding you asked about first.
findingNothing else matches. findings holds that finding alone.
dependencyA dependency finding. findings lists the other open advisories on the same package, which one upgrade clears together.

max is the cap. A key with the read scope may call this.

In the console, a finding that is part of a class lists the places under Fix it, with a Fix all N together button.

#Starting a class fix

In the console, press Fix all N together. By API, send scope: "class":

curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"scope":"class"}' \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/findings/fnd_8f3a2c9d1b4e/remediate"

The answer is one fix run with kind set to class and findingIds listing every finding it covers. Each of those findings now points at the run and shows that a fix is under way.

If no other open finding is the same flaw, the request is refused with 409 no_fix_class. Start a fix for the finding alone instead. Without scope, a fix covers only the one finding, even when a class exists.

#How a class fix is verified

  1. Each place is proven on its own. A probe is written and run for every place, to the same standard as any proof.
  2. One patch is written for every place that is covered: the same safe construct in each, through one shared helper where the project has one or the places can share one.
  3. The patch is verified against every probe that reproduced. Each place's own original payload, second payload and benign control are run against the patched code, followed by your tests.

Every place ends with one of three results, shown in the run's report and in the pull request:

resultConsole labelMeaning
verifiedVerifiedThe flaw reproduced here before the fix. After it, this place's probe no longer reproduces, a second payload is stopped too, and ordinary use still works.
coveredNot verified by reproductionThe same change covers this place, but it was not checked by reproduction here. detail says why: no working probe could be made, the code cannot run in the sandbox, or the probe had no usable second payload.
left_outLeft outA probe ran here and the code behaved safely. This place is not part of the fix, and its finding is closed as a false positive, with that evidence.

The combined fix is verified when:

  • no check failed at any place;
  • every place that has a probe passed the runs it has; and
  • at least one place passed all three of probe, variant and control.

A place with no probe does not block the others. It is reported as covered, in plain words, never as verified.

If no place could be reproduced, there is nothing to verify a patch against. A place where a probe ran and the code behaved safely is still left out. For the others, one combined fix may be proposed without proof, as a draft pull request marked "Not proven", and the run ends as unverified. Where that is switched off, no patch is written and the run ends as failed.

The checks in the run's report carry a place, as file:line, so you can see which place each result belongs to. Because the patch covers more code, its size limits are wider than for a single finding: they grow with the number of places.

#The pull request

A class fix is always delivered as a pull request, never as a suggested change. Its title ends with the number of places, and its description adds to the usual sections:

  • under What was wrong, every finding covered, each with a link;
  • under What this changes, a list of the places, each marked "verified by reproduction", "changed, not verified by reproduction" or "left out", with the reason;
  • under Proof after the fix, the three checks for each place separately.

Read the places marked "changed, not verified by reproduction" with care. The change there follows the same pattern as the verified places, but nothing was run to show it works.

#The fallback to separate runs

When the combined fix cannot be verified, Nullgap does not give up on the places. It starts a separate fix run for each place that is still open, each with its own probe, its own patch and its own checks.

  • The combined run stays on record as failed, with the reason. Its fallbackRunIds lists the runs started in its place, and its detail says how many.
  • The fallback happens only when the flaw reproduced in at least one place. If no place could be reproduced, separate runs would repeat the same proof step to the same end, so none are started.
  • Each separate run uses automation credits like any fix run. A run that cannot start for lack of credits is not started; the others are.
  • A place that turns out safe in its own run is closed as a false positive there.

So one press of Fix all N together ends in one of three ways: one pull request for the class; several pull requests, one per place; or a failed run that explains why nothing could be reproduced.

#What it spends

A class fix is one run. It uses automation credits for the work done in it, which includes a proof attempt for every place. See Automation credits.

Nullgap documentation