Nullgap.dev Docs

Pentests

What one run covers on a large codebase

How a pentest lists, ranks and examines the code in scope, how a very large codebase is assessed in parts, what the Coverage section reports, and the limits that bound a single run.

One pentest does not read every file of a large codebase in depth. It lists every source file in scope, ranks them by how exposed they are, examines the most exposed ones closely, and reproduces the most serious findings. Nothing is left out without being counted: every finished pentest has a Coverage section that says how many files were indexed, how many were examined in depth, and which were not.

This page explains how that works and gives the exact limits, so you can decide how to scope your runs.

#The phases

A pentest moves through six phases. The console shows them on the pentest's Progress card, and phase on GET /pentests/{id} carries the same value.

PhaseConsole labelWhat happens
creating-environmentCreate environmentThe version under test is checked out into an isolated working copy. Every file in scope is listed and counted, and the code is divided into components. The proof environment check starts here and runs beside the next phases: dependencies are installed and the code in scope is compiled in the sandbox, so you know within minutes whether findings can be reproduced.
building-call-graphBuild call graphSource files are indexed: functions, calls between them, entry points.
mapping-attack-surfaceMap attack surfaceEvery indexed file is ranked by exposure. Static analysis reads the most exposed files and produces leads.
tracing-data-flowTrace data flowThe selected files are examined one by one by Nullgap AI, each under several headings (injection, access control, business logic, secrets and cryptography, data exposure, and others where the technology appears). This is the long step; the log reports its progress in every batch of files, with an estimate of the time left. Serious leads are reviewed independently and proven while it goes on, and show in Found so far.
testingTest candidatesDuplicates are merged. Every candidate goes to an independent review that tries to refute it. The same mistake is looked for elsewhere. The most serious survivors are reproduced in the sandbox, and tried against a running deployment if you named one.
reportingWrite reportsFindings are stored, dependency advisories are listed apart from them, and the three reports are written.

On a codebase that is assessed in parts (see Very large codebases), the phases from building-call-graph to testing repeat for each part, so the Progress card moves back and forth. The Log tab says which part is being worked on.

#What is left out, and counted

Before anything is ranked, these files are set aside. Each kind is counted, and the counts are in the Coverage section.

Left outWhat it means
Vendored code and build outputFolders such as node_modules/, vendor/, third_party/, dist/, build/, target/, and Python virtual environments.
Tests and fixturestest/, tests/, __tests__/, spec/, fixtures/, examples/, mocks/, testdata/, e2e/, and files named *.test.*, *.spec.*, *_test.go or *_test.py.
Generated filesFiles named as generated (*.pb.go, *_pb2.py, *.generated.*, *.d.ts and similar), and files whose first lines carry a standard generated-file marker.
Minified files*.min.js, bundles, and files with very long lines.
Binary contentFiles with a source extension that hold binary data.
Files over the size limitSource files larger than 512,000 bytes.
Credential filesNever opened.

The scope overrides the first two. If paths names a vendored folder or a test folder, or a folder inside one, that folder is examined like any other. Generated, minified and binary files are left out wherever they are, because they cannot be parsed in a useful way.

Test files are still read by the index when the codebase fits in one code graph, so a call from a test is seen. They are never ranked or examined.

#How files are prioritised

The ranking is computed from the code. No AI is asked, so the same repository gives the same order every time.

A file scores higher when:

  • it contains an entry point (a route, a handler, a queue consumer), or is one call or import away from one;
  • it reads request input, decides access (sign-in, sessions, roles), or reaches a place where input does damage: queries, deserialisation, file access, commands, outbound requests;
  • static analysis flagged something in it;
  • many other files import it;
  • its path suggests a sensitive area, such as auth, payment, upload or admin.

A file scores lower, at half weight, when it is build tooling, a migration, a script or test support outside a test folder. A file with a score of zero is never selected.

#Components, and how the run is shared between them

The code in scope is divided into components: the services and packages it is made of.

  • A folder that holds a manifest or build file (package.json, go.mod, Cargo.toml, pom.xml, build.gradle, pyproject.toml, setup.py, Gemfile, composer.json, mix.exs, pubspec.yaml, Package.swift, a .csproj, a Dockerfile) is a component, with everything below it that is not a component of its own.
  • Code that belongs to no such folder is grouped by top-level folder.
  • A component of more than 3,000 source files is divided along its sub-folders.
  • Beyond 3,000 components, the least exposed are gathered by the folder they sit in. A gathered group is shown with a path ending in /*.

Components are ranked by exposure: the sum of the scores of each one's 20 most exposed files, so a large folder of helpers does not outrank a small service that takes requests.

A pentest is priced by the size of the code in scope, and that price sets how much Nullgap AI work the run may do: the same code gets the same depth on every plan. That allowance is shared between components like this:

  1. A floor for every component. The two most exposed files of every component are examined first, most exposed component first.
  2. Then by exposure. One fifth of the allowance is spread evenly across components; four fifths follow exposure. A more exposed component advances faster down its own ranking.

The run works through that order as far as its allowance reaches. Files it does not reach are dropped whole, and counted. If the allowance is too small to give every component its floor, the least exposed components are the ones not examined, and the Coverage section lists them with that reason.

If your workspace also has a monthly usage limit, what is left of it for the month applies when it is smaller. To concentrate a run on one part of a large codebase, scope it to folders: see Scoping.

In a codebase with 40 ranked source files or fewer, every general heading is applied to every selected file. In a larger one, a heading is skipped for a file that has nothing relevant to it, and a file with nothing relevant to any heading does not take a place.

#Very large codebases

The index holds a code graph in memory. One graph holds up to 25,000 source files or 40 MB of source, whichever is reached first. A scope within both limits is assessed through one graph, and every examination can follow calls across components.

A larger scope is assessed in parts:

  1. Components are packed into parts that each fit one graph. Neighbouring folders are packed together, so packages that import each other tend to share a part. A component too large for one graph is spread over several parts along its sub-folders.
  2. The parts are assessed in turn, the part holding the most exposed component first. Each part has its own graph and its share of the allowance.
  3. A final pass across components. One more graph is built from each component's most exposed files and from both ends of every import that leaves a part by a relative path. Files in it that are linked to a file in another part are examined again, with the other side in view. One tenth of the allowance is kept for this pass.

Only a single folder that is itself larger than one graph, with no sub-folders to divide it along, is cut short: its most exposed files are indexed and the rest are counted as not indexed.

Custom checks and skills run with the first part, which holds the most exposed component, so the number of files a check runs on per pentest stays what it is.

#The Coverage section

Every finished pentest states what it covered. The console shows it on the pentest's page, the full report prints it under Coverage, and GET /pentests/{id} returns it as coverage.

FieldMeaning
sourceFilesSource files in scope, after the exclusions.
indexedHow many of them the index held.
examinedHow many were examined in depth.
notExaminedThe rest: sourceFiles minus examined.
excludedFiles left out by rule, by reason.
foldersThe same counts for each top-level folder.
componentsThe counts and an outcome for each component, most exposed first: examined, partly-examined, not-examined or failed, with the reason for the last two. At most 150 are listed; componentCount is the true number.
graphsHow many code graphs the run built. More than one means it was assessed in parts.
crossComponentThe final pass across components: how many files it indexed and examined, and whether it completed. null when one graph covered the scope.
gapsComponents that could not be assessed, with the reason.
resumedHow many times the run continued from a checkpoint.

The counts are of files. partly-examined is the usual outcome for a component of a large codebase: its most exposed files were examined and the rest were not.

The executive summary and the attestation letter carry the same account in one sentence: how many source files were in scope and how many were examined in depth.

In the full report the Coverage section ends with the Proof environment: whether the code could be built and run in the sandbox, per kind of dependency, with the services it needs and any language version that differs. It is what tells "not reproduced because the sandbox could not run the code" from "tried and behaved safely". The same account is on the pentest's page from the first minutes of the run. See the proof environment.

#Long runs: checkpoints and gaps

A pentest of a large codebase can run for many hours. It records its progress as it goes: after every 50 files examined, after each part, and after proof.

  • A restarted run continues. If the worker running a pentest is restarted or stops, the pentest goes back to the queue and continues from its last checkpoint. Files already examined are not examined again, findings are reported once, and you are charged once. The work a continued run does counts against the same allowance as before.
  • The time limit bounds one stretch of work. A pentest is stopped after six hours of continuous work by default. If it recorded progress in that time, it continues from its checkpoint. If it recorded none, it is marked failed, writes no report, and anything charged for it is refunded.
  • A run ends with a report inside three days. A pentest may take up to 72 hours in all. After about 58 hours it examines no further files: what it has found is reviewed, proved and reported, and the files it did not reach are counted as not examined.
  • A component that fails does not fail the run. In a run assessed in parts, a part that cannot be assessed is recorded and the run goes on to the next. The pentest then finishes as completed with gaps: status is completed, completedWithGaps is true, the findings and reports stand, and coverage.gaps names each component that was not assessed and why. Nothing can be concluded about those components. If no part could be assessed, the pentest is marked failed and refunded.

A cancelled pentest does not continue: its checkpoint is discarded.

#The limits of a single run

These are the limits in the engine. Where a limit is reached, the lowest-ranked work is what is left out.

#Reading and ranking

WhatLimit
Source files listed and rankedNo limit. Every source file in scope is counted.
Source files in one code graph25,000 files or 40 MB of source. A larger scope is assessed in parts, each with its own graph.
Build, deployment and configuration files read by the repository-level checks80 per part
Files read by static analysisThe 3,000 most exposed source files of each part, plus the configuration files above
Static-analysis leads sent on to independent review60 per part, most severe first

#Files examined in depth

There is no fixed number. The run goes as far down the order described under Components as its allowance reaches. The Coverage section states the number for your run, and the pentest's Log tab says it in one line when the run ends: Coverage: N of M source file(s) indexed, K examined in depth, R not examined.

#Independent review, variants and proof

WhatLimit
Candidates reviewed independentlyAll of them, most serious first. If a review cannot be completed (an error, or an allowance that ran out), a critical or high candidate is kept and reported as unverified; a medium or low one is not reported.
Looking for the same mistake elsewhereSearched from up to 10 confirmed findings, most serious first. Up to 5 places are reviewed for each, and 15 in each part, most exposed first. The search covers up to 4,000 non-test files of the same language in the same part.
Findings sent for proof in the sandbox8 per pentest, most serious first. Only verified findings of critical, high or medium severity with a code location are sent, mediums after every critical and high.
Proofs running at once2
Time for one proof12 minutes. Each command a probe runs is stopped after 60 seconds.
Probe attempts per finding2

Medium and low findings are not sent for proof. They are reported as verified at most. See Proof.

#Custom checks

WhatLimit
Checks per organisation20
Files one check runs on in a pentest8: the most exposed files the check applies to. In a run assessed in parts, the most exposed files of the first part.
Files tried for a check with no file patternThe 400 best-ranked files

A file that a check adds, and that was not already selected, comes after the selected files. Under an allowance it is dropped first. See Project rules and custom checks.

#Live testing

Per running application: up to 10 findings are tried, most serious first, within 500 requests and 120 seconds. See Live testing.

#Size, dependencies and time

WhatLimit
Lines of code in scope5,000,000 for a workspace billed per run. Above that the pentest is refused with custom_quote before anything is stored or charged, and the message gives the number of lines counted. Narrow paths so each run is under the limit, or contact Nullgap.
Size of the checkoutSet by the operator of the server for repositories added by address. A repository over it is refused with checkout_failed when the estimate is made, before anything is charged. Narrowing paths does not help here, because the whole repository is fetched: connect the code host instead, or ask the operator to raise the limit.
Dependency advisories listed on a pentest500, most severe first
Packages in the Supply chain tab300
One stretch of workSix hours by default. See Long runs.
Whole run72 hours. No further files are examined after about 58 hours, so the run finishes and reports within the limit.

#What is and is not covered

CoveredNot covered
Source files in scope, as far as the allowance reaches; the Coverage section gives the numbersFiles not examined in depth, which the Coverage section counts by folder and component
Build, pipeline, container and infrastructure files in scopeVendored code, tests and fixtures, unless the scope names their folder; generated, minified and binary files
Manifests and lockfiles across the whole checkoutWhether an application can be attacked through a vulnerable dependency: advisories are listed, not tested. See Advisories and findings.
A running deployment, when you name one, for the findings the code review producedBlind scanning of a deployment. Live testing confirms findings; it does not look for new ones.
Third-party services, your infrastructure as deployed, and your people and processes
Availability testing: load and denial of service

A pentest with no findings means none were identified within this scope and method. It does not show that none exist.

#Splitting a large codebase into several runs

One run over a whole large codebase gives every component a first share and the most exposed the most. If the Coverage section shows that a component you care about was only partly examined, a run scoped to it goes deeper there.

  1. Open the Coverage section of the whole-repository run and note the components with the most files not examined.
  2. Start one pentest per component you want examined further, with paths set to its folders. Each run is priced on the lines in its own scope, and that price is its whole allowance.
  3. Put shared code (an authentication library, a data layer) in a run of its own, or include its folder in the runs of the services that depend on it most.
  4. Give each run the application context that fits it. Create one application per unit on the Context page.
for unit in services/api services/billing packages/auth; do
  curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"repoId\":\"repo_5b2e8a1c9d3f\",\"ref\":\"main\",\"paths\":[\"$unit\"]}" \
    "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/pentests"
done

What to know before you split:

  • Each run is priced on the lines in its own scope and charged as agreed for your workspace. Ask for an estimate per unit first.
  • Each run has its own reports and its own attestation letter, each stating its own scope and coverage.
  • A flaw found by two runs is stored once. See One finding across runs.
  • The dependency check reads the whole checkout in every run, so the same advisories appear on each.
  • A flaw that crosses units, such as input entering in one service and doing damage in another, is easier to miss when the two are in separate runs. Say so in the context notes of both, or scope one run to both.

To repeat the scoped runs, create one schedule per unit.

Nullgap documentation