Nullgap.dev Docs

Pentests

Scoping a pentest

Choose the repository, version and folders a pentest covers, give the testers context, read the estimate, and know what stops a pentest from starting.

A pentest assesses one repository at one version. Before it starts you decide what is in scope, what the testers should know, and whether a running deployment is tested as well. Nothing runs, and nothing is charged, until you confirm.

#What you need first

#The steps in the console

Open Pentests, then New pentest. The form has six steps.

StepWhat you give
RepositoryThe repository that holds the application. You can also add a public repository by its address or upload an archive here.
VersionA branch, tag or commit. A branch means its latest commit at the moment the pentest starts.
ScopeFolders in scope, one per line. Leave empty to test the whole repository.
ContextAn Application context (its threat model and project rules), and free-text notes under Anything else the testers should know.
Live testingOptional. See Live testing.
Review & startThe summary and the estimate. Start pentest opens a confirmation; Confirm and start queues the run.

#The repository and the version

repoId names the repository. ref is the branch, tag or commit to test, up to 200 characters. If you leave ref out, the repository's default branch is used.

The version you give is recorded on the pentest and printed on every report as the revision that was tested. Use a commit or a tag when the report must point at one exact version.

#Folders in scope

paths is a list of up to 50 folders, relative to the repository root. An entry may not start with / and may not contain ... An empty list means the whole repository.

A file is in scope when its path is one of the entries or lies under one of them. Scope decides three things:

  • which source files are indexed, ranked and examined;
  • which build, deployment and configuration files the repository-level checks read;
  • how many lines the estimate counts.

Two things are not narrowed by paths:

  • The dependency check reads the manifests and lockfiles of the whole checkout. Advisories and malicious packages are reported for every lockfile found, in scope or not.
  • If you give no folders and the application you chose has its own folder (anything other than .), the run uses that folder as its scope. The estimate still counts the whole repository in that case. Set paths yourself when you want the estimate and the scope to agree.

Narrowing the scope is also how you get deeper coverage of a large codebase. See What one run covers.

#Context notes

context is free text, up to 20,000 characters. It is given to the testers with every file they examine. Use it for what the code does not say: architecture, which parts matter most, what is deliberately out of scope.

Do not put secrets or passwords in the notes. Test accounts for live testing have their own fields.

#Applications and threat models

An application is a unit of context inside a repository: a name, a folder, a threat model and project rules. You manage them on the Context page.

The threat model records what the application protects, who can reach it, its entry points, trust boundaries and existing controls. Nullgap can generate it from the code, your team can edit it, and it gains a line under Learned from feedback each time someone marks a finding as a false positive or an accepted risk. The Project rules are extra instructions for reviewers, written by your team.

Pass applicationId to choose which application's threat model and rules the testers read. The application must belong to the repository being tested. If you pass none, the repository's default application is used: the one whose folder is ., or the first one, created empty if the repository has none.

The full report reproduces the threat model that was used.

#The estimate

The estimate counts the non-blank lines of the source files in scope. Nothing starts and nothing is charged.

curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"repoId":"repo_5b2e8a1c9d3f","ref":"main","paths":["services/api"]}' \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/pentests/estimate"

The answer carries linesOfCode and the price for your workspace in priceUsd. A key with the read scope may ask for an estimate.

CaseWhat you see
The count is above 5,000,000 linespriceUsd is null. The codebase needs a custom quote and cannot be started as it is. Narrow the scope, or contact Nullgap.
The count is above 5,000,000 lines and your workspace is not billed per runpriceUsd is null and quoteWaived is true. The pentest may start.
The count is above 300,000 linesThe console adds a note that one run does not read every file of a codebase this size. See What one run covers.

Files larger than 2 MB are not counted.

#Starting by API

curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"repoId":"repo_5b2e8a1c9d3f","ref":"main","paths":["services/api"],"context":"Payments service. Refunds and the admin console matter most."}' \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/pentests"

The answer is the pentest, with status set to queued. Read it again with GET /pentests/{id} to follow status and phase. The request and response fields are in the pentests reference.

While a pentest is queued or running, an admin can cancel it with Cancel or POST /pentests/{id}/cancel. Testing stops, no report is written, and anything charged for it is refunded. What it had finished is kept for a limited time, so it can be resumed from where it stopped: see stopping and resuming a pentest.

#What stops a pentest from starting

A pentest that cannot start stores nothing and charges nothing. Every refusal is an error with a code and a message.

Status and codeWhen
403 insufficient_scopeThe API key does not have the admin scope.
404 not_foundrepoId names no repository of this organisation, or applicationId names no application of that repository.
400 bad_requestref is empty or longer than 200 characters; paths is not a list of up to 50 strings, or an entry starts with / or contains ..; context is longer than 20,000 characters.
400 bad_requestMore than five live-testing targets; the same address listed twice; a target that is not an object; an address that is not a valid http or https URL or that contains credentials; sign-in steps longer than 5,000 characters; a username or password longer than 200 characters.
400 out_of_scopeA live-testing address is a private or loopback address, or its host is not on the organisation's scope list.
400 confirmation_requiredliveStateChanging is true without disposableConfirmed: true.
400 bad_requestliveStateChanging is true and no live-testing address was given.
400 access_check_failedThe sign-in check failed for one of the targets. The message names the host and the reason.
403 org_suspendedThe workspace is suspended.
402 usage_limitThe workspace has reached its monthly usage limit.
400 custom_quoteThe code in scope is above 5,000,000 lines and the workspace is billed per run. The message gives the number of lines counted. Nothing is stored or charged. Narrow paths so that each run is under the limit, or contact Nullgap for a quote.
422 checkout_failedThe code could not be fetched: the version does not exist, access was withdrawn, or the repository is larger than this server accepts. The message says which. Nothing is stored or charged.
402 over_capScheduled runs only: the price is above the ceiling set on the schedule.
402 payment_requiredThe workspace has no payment method and not enough pentest credits.

Once the request itself is valid, the live-testing sign-in is checked first, then suspension and the usage limit, then the size and the payment. A pentest whose sign-in fails is therefore never priced or charged.

A ref that does not exist in the repository fails when the code is checked out for the estimate, before anything is stored.

Nullgap documentation