Pentests
Scoping a pentest
Choose the repository, version and folders a pentest covers, give the testers context, read the estimate, and know what stops a pentest from starting.
A pentest assesses one repository at one version. Before it starts you decide what is in scope, what the testers should know, and whether a running deployment is tested as well. Nothing runs, and nothing is charged, until you confirm.
#What you need first
- The admin or owner role in the console, or an API key with the
adminscope. Members can read pentests and ask for an estimate, but cannot start one. - A repository Nullgap can read: one from a connected code host, a public repository added by its address, or an uploaded archive.
- For live testing only: the staging host on your organisation's scope list, and a test account.
#The steps in the console
Open Pentests, then New pentest. The form has six steps.
| Step | What you give |
|---|---|
| Repository | The repository that holds the application. You can also add a public repository by its address or upload an archive here. |
| Version | A branch, tag or commit. A branch means its latest commit at the moment the pentest starts. |
| Scope | Folders in scope, one per line. Leave empty to test the whole repository. |
| Context | An Application context (its threat model and project rules), and free-text notes under Anything else the testers should know. |
| Live testing | Optional. See Live testing. |
| Review & start | The summary and the estimate. Start pentest opens a confirmation; Confirm and start queues the run. |
#The repository and the version
repoId names the repository. ref is the branch, tag or commit to test, up to 200 characters. If you leave ref out, the repository's default branch is used.
The version you give is recorded on the pentest and printed on every report as the revision that was tested. Use a commit or a tag when the report must point at one exact version.
#Folders in scope
paths is a list of up to 50 folders, relative to the repository root. An entry may not start with / and may not contain ... An empty list means the whole repository.
A file is in scope when its path is one of the entries or lies under one of them. Scope decides three things:
- which source files are indexed, ranked and examined;
- which build, deployment and configuration files the repository-level checks read;
- how many lines the estimate counts.
Two things are not narrowed by paths:
- The dependency check reads the manifests and lockfiles of the whole checkout. Advisories and malicious packages are reported for every lockfile found, in scope or not.
- If you give no folders and the application you chose has its own folder (anything other than
.), the run uses that folder as its scope. The estimate still counts the whole repository in that case. Setpathsyourself when you want the estimate and the scope to agree.
Narrowing the scope is also how you get deeper coverage of a large codebase. See What one run covers.
#Context notes
context is free text, up to 20,000 characters. It is given to the testers with every file they examine. Use it for what the code does not say: architecture, which parts matter most, what is deliberately out of scope.
Do not put secrets or passwords in the notes. Test accounts for live testing have their own fields.
#Applications and threat models
An application is a unit of context inside a repository: a name, a folder, a threat model and project rules. You manage them on the Context page.
The threat model records what the application protects, who can reach it, its entry points, trust boundaries and existing controls. Nullgap can generate it from the code, your team can edit it, and it gains a line under Learned from feedback each time someone marks a finding as a false positive or an accepted risk. The Project rules are extra instructions for reviewers, written by your team.
Pass applicationId to choose which application's threat model and rules the testers read. The application must belong to the repository being tested. If you pass none, the repository's default application is used: the one whose folder is ., or the first one, created empty if the repository has none.
The full report reproduces the threat model that was used.
#The estimate
The estimate counts the non-blank lines of the source files in scope. Nothing starts and nothing is charged.
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"repoId":"repo_5b2e8a1c9d3f","ref":"main","paths":["services/api"]}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/pentests/estimate"
The answer carries linesOfCode and the price for your workspace in priceUsd. A key with the read scope may ask for an estimate.
| Case | What you see |
|---|---|
| The count is above 5,000,000 lines | priceUsd is null. The codebase needs a custom quote and cannot be started as it is. Narrow the scope, or contact Nullgap. |
| The count is above 5,000,000 lines and your workspace is not billed per run | priceUsd is null and quoteWaived is true. The pentest may start. |
| The count is above 300,000 lines | The console adds a note that one run does not read every file of a codebase this size. See What one run covers. |
Files larger than 2 MB are not counted.
#Starting by API
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"repoId":"repo_5b2e8a1c9d3f","ref":"main","paths":["services/api"],"context":"Payments service. Refunds and the admin console matter most."}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/pentests"
The answer is the pentest, with status set to queued. Read it again with GET /pentests/{id} to follow status and phase. The request and response fields are in the pentests reference.
While a pentest is queued or running, an admin can cancel it with Cancel or POST /pentests/{id}/cancel. Testing stops, no report is written, and anything charged for it is refunded. What it had finished is kept for a limited time, so it can be resumed from where it stopped: see stopping and resuming a pentest.
#What stops a pentest from starting
A pentest that cannot start stores nothing and charges nothing. Every refusal is an error with a code and a message.
| Status and code | When |
|---|---|
403 insufficient_scope | The API key does not have the admin scope. |
404 not_found | repoId names no repository of this organisation, or applicationId names no application of that repository. |
400 bad_request | ref is empty or longer than 200 characters; paths is not a list of up to 50 strings, or an entry starts with / or contains ..; context is longer than 20,000 characters. |
400 bad_request | More than five live-testing targets; the same address listed twice; a target that is not an object; an address that is not a valid http or https URL or that contains credentials; sign-in steps longer than 5,000 characters; a username or password longer than 200 characters. |
400 out_of_scope | A live-testing address is a private or loopback address, or its host is not on the organisation's scope list. |
400 confirmation_required | liveStateChanging is true without disposableConfirmed: true. |
400 bad_request | liveStateChanging is true and no live-testing address was given. |
400 access_check_failed | The sign-in check failed for one of the targets. The message names the host and the reason. |
403 org_suspended | The workspace is suspended. |
402 usage_limit | The workspace has reached its monthly usage limit. |
400 custom_quote | The code in scope is above 5,000,000 lines and the workspace is billed per run. The message gives the number of lines counted. Nothing is stored or charged. Narrow paths so that each run is under the limit, or contact Nullgap for a quote. |
422 checkout_failed | The code could not be fetched: the version does not exist, access was withdrawn, or the repository is larger than this server accepts. The message says which. Nothing is stored or charged. |
402 over_cap | Scheduled runs only: the price is above the ceiling set on the schedule. |
402 payment_required | The workspace has no payment method and not enough pentest credits. |
Once the request itself is valid, the live-testing sign-in is checked first, then suspension and the usage limit, then the size and the payment. A pentest whose sign-in fails is therefore never priced or charged.
A ref that does not exist in the repository fails when the code is checked out for the estimate, before anything is stored.
#Related
Nullgap documentation