Pentests
Live testing
Let a pentest try its findings against a running deployment you own, inside an agreed scope, with named test identities, read-only unless you allow more.
Live testing is an optional part of a pentest. After the code has been reviewed, the findings that survived are tried against a running deployment you name, to see whether they can be shown there. It confirms findings from the code. It does not scan the deployment for new flaws, with one exception: with test identities, it compares what each identity can read and reports a read of another identity's data once it is proven (see The access matrix).
Point it at a staging deployment with test accounts. Never at production.
#What you need first
- The admin or owner role, or an API key with the
adminscope. - The host of the deployment on your organisation's scope list.
- A deployment reachable from the public internet over
httporhttps. - Throwaway test accounts, if the application needs sign-in: up to five identities per target. Two or more, with different rights or different data, if you want access-control flaws shown.
- Authority to test the system. The console asks you to confirm that you own it or have written authorisation.
#The scope list
The scope list is the set of hosts your organisation allows live testing to reach. A pentest whose address is not on it does not start. A new organisation has an empty list, so live testing is refused until an admin sets one.
#Reading and setting the list
In the console: Settings, Targets, Live-testing scope. Every member can see the list, with what each entry covers written beside it. An admin can add a host and remove one.
By API. Any member, or a key with the read scope, can read the list:
curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/scope-hosts"
{ "hosts": ["staging.shop.acme.example", "*.preview.acme.example:8443"] }
It is set with scopeHosts on the organisation's settings:
curl -s -X PATCH -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"scopeHosts":["staging.shop.acme.example","*.preview.acme.example:8443"]}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/settings"
- Setting the list needs the admin role (an API key with the
adminscope works). - The list you send replaces the whole list. Send every host each time. Send an empty list to remove them all.
- Up to 100 entries. Entries are stored in lower case.
- The answer to the settings request does not carry the list; read it back with the first call. The change is written to the audit log as
settings.updated.
#What an entry may be
| Entry | Matches |
|---|---|
staging.shop.acme.example | That host, on any port |
staging.shop.acme.example:8443 | That host on that port only |
*.preview.acme.example | Every host that ends in .preview.acme.example, at any depth, on any port. It does not match preview.acme.example itself. |
*.preview.acme.example:8443 | Every host that ends in .preview.acme.example, on port 8443 only |
An entry is a host name with at least one dot, with an optional *. in front and an optional port. Paths and schemes are not part of an entry. An address with no port in it is on its scheme's own port: 443 for https, 80 for http. So the entry shop.acme.example:443 covers https://shop.acme.example/ and not http://shop.acme.example/.
#Private addresses are refused
An entry is refused with 400 bad_request when it is not a public host name. That covers localhost, names ending in .localhost, .local or .internal, and addresses in the private, loopback, link-local and carrier-grade NAT ranges.
The same test is applied to each target address when a pentest starts (400 out_of_scope), and again to every request during the run: Nullgap resolves the host name itself, refuses the request if any address it resolves to is private, loopback or link-local (which includes cloud metadata addresses), and connects only to the address it checked. A name that points at a public address when you add it and at an internal one later is still refused.
#Targets
A pentest can test up to five running applications. Give them as dynamicTargets when you start the pentest, or in the Live testing step of the console with Also test a running deployment switched on. A single dynamicTarget object is accepted as well.
| Field | Meaning |
|---|---|
url | The address of the deployment. http or https, no credentials in the URL, up to 500 characters. The same address may not be listed twice. |
signInSteps | How to sign in, in plain words: where the sign-in form is and which fields it has. Up to 5,000 characters. Leave empty to test signed out. |
identities | Up to five test identities. Each has a label (unique, up to 60 characters, for example Admin or Tenant B user), a level (admin, user or readonly, shown as Admin, Standard user and Read-only), and either a username and password or a bearerToken. |
username, password, secondUsername, secondPassword | The older way to give one or two accounts. They still work and become the identities "Account 1" and "Account 2", at the Standard user level. Ignored when identities is given. |
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"repoId":"repo_5b2e8a1c9d3f","ref":"main","dynamicTargets":[{"url":"https://staging.shop.acme.example","signInSteps":"Post username and password as a form to /login.","identities":[{"label":"Admin","level":"admin","username":"pentest-admin@acme.example","password":"<test password>"},{"label":"Tenant B user","level":"user","username":"pentest-b@acme.example","password":"<test password>"}]}]}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/pentests"
A list with two identities of the same label (in any case), an unknown level, an identity with neither a username and password nor a token, or more than five identities is refused with 400 bad_request.
Each target is tested within its own origin: the scheme, host and port of its url. Other hosts are not reached during that target's run, even when they are on the scope list.
For each target, Nullgap signs in, follows the application's links and forms to learn its endpoints (up to 40 pages, three links deep), then tries up to 10 findings, most serious first. A finding already shown on one application is not tried on the next.
#Test identities
- Passwords and tokens are stored encrypted, for that pentest and its retest only.
- They are never returned. A pentest shows each identity's label and level (
identities: [{"label":"Admin","level":"admin","hasSecret":true}]), and, for older clients,hasCredentials: trueand, with two or more identities,hasSecondAccount: true. - They are not written to logs, evidence or reports. Recorded requests show cookie names with their values removed and authorisation headers as redacted. The password and token of every identity are also removed by their exact value wherever they would appear, whatever their length, together with any form or JSON field named like a password, secret or token.
- Put passwords in the password fields, never in the sign-in steps. Nullgap AI reads the steps to build the sign-in request with placeholders; the account's values are filled in afterwards by code.
Sign-in is one request: a form or a JSON body sent to one address on the target. A sign-in that needs a second factor, a redirect to another identity provider, or a captcha cannot be performed. Use a test account that signs in with a username and password alone, or test signed out.
If you give an account but no sign-in steps, the application is tested signed out. An identity with a token is not signed in: the token is sent with each of its requests.
#More than one identity
With two or more identities, an access-control flaw can be shown directly: a request is made as the identity that owns the data, then the same request is replayed as another identity or signed out, which should be refused. The flaw counts as shown only if the other identity gets the owner's data.
Give the identities different rights, or different data, so that the difference means something. Signed out is always compared as well; you do not add it.
#The access matrix
After the crawl, when at least one identity has signed in, Nullgap requests the read-only pages it found once as every identity and once signed out, record pages (an address with an id in it) first. Code, not Nullgap AI, decides what goes in the matrix:
- Allowed: the identity got an answer it may be entitled to (its own data, a shared page, or a less privileged identity's data read by a more privileged one).
- Denied: it got no answer, a signed-out page, or an answer without the owner's data. A status code on its own is never a leak: a page that answers
200with an error message is Denied. - Leaked: it got data that belongs to another identity, shown by code and accepted by review: the owner's response carries a distinctive value that is not on other pages and not in anything the reading identity gets elsewhere; the reading identity's replay returns that value twice, interleaved with a control of the same identity (another record, or the home page) that never does; and an independent review accepts it. Only then does it become a Proven finding.
- Not tried: the replay budget ran out, or the identity did not sign in.
A leak between identities of the same level is reported as a horizontal access flaw (CWE-639). A less privileged identity, or a signed-out request, reading a page that belongs to a more privileged one is reported as missing authorisation (CWE-285, or CWE-862 for signed out). Severity follows the gap in levels and how sensitive the data looks. The finding's evidence lists the redacted requests by identity label.
The replay only reads, whatever the state-changing permission, stays inside the run's limits, tries at most 15 pages and spends at most 150 requests per target, and stops when the target starts returning errors. The pentest page shows the matrix under Access matrix, and the full report prints it.
#The sign-in check
Before a pentest starts, Nullgap checks that each target answers and that the sign-in works. A pentest whose sign-in check fails does not start and is not charged: the request is answered with 400 access_check_failed and a message naming the host and the reason.
You can run the same check on its own first. In the console it is the Check sign-in button on each target, and the form does not let you continue until it has passed. By API:
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://staging.shop.acme.example","signInSteps":"Post username and password as a form to /login.","username":"pentest-a@acme.example","password":"<test password>"}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/pentests/verify-access"
{ "ok": true, "detail": "Signed in and opened the application (status 200)." }
What the check does:
- With no sign-in steps: one read of the address. It fails only if the application answers with a server error.
- With sign-in steps: it signs in, then reads the address as the signed-in account. It fails if the sign-in is refused, if the application then answers with a server error, or if it still shows a signed-out page.
- With several identities: each is checked in turn and every one must work. The check stops at the first that does not, and the answer names it:
{"ok": false, "failedIdentity": "Tenant B user", "detail": "..."}.
The check makes at most 12 requests per identity and gives up after 45 seconds for each. Nothing is stored. The check and its result are written to the audit log as pentest.access_checked.
detail says | What to do |
|---|---|
| Sign-in did not work | Check the steps and the test account. |
| The sign-in was accepted, but a signed-out page is still shown | The session is not being kept. Check the account, and that the steps name the real sign-in address. |
| The sign-in steps point at a different site | The sign-in address must be on the target itself. |
| The address resolves to a private or internal network | The deployment must be reachable from the public internet. |
| The application could not be reached | Check the address and that the deployment is running. |
#Read-only by default
Unless you allow more, a live test only reads. The methods it may send are GET, HEAD and OPTIONS. The one exception is the sign-in request itself: one POST to one sign-in address on the target, fixed the first time it is used.
Some flaws cannot be shown by reading. A finding whose demonstration would need a request that changes data is then recorded as not testable live, and keeps the assurance level it had from the code review.
#The state-changing permission
To allow requests that change data, send both of these when you start the pentest:
liveStateChanging: truedisposableConfirmed: true
In the console they are the Allow tests that change data switch and the checkbox confirming that every target is a disposable staging environment, whose data may be altered or lost and which shares no data with production. liveStateChanging without disposableConfirmed is refused with 400 confirmation_required.
Exactly what the permission allows:
POST,PUT,PATCHandDELETErequests to the target, so that a test may create, change or delete records there.- Only for a specific finding. Every such request is tied to the one finding it is trying to show, and a state-changing request that is not tied to a finding is refused. There is no blind fuzzing.
What it does not change:
- The target's origin, the excluded paths, the request limits and the address checks all still apply.
- The permission is for that pentest and its retest. It is recorded in the audit log as
pentest.state_changing_allowed, with the hosts.
#What is never done
These are enforced in code, on every request, whatever the testing step asks for.
- No host outside the target. A request to any origin other than the target's own is refused. Redirects are followed by hand, at most five, and every hop is checked again.
- No private or internal address. See Private addresses are refused.
- No request to these paths, or anything beneath them, with or without the state-changing permission:
/logout,/log-out,/signout,/sign-out,/logoff,/delete-account,/account/delete,/delete,/billing,/payment,/payments,/charge,/subscribe,/unsubscribe. - No state-changing method without the permission, and none that is not tied to a finding.
- No URL with credentials in it, and no scheme other than
httpandhttps. - No live testing you did not ask for. A pentest with no target makes no request to any deployment. A scheduled pentest never includes live testing.
- No lowering of a finding. A live test that does not reproduce a finding is recorded as evidence. It does not take away a proof from the sandbox or a verification from the code review.
The testing step is also instructed never to send a payload that deletes data, exhausts resources, takes the service down, or makes the database write, delete or sleep. That is an instruction to Nullgap AI, not a filter on request bodies, which is one more reason to test a staging deployment only.
#Limits on every run
These apply to each target separately.
| Limit | Value |
|---|---|
| Requests started per second | 5 |
| Requests in flight at once | 2 |
| Requests in total | 500 |
| Time in total | 120 seconds |
| Time for one request | 15 seconds |
| Response bytes read in total | 50 MiB |
| Largest single response read | 5 MiB; anything longer is cut |
| Stop on a struggling target | After 5 server errors or timeouts in a row, the run against that target stops |
| Access matrix | At most 15 pages and 150 requests, inside the totals above |
When a limit is reached, the remaining findings for that target are recorded as not tested live.
#Finding the requests in your logs
Every request carries a User-Agent that starts with Nullgap-DAST/ and an x-nullgap-run header with the id of the run. The pentest's Log tab names the run id when live testing starts.
#What you see afterwards
- The pentest's Live testing figure shows On with the addresses, and
read-onlyorchanges data. - Each finding that was tried has an evidence entry of kind
dynamic, with the verdict (confirmed-live,not-reproduced-liveorcould-not-test), the reason, and the redacted requests it rests on. - A finding is marked Proven from a live test only when the demonstration met the live proof standard. Otherwise the result is attached as evidence and the assurance level does not change.
- With test identities, the pentest page lists them (labels and levels, never secrets) and shows the Access matrix: one row per page, one column per identity and signed out. A Leaked cell links to its proven finding.
- If live testing could not complete for a target, the Log says so and the pentest finishes on the code alone.
- The reports state whether a deployed instance was exercised.
Nullgap documentation