Pull request review
Project rules and custom checks
Tell reviewers about your application with a threat model and project rules, and add checks that are specific to your code.
There are two ways to shape what reviews and pentests look for on this page.
- An application carries context: a threat model, your notes and your project rules. It tells reviewers what your system is and what matters in it.
- A custom check is a specific thing that must always, or never, happen in your code, written in plain words. It runs next to the built-in checks.
There is a third way, for when a sentence or two is not enough: a skill is a whole playbook with tools and its own output, and it can run by itself. See custom checks or skills.
#Applications and their threat model
An application is one application inside a repository: a name, a folder, a threat model and project rules. You manage applications in the console under Context; how that page works is on context and threat models. One folder of a repository has at most one application, and names are unique within a repository.
Every repository gets one application automatically the first time it is reviewed or pentested. It covers the whole repository (folder .), is named after the repository, and starts with an empty threat model and no rules.
#Which application a run uses
| Run | Application used |
|---|---|
| Pull request review | The repository's application whose folder is .. If there is none, the first application of the repository. |
| Pentest | The one you choose when starting it (applicationId). Otherwise the same as for a review. |
For a review, the application's folder also limits what is looked at: only changed files inside the folder are reviewed. Leave the folder at . unless you mean that.
#The threat model
| Field | In the console | What it is |
|---|---|---|
summary | Summary | What the application does and what would hurt most if it went wrong. |
assets | Assets | What needs protecting. |
actors | Actors | Who can reach the application, including machines. |
entryPoints | Entry points | Where input comes in: a name, an optional file or folder, a description. |
trustBoundaries | Trust boundaries | Where data crosses from one level of trust to another. |
controls | Controls | What already stands in an attacker's way. |
notes | Your notes | Free text from your team, passed to reviewers word for word. |
learned | Learned from feedback | Lessons added when your team marks a finding as a false positive or an accepted risk. Read-only in the console. |
You can write the threat model yourself, or press Regenerate on the application to have it drafted from the code on the default branch. Regenerating replaces the generated fields and always keeps Your notes and Learned from feedback.
#Project rules
Project rules (rules on the application) are extra instructions for reviewers of this application. Use them for facts a reviewer cannot read from the code:
All routes under /admin are behind the corporate VPN and require the staff role.
Tenant isolation is enforced by row-level security in the database, not in handlers.
User-supplied HTML is only ever rendered through renderSafeHtml().
Treat anything that reads the BILLING_* settings as sensitive.
What rules do and do not do:
- They are given to the reviewers that look for flaws and to the independent reviewer that tries to refute each candidate, together with the threat model.
- They are context. They cannot change how a finding is reported, what counts as evidence, or how sure Nullgap AI must be before reporting. A rule such as "never report SQL injection" is not a switch; to stop a specific finding being raised, triage it with a reason.
- Reviewers are given at most 6,000 characters of threat model, notes and rules together, in this order: summary, assets, actors, entry points, trust boundaries, controls, your notes, project rules. Text beyond that is cut off, and the rules come last. Keep the threat model and the rules short and specific.
- Of the lessons learned from feedback, the latest 20 are used.
#Limits
| Field | Limit |
|---|---|
name | 80 characters |
path | A relative folder inside the repository, up to 300 characters; . for the whole repository |
rules | 20,000 characters stored (see the 6,000-character note above) |
threatModel.summary | 10,000 characters |
threatModel.notes | 20,000 characters |
assets, actors, trustBoundaries, controls, learned | 200 entries each |
entryPoints | 500 entries; name 200, file 400, description 1,000 characters |
#Applications in the API
List the applications (any key):
curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/applications"
Set the project rules and your notes (a key with the write scope). Only the fields you send are changed, and each list you send replaces that list.
curl -s -X PATCH -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"rules": "All routes under /admin require the staff role.", "threatModel": {"notes": "Refunds are the most sensitive flow."}}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/applications/app_2b3c4d5e6f7a"
Add an application for a folder (a key with the admin scope). A folder or a name the repository already has an application for is refused with 409 application_exists:
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"repoId": "repo_3f9a1c2b7d4e", "name": "Payments API", "path": "services/api"}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/applications"
Have the threat model drafted again from the code:
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/applications/app_2b3c4d5e6f7a/regenerate"
The full field list is in the applications reference.
#Custom checks
A custom check describes something specific to your code, in plain words:
Every handler under /internal must call requireStaff() before it reads or changes anything.
Enabled checks run in every pull request review and every pentest of your organisation's repositories. You manage them in the console under Settings, on the Custom checks tab. Everyone can read them; admins and owners can change them, and each change is written to the audit log.
#Fields
| Field | In the console | Limit | Meaning |
|---|---|---|---|
name | Name | 80 characters, one line, unique | Shown on every finding the check produces. |
brief | What to look for | 20 to 1,500 characters | The check itself. Say what must or must not happen, in a sentence or two. |
notFinding | What is not a finding | 800 characters, optional | Cases that do not count. |
filePatterns | Files | Up to 10, each up to 200 characters, optional | Globs such as src/internal/** or *.tf. * stays inside one folder, ** crosses folders, ? is one character. A pattern without a / matches a file name anywhere. Empty means any file. |
match | Only where the file matches | 200 characters, optional | A regular expression. The check runs on a file only if the file's text matches it. |
severityGuidance | Severity guidance | 400 characters, optional | How to choose a severity. |
enabled | Run this check | false keeps the check without running it. |
An organisation can have 20 custom checks.
#How a check runs
- In a pull request review, the checks that apply to at least one changed file are run together, in one pass over the change.
- In a pentest, each check runs on up to 8 files: the most exposed files it applies to.
- A check applies to a file when the file matches one of its
filePatterns(or it has none) and the file's text matchesmatch(or it has none).
#What a check can and cannot do
- Its text says what to look for and what does not count. It cannot change how a finding is reported, what counts as evidence, or how sure Nullgap AI must be before reporting. Text that tries to is disregarded.
- Every finding from a custom check goes through the same independent review as any other finding, by a reviewer that is not shown the check's text, and serious ones are reproduced in the sandbox like the rest. See assurance levels.
- A check that describes a preference or a style rule, not a flaw an attacker can use, will mostly produce nothing. That is the intended result.
- A check cannot switch off a built-in check, and it cannot make a review pass or fail by itself. Its findings follow the severity gate like any others.
Findings carry the name of the check: finding.source is llm-custom: followed by the check's id, the rule id starts with custom/, and the first evidence entry reads Found by your custom check followed by the name. Removing a check keeps the findings it already produced.
#Refused match patterns
A match pattern is refused when its shape could take very long to run on a large file:
| Refused | Example |
|---|---|
| A repeated group that itself contains a repeat or alternatives | (a+)+ or a repeated group of alternatives |
| Lookahead and lookbehind | (?=...) and (?<=...) |
| Backreferences | \1 |
More than eight open-ended repeats (*, +, or a wide counted repeat) | |
| A counted repeat above 100 | a{500} |
| An invalid or empty expression, or one longer than 200 characters |
The rule is stricter than it strictly needs to be, so a few harmless patterns are refused too. Rewrite without the outer repeat: to ask "does this file mention X", one occurrence is enough.
#Custom checks in the API
Read the checks and the limits (any key). The answer is { "checks": [...], "limits": {...} }.
curl -s -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/custom-checks"
Add a check (a key with the admin scope):
curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "Internal routes need staff", "brief": "Every handler under /internal must call requireStaff() before it reads or changes anything.", "notFinding": "Health checks under /internal/healthz.", "filePatterns": ["src/internal/**"], "match": "router\\.(get|post)\\("}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/custom-checks"
Change one; fields you leave out keep their value:
curl -s -X PATCH -H "Authorization: Bearer $NULLGAP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"enabled": false}' \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/custom-checks/cc_6f7a8b9c0d1e"
Remove one:
curl -s -X DELETE -H "Authorization: Bearer $NULLGAP_API_KEY" \
"$NULLGAP_URL/api/orgs/$NULLGAP_ORG/custom-checks/cc_6f7a8b9c0d1e"
Errors are all 400:
| Code | Meaning |
|---|---|
invalid_custom_check | A field is over its limit, a file pattern is not a glob, or the match pattern is refused. The message says which. |
duplicate_name | Another check already has that name. Names are compared without regard to case. |
limit_reached | The organisation already has 20 checks. Remove one first. |
The full field list is in the custom checks reference.
Nullgap documentation