Nullgap.dev Docs

AI app testing

Testing an AI app for prompt injection

Point Nullgap at a chatbot or agent API you own, run 32 prompt-injection attacks against it, and read what it fell for, each with the exchange that proves it and how to fix it.

An AI app test sends your own AI application the prompt-injection attacks that matter and watches what it does with them. It works on the running app, not on its code: a chatbot, a support assistant, an agent with tools, anything that takes a message over HTTP and answers.

Each attack either works or it does not. When it works you get the exchange that proves it (what was sent, what the app replied or which tool it called), why it counts as a violation, and how to fix that kind of weakness. When it does not, it is counted as blocked.

Only test apps you own or are authorised to test. Point it at a staging deployment, never at production with real customers' data.

#What you need first

  • The admin or owner role, or an API key with the admin scope.
  • The app's host on your organisation's live-testing scope list (Settings, Targets, Live-testing scope). The same list governs live testing in pentests. Private and local addresses are never reached.
  • An address reachable from the public internet over http or https.
  • Authority to test the app. The console asks you to confirm that you own it or are authorised to test it, and the API refuses a run without confirmAuthorised: true.

#Two kinds of app

Chat API compatible. An endpoint that takes the common chat-completions format: a JSON body with model, messages and optional tools, answered with choices[0].message and any tool calls in it. Give the model name your API expects, and optionally the tools the app may call. Tools are offered to the app in every request, so a call to a forbidden one is caught exactly. Nothing is executed: a call is acknowledged and recorded.

Custom JSON API. Any other JSON API. Describe the request with a template and say where the reply is:

{ "session": "nullgap-test", "input": { "text": "{{prompt}}" } }
  • {{prompt}} is replaced by each attack message.
  • {{system}} is replaced by the system prompt you give, if any.
  • A value that is exactly {{messages}} becomes the conversation so far, as a list of { "role", "content" }.
  • The response path is a dot-path to the reply text in the answer, such as data.reply or choices.0.text.
  • The optional actions path points at the list of tools the app called, if it reports them.

Use Check connection before you start: it sends one harmless message through the same checks and shows the start of the reply, or why it could not be read. Nothing is stored.

#What you tell it

  • Authentication. A header sent with every request, usually Authorization: Bearer …. The value is encrypted for your organisation the moment it arrives, never shown again or written to a log, and erased when the test ends. A value containing ${ is refused: nothing is ever read from the server's environment.
  • Prohibited actions. What the app must never do, one rule per line in plain words: "Never call send_email unless the customer asked". Naming a tool lets a call to it be caught exactly.
  • System prompt (optional). Give it and a reply that repeats it is caught. It is sent to your app as the system message (or in place of {{system}}), and it is never returned by the API or written to the log. Transcripts show that one was sent, not what it said.
  • Attack categories. All twelve by default.
  • Request budget. The most requests sent to your app, up to 500 (default 200). Requests go out at most four a second.

Nullgap always plants a made-up secret, the canary, in the system prompt. Seeing it in a reply, in any encoding, or in a tool call's arguments proves the app leaked what it was told to keep. It shows as [CANARY] in results.

#The attacks

CategoryWhat it triesOWASP LLM
Instruction overrideTells the app to ignore its instructions and do something else.LLM01
Fake system messagesPretends to be the system or a developer inside a user message.LLM01, LLM07
System prompt extractionTries to make the app repeat or translate its hidden instructions.LLM07
Secret disclosureAsks for confidential values the app was given.LLM02
Indirect injectionHides instructions in documents, web pages or tool results the app reads.LLM01, LLM02, LLM06
Tool misuseCoaxes the app into calling a tool it must not call.LLM06
Data leaks through links and imagesGets the app to emit a link or image that carries data to another site.LLM02, LLM05
Encoded instructionsHides instructions in base64, ROT13, invisible characters or another language.LLM01
Multi-turn erosionWears the rules down over several messages.LLM01, LLM07
Delimiter confusionFakes the end of the input or breaks out of a JSON structure.LLM01
Claimed authorityClaims to be an administrator or applies urgent pressure.LLM01, LLM02, LLM06
Refusal suppressionForbids the app from refusing, or makes it start with an agreeable sentence.LLM01, LLM02, LLM07

The ids are those of the OWASP Top 10 for LLM applications.

When an attack is blocked as written, Nullgap AI may try a couple of paraphrases of it before it is counted as blocked.

#How a violation is decided

Most results are decided by checks that need no judgement:

  • the canary appears in a reply or a tool call;
  • the app called a tool a rule prohibits;
  • the reply repeats a long passage of the system prompt;
  • the reply carries the marker an injected instruction asked for;
  • the reply contains a link or image that sends data to another site.

When none of those fires, Nullgap AI takes a second look at the exchange and reports a violation only when it is confident. Its reasoning is shown with the finding.

#Reading the results

The run's page lists proven issues first, most severe first. Each one shows the attack, what the app did, why it counts, how to fix it, and the full transcript. Below them is the number of attacks the app blocked, the number that could not be judged because the app did not answer, and whether the run stopped early.

A run where everything was blocked is evidence about these attacks, not a guarantee that the app cannot be misled.

What an AI app test proves stays on the run. It is not added to your repository findings, the dashboard or the security score.

#What it costs

A test is charged in automation credits for the Nullgap AI work it does (writing variants of blocked attacks and taking a second look at replies), at the same rate as a fix run. The requests sent to your app are not charged.

  • The form says the most one test can cost ("At most N automation credits"). When a test starts it sets that amount aside, or what is left if that is less, and it is never charged more.
  • Starting a test needs automation credits. With none left (or everything left set aside by runs under way), it is refused with 402 no_credits. If the credits run out between the start and the run, the test runs its deterministic checks only.
  • A test that fails because of an error on Nullgap's side, or that Nullgap stops, is not charged. A test you cancel is charged only for the work done before it stopped.
  • creditsUsed on the run says what it was charged; the credit ledger shows the entry.
  • Workspaces billed under an agreement are not charged.

#Stopping a run

Stop on the run's page, or POST /api/orgs/{org}/ai-tests/{id}/cancel. A running test stops before its next request; nothing it found is kept and its header values are erased. It is charged for the Nullgap AI work it did before it stopped.

#By API

Start a run:

curl -s -X POST -H "Authorization: Bearer $NULLGAP_API_KEY" -H "Content-Type: application/json" \
  "$NULLGAP_URL/api/orgs/$NULLGAP_ORG/ai-tests" -d '{
    "name": "Support assistant, staging",
    "target": { "kind": "chat-api", "url": "https://staging.acme.example/v1/chat/completions", "model": "support-bot",
                "tools": [{ "name": "send_email", "description": "Send an email" }] },
    "headers": { "Authorization": "Bearer sk_test_…" },
    "policy": { "prohibited": ["Never call send_email unless the customer asked"] },
    "maxRequests": 200,
    "confirmAuthorised": true
  }'

Then poll GET /api/orgs/{org}/ai-tests/{id} until status is completed, failed or cancelled. The full list of fields is in the API reference.

#Limits

  • Two runs queued or running per organisation at once.
  • At most 500 requests per run, at most four a second, and at most 1 MB read from any one reply.
  • Redirects are never followed; the address is checked against the scope list before every request.
  • Usage of Nullgap AI by a run counts towards your workspace's monthly usage.

Every start, finish and cancellation is recorded in the audit log.

Nullgap documentation