Skip to main content

Free audit program · v1.0

LLM Application Security

If you ship an app that calls an LLM (chatbot, copilot, RAG over your data), this is the controls baseline you actually need.

  • General target area
  • OWASP LLM Top 10 framework
  • 8 controls in this program
  • Cyentrix Cyentrix Trusted Author
Sign in to run → Back to library

About this program

If you ship an app that calls an LLM (chatbot, copilot, RAG over your data), this is the controls baseline you actually need.

Risks addressed

  • Critical Prompt injection takes over the LLM and exfiltrates data
  • Critical LLM agent given tool access it should not have
  • High Sensitive PII / IP fed into a 3rd-party LLM provider
  • High Insecure output handling u2014 LLM-generated XSS / SQLi

Controls (8)

  1. Input + output sanitisation around the LLM

    High

    Input and output sanitisation around Large Language Models (LLMs) involves validating, filtering, and encoding data entering and exiting the model to prevent injection attacks, data leakage, and harmful content generation. Input sanitisation removes or escapes special characters, enforces length limits,…

    How to test + evidence

    Testing procedure: Treat LLM output as untrusted. Render-safe HTML; param-bind SQL; no eval / shell of LLM output.

    Evidence to collect: Code review + escape mapping.

  2. Prompt-injection defenses (system isolation, allowlists)

    High

    Prompt-injection defenses protect AI systems (especially large language models) from malicious manipulation of input prompts that could override system instructions, extract sensitive data, or alter intended behavior. This control employs system isolation (separating user inputs from system prompts using delimiters,…

    How to test + evidence

    Testing procedure: System prompts isolated; tool calls validated; clear separation of user vs system content.

    Evidence to collect: Architecture diagram + tests.

  3. Least-privilege tool access for agents

    Critical

    This control restricts AI agents, automation tools, and service accounts to the minimum set of tools, APIs, and system functions necessary to perform their designated tasks. Access is granted based on role-specific need, with explicit allowlists defining which tools each…

    How to test + evidence

    Testing procedure: Each function-callable tool scoped to the minimum (read-only, single resource, etc.).

    Evidence to collect: Tool inventory + scope export.

  4. PII / secret redaction before sending to model

    High

    This control ensures that Personally Identifiable Information (PII), secrets (API keys, passwords, tokens), and other sensitive data are identified and removed or masked from user inputs, prompts, and data payloads before transmission to large language models or third-party AI services.…

    How to test + evidence

    Testing procedure: Outbound prompts scrubbed with DLP / regex / classifier before they leave.

    Evidence to collect: Redaction config + tests.

  5. Rate limiting + per-user quotas

    Medium

    Rate limiting restricts the number of requests a user, IP address, or session can make to an application or API within a defined time window (e.g., 100 requests per minute). Per-user quotas enforce resource consumption limits tailored to individual accounts…

    How to test + evidence

    Testing procedure: Prevent denial-of-wallet via per-user / per-IP token + request quotas.

    Evidence to collect: API gateway config.

  6. Logging of prompts + outputs to SIEM

    High

    This control requires that all prompts submitted to generative AI systems and their corresponding outputs be logged and forwarded to a centralized Security Information and Event Management (SIEM) platform in near real-time. Logs must include metadata such as user identity,…

    How to test + evidence

    Testing procedure: All prompts + completions logged with user attribution; retained per policy.

    Evidence to collect: SIEM source.

  7. Eval harness for safety + grounding

    Medium

    An evaluation harness for safety and grounding validates that AI model outputs adhere to safety policies (preventing harmful, biased, or prohibited content) and remain grounded in authoritative source data rather than hallucinating facts. This control involves automated test suites that…

    How to test + evidence

    Testing procedure: Automated evals run on every prompt / model change — red-team prompts included.

    Evidence to collect: Eval suite + last run.

  8. Model + prompt versioning

    Medium

    Model + prompt versioning establishes a formal change control and tracking mechanism for all iterations of machine learning models, large language models (LLMs), and their associated prompt templates deployed in production systems. Each model binary, configuration, and prompt must be…

    How to test + evidence

    Testing procedure: Models pinned; prompt changes go through change control.

    Evidence to collect: Versioning evidence.