Prompt Governance Framework: Securing AI System Prompts

A prompt governance framework treats the system prompt as a controlled asset, not config text. It covers four things: who can change the prompt and how (change control), how you stop injection and leakage (security), what you test before and after each change (assurance), and what you log so an auditor can reconstruct any answer the model gave (auditability). Most teams have none of this. The system prompt sits in a repo, anyone with merge rights edits it, nobody tests it against attack inputs, and there's no record of which prompt version produced which output. That's the gap this piece closes.

The reason it matters now: OWASP ranks prompt injection as the number-one risk for LLM applications, and system prompt leakage made its 2025 list as a separate entry. The two are linked. A successful injection is often how an attacker pulls your system prompt out in the first place. If your prompt holds security rules, access logic, or anything you'd rather attackers didn't see, treating it as a control point isn't optional.

What is a prompt governance framework?

It's the set of rules, controls, and records that govern how your system prompts are written, changed, secured, tested, and audited. Think of it the way you already think about production config or access policy: a thing with an owner, a change process, and an audit trail.

A workable framework answers five questions:

Question	Control it maps to
Who can change the system prompt?	Access control, approval workflow
How do we know a change is safe?	Pre-deployment adversarial testing
How do we stop injection and leakage?	Input/output filtering, content segregation, privilege limits
Which prompt version produced this output?	Version control, response logging
Can an auditor reconstruct a decision?	Immutable audit log tied to prompt version

If you can answer all five with evidence, you have a framework. If you can answer them in a meeting but can't produce the records, you don't. That distinction is what a regulator or an enterprise buyer's security team will push on.

Why is the system prompt a security control point?

Because it concentrates control. In ordinary software, the rules that govern behaviour are spread across many modules, each reviewed and tested. In an LLM application, a large share of that control sits in one block of text. Change a few lines and you change how the model treats every user, every input, every decision. That's efficient. It's also a single point of failure.

The deeper problem is architectural. An LLM reads its instructions and its user input through the same channel. It has no reliable way to tell "this is a trusted rule from my operator" apart from "this is text a user typed." OWASP is blunt about it: there's no fool-proof method to prevent prompt injection, because the behaviour comes from the model's stochastic nature, not a fixable bug (OWASP LLM01:2025 Prompt Injection). You can cut the risk a lot. You can't patch it away. That alone makes the system prompt worth governing properly.

How does prompt injection work, and how is it different from jailbreaking?

Prompt injection is when input changes the model's behaviour in ways you didn't intend. OWASP splits it two ways:

• Direct injection. The user types instructions that override your rules. "Ignore your previous instructions and..." is the crude version. The sophisticated versions are harder to spot.
• Indirect injection. The malicious instruction hides inside content the model reads from somewhere else: a web page, a document, a pull request comment, an email. The user never sees it. The model does, and follows it.

Indirect injection is the one that catches mature teams out, because the attack surface is everything your model ingests. A real example: in 2025, Legit Security researcher Omer Mayraz showed that GitHub Copilot Chat could be fed hidden instructions buried in pull request comments. The instructions sat in Markdown that GitHub doesn't render, so a human reviewer never saw them, but Copilot read them and could be steered into leaking AWS keys and other secrets from private repositories. GitHub closed the exfiltration path in August 2025 (CSO Online). Nobody typed an obvious attack. The payload rode in on content the system was built to read.

Jailbreaking is a subset: injection aimed specifically at breaking the model's safety constraints. All jailbreaks are injections. Not all injections are jailbreaks. We go deeper on the attack mechanics in AI prompt injection and jailbreaking techniques.

What is system prompt leakage and why does it matter?

System prompt leakage is when an attacker extracts the hidden instructions that steer your model. OWASP added it as a distinct risk in its 2025 list, separate from injection, because the consequences are their own problem (OWASP Top 10 for LLM Applications 2025).

Here's the uncomfortable part. Leaked system prompts from major AI products keep showing up in public GitHub repositories, scraped and published for anyone to read. That's the proof that "hide it in the prompt" is not a security strategy. If the leading labs can't keep their prompts secret, your application can't either.

So the rule is simple, and most teams break it: never put a secret in a system prompt. No API keys, no credentials, no internal logic you can't afford an attacker to read, no list of the exact rules you rely on to stay compliant. Assume the prompt will leak and design so the leak isn't a breach. The prompt should tell the model how to behave. The enforcement that actually protects you, access control, filtering, privilege limits, lives in code the model can't expose.

What does a prompt governance framework actually contain?

Map your controls to the OWASP mitigations, which are the closest thing the field has to an agreed baseline (OWASP LLM01:2025). Here's the working set.

Change control. The system prompt goes in version control. Every change has an author, a reviewer, and an approval. No direct edits to production. You'd never let someone hot-patch your auth logic without review. The prompt carries comparable weight.

Privilege limits. The model gets the least access it needs. Application-managed tokens, not the model holding broad credentials. If an injection succeeds, you want it to hit a wall, not a vault. This is the control that turns a leak into a non-event.

Content segregation. Mark untrusted external content clearly so the model treats it as data, not instructions. Indirect injection lives here. Anything the model reads from outside, web pages, documents, tickets, gets quarantined and labelled.

Input and output filtering. Semantic and string-level checks on the way in and the way out. Define the output format and validate it with deterministic code, so a model that's been steered off-script produces something your code rejects rather than something your user trusts.

Human approval for high-risk actions. A human in the loop on anything consequential: moving money, changing records, sending external communications. The model proposes. A person commits.

Adversarial testing. Test the prompt against attack inputs before it ships and on a schedule after. Treat the model as an untrusted user and try to break it. This is the step almost everyone skips, and it's the one that catches the leak before an attacker does. Our AI red team assessment covers how to run this properly.

Audit logging. Log the prompt version alongside the output. When something goes wrong, or a regulator asks, you need to reconstruct which exact prompt produced which exact answer. A prompt change with no record is an unverifiable system. See our AI compliance audit guide for what auditable looks like end to end.

How does this connect to regulation?

Directly, and the clock is running. NIST's Generative AI Profile (NIST AI 600-1, published July 2024) names direct and indirect prompt injection as information-security risks and maps them to its Measure and Manage functions: adversarial testing, input validation, output filtering, incident response (NIST AI RMF). That's the same control set above, in regulatory language.

The EU AI Act goes further for anyone running high-risk systems. Article 15 requires an appropriate level of accuracy, robustness, and cybersecurity, including measures against data poisoning, model poisoning, and adversarial inputs designed to make the model err (EU AI Act Article 15). The rules for high-risk systems in Annex III apply from 2 August 2026 (EU AI Act implementation timeline). Prompt injection is exactly the kind of adversarial input Article 15 has in mind. If your high-risk system has no prompt governance, you have a compliance gap with a dated deadline.

For teams running autonomous agents, the stakes climb again, because an agent that can act on a successful injection does real damage. We cover that in AI agent governance for autonomous systems.

Frequently asked questions

Can you fully prevent prompt injection?

No, and any vendor who says yes is selling something. OWASP is explicit that there's no fool-proof prevention, because the behaviour comes from how the model works, not a single fixable flaw. You reduce the risk with layered controls: privilege limits, content segregation, filtering, human approval, and adversarial testing. The goal is to make a successful injection harmless, not to make injection impossible.

Should secrets ever go in a system prompt?

Never. System prompts leak, including those of the biggest AI labs, which keep turning up in public repositories. Keep API keys, credentials, and any logic you can't afford to expose out of the prompt entirely. Enforce protection in application code the model can't reveal.

Who should own prompt governance?

Joint ownership. Security owns the threat model and testing. Engineering owns version control and deployment. Compliance owns the audit trail and the mapping to NIST and the EU AI Act. The failure mode is when the prompt is treated as a content task with no security or compliance owner attached.

What's the difference between prompt injection and jailbreaking?

Jailbreaking is injection aimed at defeating the model's safety rules. Injection is the wider category: any input, direct or hidden in external content, that changes the model's behaviour against your intent. Every jailbreak is an injection. Not every injection is a jailbreak.

The bottom line

Stop thinking of the system prompt as configuration and start treating it as a security control point with an owner, a change process, and an audit trail. Here's the opinion most teams need to hear: if you can't name who approved your current production prompt, can't show it was tested against attack inputs, and can't tie a given output back to a prompt version, you don't have a governance gap, you have an ungoverned system. The fix isn't a clever prompt. It's the boring discipline you already apply to access control and production config, pointed at the prompt.

The deadline does part of the arguing for you. With EU AI Act high-risk rules applying from August 2026 and NIST already naming injection as a security risk, prompt governance is moving from good practice to documented requirement. Build the framework now, while it's a head start rather than a scramble.

More on how we approach it: board-level AI governance.