Governing AI agents that touch money: the agent proposes, it never executes 让碰钱的 AI Agent 受治理:agent 只提议,永不执行

One rule holds the whole design up: the agent can propose, it can never act on anything material. Every write — a payment, a journal entry, an account change — leaves the model's reach and passes through an authorization service the model cannot author or bypass, and then a human with the right role. Governance built as architecture, not as a carefully worded prompt. 整套设计靠一条铁律撑起来:agent 可以提议,但永远不能对任何有物质后果的东西动手。每一个写操作——一笔付款、一条记账、一次账户变更——都离开模型可及范围,过一个模型无法编写、也无法绕过的授权服务,再过一个有对应角色的人。治理是当架构来建,不是靠一段措辞讲究的提示词。

Why "just prompt it carefully" fails为什么"把提示词写仔细点"注定失败

If the model holds a tool that can move money, then the safety of your money depends on the model's judgment on every single call — including the calls where a retrieved email, a poisoned document, or an ambiguous instruction pushed it somewhere you didn't intend. A prompt is a request for good behavior. It is not an enforcement boundary. The moment a material action is reachable from inside the model's tool surface, you've made the model your last line of defense — and the model is precisely the part you can't fully predict. 如果模型手里握着一个能动钱的工具,那你钱的安全就取决于模型在每一次调用上的判断——包括那些被检索到的邮件、被投毒的文档、或一句有歧义的指令带偏的调用。提示词是一种"请你表现良好"的请求,它不是一道强制边界。一旦某个有物质后果的动作能从模型的工具面里够得着,你就把模型变成了最后一道防线——而模型恰恰是你没法完全预测的那部分。

So the fix isn't a better prompt. It's to make the dangerous action unreachable from where the model lives. 所以解法不是更好的提示词,而是让危险动作从模型所在的地方根本够不着。

The one rule, and how each guarantee is enforced那一条铁律,以及每条保证怎么被强制

The agent reads from least-privilege, read-only sources and drafts a proposed action. That proposal — never a direct write — is all it can emit. Everything that matters is enforced in a layer the agent has no tool to reach: agent 从最小权限、只读的数据源读取,起草一个"提议的动作"。它能吐出的只有这个提议——绝不是一次直接写入。所有要紧的事,都强制在一个 agent 没有任何工具能触达的层里:

• No unauthorized payment or journal. Write tools are not in the agent's tool surface at all. Only the authorization service can reach them, and only after a role-checked human approval. 没有未授权的付款或记账。写工具压根不在 agent 的工具面里。只有授权服务能触达它们,且必须先过一个按角色核验的人工审批。
• Segregation of duties. The identity that requests an action can never be the identity that approves it — enforced in the authorization service, not left to convention. 职责分离。请求某个动作的身份,永远不能是批准它的身份——这由授权服务强制,不靠约定俗成。
• Every figure is traceable. Each number carries a lineage pointer — source document, page, retrieval citation. No lineage means it's shown as "unsourced", never presented as fact. 每个数字可溯源。每个数值都带一个溯源指针——来源文档、页码、检索引用。没有溯源的,就标成"无来源",绝不当作事实呈现。
• Prompt injection can't act. Retrieved emails and documents are untrusted data, never instructions; tool use is allowlisted; outputs are schema-validated. A malicious document can lie to the model, but it cannot reach a write. 提示注入动不了手。检索来的邮件和文档是不可信的数据,不是指令;工具调用走白名单;输出经 schema 校验。一份恶意文档可以骗过模型,但它够不到任何写操作。
• Bounded blast radius. Each integration runs on its own least-privilege, read-only credential — one compromise is one read-only surface, with no lateral movement to the rest. 爆炸半径有界。每个集成跑在自己独立的最小权限、只读凭据上——一处被攻破,就是一个只读面,没法横向移动到其余部分。

A worked example: where money actually escapes an escrow flow一个实例:钱到底从一条托管支付流的哪里溜走

The same instinct — assume the dangerous path will be taken, and design so it's contained — is what separates an escrow flow that holds from one that quietly leaks. Two parties negotiate, they agree, a hold goes on the card. Drawn end to end, here are four places the money actually escapes, and the design answer for each: 同一种直觉——假设那条危险路径一定会被走到,并把它设计成可控——正是"守得住的托管支付流"和"悄悄漏钱的托管支付流"之间的分界。两方谈判、达成一致、卡上放一笔预授权。把它端到端画出来,以下是钱真正溜走的四个地方,以及每个的设计对策:

• "Zero-UI" vs strong authentication. An off-session charge on a 3DS card comes back "authentication required" — and there is no screen to authenticate on, so the hold silently never happens. The answer: establish the mandate before negotiation (a setup step that saves an off-session payment method), plus a defined fallback for cards that still challenge. "零 UI"撞上强认证。对一张 3DS 卡做离线扣款,返回"需要认证"——可根本没有界面去认证,于是那笔预授权悄无声息地没发生。对策:在谈判之前就把授权确立好(一个保存离线支付方式的初始化步骤),再加一条针对仍要挑战的卡的明确兜底。
• Hold expiry. An uncaptured authorization lapses in roughly a week. If your capture fires after the appointment and the booking was far enough out, the hold is already dead. The answer: bound how far ahead a slot may be booked, or re-authorize near the appointment — and treat "released because visited" and "expired because we were late" as two different events. 预授权过期。一笔没扣的授权大约一周就失效。如果你的扣款发生在预约之后,而预约又订得够远,那笔预授权早就死了。对策:限制一个时段最多能提前多久预订,或在临近预约时重新授权——并且把"因到店而释放"和"因我们太晚而过期"当成两个不同的事件。
• Webhook retries. Payment webhooks are delivered at-least-once — the same event will arrive twice, and without dedup a retry captures the deposit twice. Worse, an unverified endpoint is a public "capture this hold" button. The answer: verify the signature against the raw body, persist each event id and drop repeats at the database, and send idempotency keys on capture and release. Webhook 重试。支付 webhook 是至少投递一次的——同一个事件一定会来两遍,没有去重的话,一次重试就把押金扣两次。更糟的是,一个没验签的端点就是一个公开的"扣这笔预授权"按钮。对策:对原始报文体验签、把每个事件 id 落库并在数据库层丢掉重复、在扣款和释放时都带幂等键。
• A hashed phone number is not anonymous. Phone numbers live in a tiny space; a plain hash of one is brute-forced offline in seconds, so a "hashed key" leaks identity to anyone who reads the table. The answer: a keyed hash with a server-held secret — not reversible without the secret. 哈希过的手机号并不匿名。手机号的取值空间极小;对它做一次普通哈希,离线几秒就被暴力破解,于是"哈希后的键"会把身份泄露给任何能读到这张表的人。对策:用一个服务端持有密钥的带密钥哈希——没有那个密钥就不可逆。

The through-line贯穿其中的那条线

Whether it's an accounting agent or a payment flow, the discipline is the same: keep the irreversible action out of reach of the part you can't fully trust, make every consequential number carry its source, give each integration only the access it needs, and assume the ugly path will be taken so you design for it up front. It's the same rule I apply on the industrial side — the safety-critical limit is re-checked on the controller, never trusted from the layer above. Governance you can bypass isn't governance. 无论是一个记账 agent 还是一条支付流,纪律都一样:把不可逆的动作,放在你没法完全信任的那部分够不着的地方;让每个有后果的数字都带着它的来源;只给每个集成它真正需要的那点权限;并假设最丑的那条路一定会被走到,于是提前为它做设计。这和我在工业侧用的是同一条规则——安全关键的限值在控制器侧重新核验,绝不从上层拿来就信。能被绕过的治理,不叫治理。

This is how I design these systems — the pattern comes from years of backend work and from building production LLM pipelines and MCP tool-calling systems where a wrong write has a real cost. 这就是我设计这类系统的方式——这套模式来自多年后端工作,以及构建那些"一次写错就有真实代价"的生产级 LLM 流水线与 MCP 工具调用系统的经历。

← All notes← 全部笔记