diff --git a/.gitignore b/.gitignore index 4233a15..7bcecf9 100644 --- a/.gitignore +++ b/.gitignore @@ -33,3 +33,9 @@ web-ext.config.ts ord/ .artifacts/ + +# Tool-specific clients are developed and packaged outside this repository. +/integrations/browser-transform/ + +# Product and research documents are maintained locally and are not versioned. +/docs/ diff --git a/docs/AUTO_PROFILE_INFERENCE_ARCHITECTURE.md b/docs/AUTO_PROFILE_INFERENCE_ARCHITECTURE.md deleted file mode 100644 index 7cc074d..0000000 --- a/docs/AUTO_PROFILE_INFERENCE_ARCHITECTURE.md +++ /dev/null @@ -1,589 +0,0 @@ -# Browser Profile 自动推断与 AI 协作架构 - -## 1. 产品决定 - -自动推断 Profile 不是明文网关的辅助功能，而是浏览器现场的默认完成路径。 - -用户不应先理解混淆变量、复制密钥、编写包装函数，再手工配置参数路径和输出映射。正常流程必须从一次真实业务操作开始： - -```text -用户执行登录 / 查询 / 提交 - -> Recorder 生成有界业务 Trace - -> 确定性推断器关联明文点、页面调用和请求字段 - -> 已知模式直接生成候选 - -> 未知模式请求 AI 解释业务帧和参数语义 - -> 必要时引导用户再执行一次操作以捕获业务闭包 - -> 编译为文档绑定的 Profile - -> 使用录制样本做页面内回放校验 -``` - -手写 JavaScript 保留为高级模式，不再作为主流程或文档中的首选方案。 - -本设计不包含“发送真实 HTTP 请求验证”。真实请求仍由 Yak / Web Fuzzer 的既有数据面负责。本阶段只负责发现、推断、捕获、编译和页面内样本校验。 - -## 2. 用户结果 - -以一次 CryptoJS 调用为例，默认界面应展示： - -```text -已识别请求转换 - -POST /api/login -JSON 明文 -> CryptoJS.AES.encrypt -> body.encryptedData - -输入 argument 0 <- 请求明文 JSON -Key argument 1 <- 页面内 WordArray · 16 B -IV options.iv <- 页面内 WordArray · 16 B -模式 CBC / Pkcs7 -输出 toString -> URL encode -> encryptedData - -证据 4 项 · 高置信度 -[生成 Profile] -``` - -`_0x67b862` 一类混淆名称只能出现在折叠的原始证据中。主界面使用 `Key`、`IV`、`明文输入`、`请求字段` 等语义角色。 - -用户应能回答三个问题： - -1. 插件为什么认为这是加密链路； -2. 哪些结论是确定事实，哪些是推测； -3. 还需要用户执行什么操作才能完成 Profile。 - -## 3. 设计原则 - -### 3.1 证据先于 AI - -指纹相等、请求字段解析、调用顺序、运行时对象类型和调用栈属于确定性证据。AI 不重复判断这些事实，只消费其结构化结果。 - -### 3.2 AI 不能成为执行边界 - -AI 可以： - -- 给业务 frame 排序； -- 将参数标注为 payload、key、iv、nonce、timestamp 或 signature； -- 从有限源码片段中解释序列化和包装步骤； -- 在多个候选之间给出理由； -- 建议下一次捕获点。 - -AI 不可以： - -- 直接提交任意 JavaScript 作为生产 Profile； -- 引用不存在的事件、frame、参数或页面函数； -- 读取或输出 key、Cookie、token、密码等原始值； -- 绕过 grant、document、origin 或人工接管状态； -- 将猜测标记为已经验证的事实。 - -### 3.3 页面是执行环境，不是密钥导出器 - -Key、IV、CryptoKey、key promise、WASM 实例和闭包变量继续保留在原页面。Profile 只保存页面内 opaque callable 引用和经过校验的参数映射。 - -### 3.4 已知模式不依赖 AI，未知模式不依赖库清单 - -WebCrypto、CryptoJS、JSEncrypt 以及后续 sm-crypto、node-forge 等已知模式，连同 URLSearchParams、JSON、FormData 和常见编码链，应优先由确定性规则推断。AI 只处理业务语义和未知代码，避免增加延迟、成本和不确定性。 - -录制协议只暴露统一的 `crypto` 事件，库差异进入结构化 `adapterId / providerKind / family / operation / algorithm / mode / padding / encoding / state / key metadata`。推断器、时间线、Deep Capture 和 Agent 不再分别判断 `webcrypto`、`cryptojs` 等事件类型。新增密码库时只扩展 MAIN-world adapter、扩展自带的 manifest 和受限元数据归一化器，不扩展整条产品协议。 - -已知 adapter 只负责提供更准确的参数角色、算法和状态语义，不是通用性的唯一来源。对于 ESM/Webpack 闭包、Worker、WASM 或完全未知的业务封装，系统必须从请求/消息边界和调用栈恢复上层业务 callable；算法尚未命名不能单独成为 `insufficient-evidence`。adapter 协议的开放化、Worker/MessagePort 边界、高价值库优先级与反靶场特化验收见 [`FRONTEND_CRYPTO_GENERALIZATION_ROADMAP.md`](FRONTEND_CRYPTO_GENERALIZATION_ROADMAP.md)。 - -### 3.5 无兼容负担 - -插件尚未正式投入使用。页面配方、运行时适配器和 Transform Profile 可以直接收敛到新模型，不保留旧数据迁移或双写逻辑。 - -## 4. 总体架构 - -```text -MAIN-world Recorder - | bounded events + opaque handles + semantic argument metadata - v -Evidence Normalizer - | request fields / call slots / encodings / exact & normalized links - v -Evidence Graph - | proven edges + supported edges + hypotheses - +-----------------------+ - | | - v v -Deterministic Inference AI Analysis - | known patterns | frame ranking / semantic labels / unknown code - +-----------+-------------+ - v - Candidate Merger - | schema validation + evidence reference validation - v - Pipeline Compiler v2 - | page callable graph, no arbitrary generated code - v - Local Sample Replay - | deterministic compare or structural assertions - v - Document-bound Profile -``` - -推断计算在扩展后台完成。Yakit、Options 和 AI Agent 读取同一候选结构，不各自实现一套启发式规则。 - -## 5. Evidence Graph - -### 5.1 节点 - -```ts -type EvidenceNode = - | RecordingEventNode - | RecordedValueNode - | RequestFieldNode - | CallableNode - | CallArgumentNode - | StackFrameNode - | SourceExcerptNode -``` - -节点只使用录制会话内稳定 ID。原始敏感值不是图节点属性。 - -### 5.2 边 - -```ts -type EvidenceStrength = "proven" | "supported" | "hypothesis" - -type EvidenceEdgeKind = - | "exact-value" - | "normalized-value" - | "parent-call" - | "same-trace" - | "stack-frame" - | "argument-role" - | "request-destination" -``` - -- `exact-value`：同一录制盐下的指纹完全相同； -- `normalized-value`：经过有界白名单转换后相同，例如 URL decode、JSON field extraction 或 Base64 表示； -- `parent-call`：Recorder 的同步父调用关系； -- `same-trace`：弱证据，只证明时间和用户操作相关； -- `hypothesis`：只能由 AI 或启发式产生，必须列出依据。 - -### 5.3 请求边界归一化 - -网络事件在边界处解析，不全局 Hook `JSON.stringify` 或 `encodeURIComponent`： - -- JSON：递归提取最多 64 层、100,000 节点； -- `application/x-www-form-urlencoded`：字段级 URL decode； -- `FormData`：字段名、字符串值和文件元数据； -- Headers：规范化名称但保留原始大小写用于展示； -- Query：字段级解析； -- 原始 body：保留整体指纹和类型。 - -归一化候选只允许白名单操作并设置总预算。不得对每个值进行无界编码组合爆炸。 - -### 5.4 参数语义 - -Recorder 对已知库记录参数角色而不是变量名： - -```ts -interface CallArgumentEvidence { - index: number - role: "data" | "key" | "iv" | "algorithm" | "options" | "signature" | - "salt" | "nonce" | "aad" | "unknown" - dataType: string - byteLength?: number - replaceable: boolean - retained: boolean - summary?: string -} -``` - -例如 CryptoJS AES： - -- `argument 0`：data，可替换； -- `argument 1`：key，不导出，页面内保留； -- `argument 2`：options，提取 mode、padding 和 IV 长度，不提取 IV 值。 - -例如 JSEncrypt RSA： - -- `argument 0`：UTF-8 data，可替换；对象输入按稳定 JSON 序列化后再交给原函数； -- receiver：保留实际 JSEncrypt 实例，不重建、不导出； -- key：只记录 public/private、模数位数和本次录制随机加盐的指纹； -- padding：记录 `PKCS1-v1_5` 等可解释元数据； -- output：记录 Base64 形态并与 JSON/Form/Header/Query 请求字段做 exact link； -- 公私钥 PEM、模数、指数和页面实例永不进入候选或 AI 上下文。 - -如果一次 RSA 输出精确进入一个请求字段，且原函数、receiver 和参数模板仍在当前 document 中，候选可以直接进入 `ready`，不要求用户填写函数表达式或先进入 Deep Capture。 - -## 6. 统一 Page Callable - -当前页面配方和深度捕获适配器表达的是同一概念：在当前文档中可重复调用的页面函数。两套注册表应合并为 `BrowserPageCallable`。 - -```ts -interface BrowserPageCallable { - id: string - kind: "recorded-call" | "business-closure" | "global-function" - name: string - target: BrowserTarget - lifecycle: "document" - inputSlots: CallableInputSlot[] - output: CallableOutputShape - provenance: { - recordingId?: string - traceId?: string - eventId?: string - frameId?: string - sourceUrl?: string - lineNumber?: number - } -} -``` - -`recorded-call` 保存原函数、receiver、固定参数模板和可替换槽位；`business-closure` 保存 CDP 暂停时捕获的业务函数与闭包；两者使用同一执行、授权、生命周期和审计接口。 - -Profile 不再引用 `recipeId` 或 `adapterId`，只引用 `callableId`。 - -## 7. Pipeline v2 - -手工 JavaScript 中常见的 JSON 序列化、编码、调用和封装应变成可审计的类型化节点： - -```ts -type PipelineNode = - | { kind: "context.read"; path: string } - | { kind: "builtin"; operation: BuiltinOperation; inputs: NodeRef[]; options?: object } - | { kind: "page.call"; callableId: string; arguments: NodeRef[] } - | { kind: "output.write"; destination: string; source: NodeRef; encoding: ValueEncoding } -``` - -首批 `BuiltinOperation`： - -```text -value.literal -json.stringify -json.parse -text.toString -url.encode -url.decode -base64.encode -base64.decode -hex.encode -hex.decode -object.pick -object.compose -form.compose -``` - -`value.literal` 只允许字符串、数字、布尔值或 `null`，用于编译器生成固定的协议元数据，例如表单 -`Content-Type`。它不接受输入，也不能持有函数、对象或页面秘密。 - -每个节点有明确输入输出类型和大小预算。未知操作不能通过 AI 临时创造；用户确实需要自定义代码时，进入独立的高级节点，并沿用程序 Eval 的高风险授权。 - -## 8. 推断候选 - -候选不是立即生效的 Profile： - -```ts -interface BrowserProfileInferenceCandidate { - id: string - recordingId: string - traceId: string - target: BrowserTarget - request: { eventId: string; method: string; url: string } - direction: "request" | "response" - status: "ready" | "capture-required" | "mapping-required" | "insufficient-evidence" - confidence: { score: number; level: "high" | "medium" | "low" } - summary: string - pipeline: PipelineNodeDraft[] - evidence: InferenceEvidenceRef[] - missing: InferenceMissingStep[] - aiContext: BrowserInferenceAIContext -} -``` - -置信度不是 AI 的主观百分比。分数由固定规则产生，并在 UI 中解释： - -- 请求字段与加密输出 exact link：强加分； -- 可重复 callable 已保留：强加分； -- 同一用户 Trace 且顺序正确：中等加分； -- 仅时间接近：弱加分； -- 多个同分候选：降分； -- 缺少输入映射或输出封装：状态不能为 ready。 - -## 9. 自动业务函数捕获 - -低层 `CryptoJS.AES.encrypt` 或 `crypto.subtle.encrypt` 往往不足以构造完整线上报文。推断器应把它作为断点入口，然后寻找上层业务函数。 - -```text -候选指出需要业务 callable - -> 用户点击“自动捕获完整加密流程” - -> 插件在已知低层调用处 arm 一次性断点 - -> 用户重复相同操作 - -> 页面暂停并立即显示控制面 - -> 后台排除 Hook/依赖帧，并使用多来源共同祖先提示排序页面帧 - -> 纯函数用 selected-frame；负责 DOM 取值/组包/发送的函数用 request-transaction - -> 页面立即恢复 - -> 新明文映射到参数或页面控件，仅返回被拦截的线上 envelope -``` - -录制器只从每个来源事件的有界同步栈提取页面帧提示，并对 `functionName + script URL` 求交集；支持来源更多、平均深度更浅的共同祖先优先。捕获入口选择最早已确认的密码来源，而不是已经离开上层异步函数后的 Fetch 边界。后台再结合真实 CDP `scriptId`、函数位置、来源分类和副作用检查做最终选择，因此前端不能通过提交 URL、行号或函数名把任意对象伪装成推荐帧。 - -当最高候选唯一、可解析且未发现副作用时，默认路径使用 `selected-frame`。如果多个密码来源的最近共同页面祖先本身包含网络、DOM 或条件导航，系统不会跳过它去选更外层的事件 handler，而是建立 `request-transaction`：保留真实函数、receiver 和固定参数，在页面内替换明文控件，拦截唯一的目标 Fetch/XHR/Beacon/Form，校验所有预期输出字段后回滚 DOM。 - -存储副作用、多个或未授权请求、无法唯一绑定函数、或共同祖先证据并列时，系统保持页面暂停并解释原因。函数引用表达式只存在于高级模式。页面暂停不等待远程 AI；AI 只能在页面恢复后基于同一份有界证据做解释和候选补丁。 - -函数捕获后，后台从 `Function.prototype.toString` 恢复包括默认参数在内的有序参数名。单参数业务函数默认读取整个逻辑 Body；多参数且名称可靠时，引导配置生成 `body.` 读取节点；`arg0` 这类占位名不会被冒充为已确认字段。Options 同时从已授权暂停帧的 local/block/closure scope 取同名原始值，构造一次性的本地回放 Body。完整暂停作用域始终只存在于当前会话；只有用户明确生成并保存明文网关后，选中的短时样本才会复制到独立的本机回放草稿。该草稿按 `profileId + request/response` 隔离，不写入 Profile、Bridge、审计、Yak/AI、诊断或导出，并可由用户单独清空。 - -### 9.1 多密码调用按请求建图 - -一个请求可能同时包含 AES ciphertext、RSA-encrypted session key、HMAC signature、nonce 和 timestamp。即使每个低层输出都与请求字段精确匹配，也不能把这些调用分别保存后独立回放：它们可能共享同一随机 key、IV、nonce 或闭包状态。 - -推断器因此按请求边界合并多个来源，生成一个 request-level candidate： - -```text -plaintext -----------------> AES.encrypt ----------> body.data -dynamic AES key -----------> JSEncrypt.encrypt ----> body.encryptedKey -canonical request fields --> HMAC.sign ------------> header.X-Sign - | - +-- 同一上层业务 callable 保证动态值一致 -``` - -界面展示每个密码调用及其线上目标，但状态固定为 `capture-required`。用户点击“自动捕获完整加密流程”时，Deep Capture 优先在仍保留上层业务调用栈的密码来源处武装断点，并捕获一次上层业务封装；系统不会把多个看似 ready 的低层调用拆成多个可执行 Profile，也不会误导用户反复缩短已经足够短的录制操作。 - -### 9.2 请求事务的输入与输出契约 - -`request-transaction` 对明文只暴露一个 `body` 输入槽，对 Pipeline 返回被页面业务代码生成的整个请求 Body。因此 AES + RSA 之类多输出流程会直接编译为： - -```text -context.read(body) - -> page.call(sendDataAesRsa 请求事务) - -> output.write(body) -``` - -事务保留暂停现场的 URL/event/receiver 等固定参数。逻辑 Body 是对象时，先按 input `name/id` 向页面控件做同名映射；参数名明确是 `payload/data/body/request/params/input` 时才直接替换参数。已混淆的单参数如果其保留值解析后等于目标 URL，必须继续保留，不得被明文对象覆盖。 - -## 10. AI Agent 集成 - -### 10.1 绑定资源 - -Yakit 从浏览器集成页启动 AI 分析时，附加一个类型化资源： - -```text -AttachedResourceInfo.type = browser_session -AttachedResourceInfo.key = context -``` - -Value 只在 Yak 进程内解析，包含 device、grant、document、selected trace 和 candidate ID。渲染给模型的内容只包含安全摘要，不暴露 device token、grant secret 或录制值。 - -资源必须绑定： - -```text -timeline session -AI task -deviceId -grantId -tabId + frameId + documentId + origin -expiresAt -``` - -### 10.2 Agent 工具 - -不要把几十个 Bridge RPC 原样暴露给模型，也不要提供通用 `method + params` 工具。首批提供三个领域工具： - -```text -browser_observe - page summary / actionable nodes / trace / inference / status / diff - -browser_inference - list candidates / inspect evidence / arm capture / choose callable / - propose mapping / compile candidate / local replay - -browser_act - stable node action / tab activation / human handoff -``` - -工具回调从当前 AI task 的 `browser_session` 资源解析绑定，AI 参数中不存在 `deviceId`、`grantId` 或任意 Bridge method。 - -`browser_observe` 默认只读；`browser_inference` 的读取和推断无需额外确认，arm debugger、创建 callable 和发布 Profile 使用现有细分 scope；`browser_act` 遵循 Agent review policy 和人机接管状态。 - -### 10.3 AI 输出 Schema - -AI 只能返回候选补丁： - -```ts -interface AIInferencePatch { - candidateId: string - labels: Array<{ evidenceId: string; role: SemanticRole; reason: string }> - preferredFrameId?: string - argumentBindings?: Array<{ slotId: string; contextPath: string; reason: string }> - suggestedBuiltins?: Array<{ operation: BuiltinOperation; evidenceIds: string[] }> - unresolved: string[] -} -``` - -Candidate Merger 必须验证所有 ID 存在、document 未变化、操作在白名单内、映射路径合法。验证失败只产生新的待处理项，不能退化为执行 AI 代码。 - -## 11. UI / UX - -录制是入口，自动推断是录制完成后的主结果。三列工作台保持不变： - -```text -Trace 列 | 数据流与候选 | 推断证据 / 下一步 -``` - -右侧主区域按状态显示： - -- `ready`：一键生成 Profile； -- `capture-required`：解释原因并提供“自动捕获完整加密流程”； -- `mapping-required`：只让用户选择少量无法确定的明文字段； -- `insufficient-evidence`：建议重新录制，并明确缺少哪类证据。 - -证据采用三种强度： - -- 已证实：实线和明确措辞； -- 有支持：普通文本并展示依据； -- 待确认：虚线或次级文本，不使用成功色。 - -AI 是候选的解释者，不单独占据一个聊天面板。主要入口是“让 AI 深入分析”，结果回填到同一证据区域。需要继续对话时再打开 Yakit AI 会话，并携带相同 `browser_session` 资源。 - -手工 Pipeline 编辑器移入“高级编辑”，默认只展示推断出的可读流程和少量可修改字段。 - -默认 Profile 编辑器不是节点画布，而是三个业务决定： - -```text -1. 明文从哪里来 -2. 交给哪个页面函数 -3. 线上结果写到哪里 -``` - -当第三步选择“写入表单字段”并填写 `encryptedData` 时，编译器自动生成 -`form.compose(keys=["encryptedData"])`、固定 Content-Type、Header 输出和 Body 输出。用户不需要看到或填写 -`keys`、节点 ID、输入引用和输出引用。已有非规范 DAG 不会被静默改写，只能继续在高级模式中编辑，或由用户明确替换为引导流程。 - -## 12. 性能预算 - -- 单次快照最多 500 事件、每事件 48 个 evidence； -- 图构建使用 fingerprint/path 索引，目标复杂度 `O(E + V)`； -- normalized link 每值最多生成 8 个白名单变体； -- 候选最多 16 个，发送给 AI 的候选最多 3 个； -- scope 每次最多 8 个 frame，源码片段按需读取并限制总字节； -- 推断结果按 `recordingId + event revision` 缓存，增量追加事件时只处理新增部分； -- 不在页面主线程执行全量源码搜索、AST 构建或全局 JSON/URL 编码 Hook； -- Pipeline 在目标 document 内一次执行完成，每次请求/响应只跨扩展到页面边界一次，不按节点往返； -- 页面暂停路径绝不等待网络或 AI。 - -## 13. 隐私与授权 - -- 默认推断只使用指纹、类型、长度、路径、算法摘要和源码位置； -- 敏感录制预览即使被用户开启，也不自动进入 AI context； -- Key、IV、CryptoKey 和闭包值只显示语义、类型与长度； -- 源码片段可能包含硬编码 secret，发送 AI 前先进行字面量脱敏并由用户授权； -- 推断读取使用 `browser.recording.read`； -- scope/source 深入读取使用 `browser.debugger.read`； -- arm/resume 与 callable 创建使用 `browser.debugger.control`； -- callable 创建、执行与本地回放使用 `browser.callable.execute`；从暂停 frame 捕获 callable 还需要 `browser.debugger.control`； -- Profile 发布使用 `browser.transform.manage`； -- document、origin 或 grant 变化后候选立即标记 stale，不静默重绑。 - -## 14. 生命周期与恢复 - -Profile 是 document-bound。刷新后不能继续调用旧闭包，但推断定义可以保留为恢复计划： - -```text -页面刷新 - -> callable stale - -> Profile disabled - -> 插件按原 operation / script / route 重新 arm - -> 用户正常执行一次业务操作 - -> 重新捕获 callable - -> 本地样本校验 - -> 用户确认后重新启用 -``` - -恢复计划不保存 key 或源码计算结果，只保存捕获入口、业务 frame 特征、参数语义和映射结构。 - -## 15. 分阶段实现 - -### P0：证据与候选基线 - -- [已完成] 使用统一 `crypto` 事件记录 WebCrypto / CryptoJS / JSEncrypt / sm-crypto / node-forge 的 adapter、provider kind、family、调用、参数角色、类型、长度和 state/retained 状态； -- [已完成] MAIN-world 密码适配器注册表支持稳定 adapter 与运行时晚加载 adapter； -- [已完成] JSEncrypt RSA encrypt/decrypt/sign/verify 保留真实 receiver，并仅输出公私钥类型、位数和加盐指纹； -- [已完成] 为 CryptoJS 结果补充安全的字符串表示 evidence； -- [已完成] 从 exact link、请求字段和调用顺序生成只读候选； -- [已完成] Options / Yakit 展示置信度、证据和缺失步骤； -- [已完成] 候选结构可通过 `browser.recording.get` 提供给 Agent。 - -### P1：统一 Callable 与 Pipeline v2 - -- [已完成] 删除 recipe / adapter 双模型，不保留旧方法别名或迁移分支； -- [已完成] 页面 callable 使用统一注册表、来源信息、生命周期和命名 input slot schema； -- [已完成] Pipeline v2 使用有序 DAG，并加入类型化 context.read / builtin / page.call / output.write 节点； -- [已完成] builtin 限定为 JSON、文本、URL、Base64、Hex、对象和表单组合白名单； -- [已完成] 输出支持 body、字段级 body、header 和 query，并由 Yak 二次限制 URL 只能改变 query； -- [已完成] 单条 exact value link 且保留可执行调用句柄的 stateless/receiver 模式可直接编译候选；stateful/stream 模式必须捕获上层 callable； -- [已完成] 同一请求的多个密码来源合并为 request-level candidate，并强制捕获上层业务 callable 以保持动态值关系； -- [已完成] JSON 字段、表单字段、Header、Query 和完整 Body 会编译为对应的引导式输出，不要求用户理解 DAG； -- [已完成] 录制短时样本自动填入 Options/Yakit 明文网关本地回放，并允许编辑后恢复原样本； -- 为页面内回放生成确定性/结构性断言。 - -### P2：自动业务函数捕获 - -- [已完成] 候选一键 arm，并在已有捕获等待或页面暂停时拒绝覆盖； -- [已完成] 业务 frame 使用来源、边界距离、函数可解析性、副作用、命名和作用域信息做确定性排序； -- 参数槽位与 request context 自动映射； -- 文档刷新后的引导式重新捕获。 - -### P3：Yak AI Agent - -- `browser_session` attached resource； -- task-bound 三个 Agent 工具； -- AIInferencePatch schema 与 Candidate Merger； -- Yakit 从候选直接启动带上下文的 AI 会话； -- Agent 操作写入现有 session timeline。 - -### P4：复杂应用 - -- Axios/interceptor、GraphQL、WebSocket frame、protobuf 与自定义 serializer； -- [已完成] 按通用化路线迁移 adapter host，加入 sm-crypto、node-forge 与 Beacon/Worker/MessagePort 边界，并通过随机 ESM + WASM holdout； -- jsrsasign、jose 与后续现代密码生态按真实样本继续推进； -- sourcemap 存在时的业务 frame 增强； -- 多候选对比和跨操作共用 callable 识别。 - -P4 的实现顺序、协议草案、性能门禁和随机化测试矩阵以 [`FRONTEND_CRYPTO_GENERALIZATION_ROADMAP.md`](FRONTEND_CRYPTO_GENERALIZATION_ROADMAP.md) 为准。 - -## 16. 验收夹具 - -至少覆盖： - -1. 固定 CryptoJS AES，混淆变量名，JSON 字段输出； -2. 真实 JSEncrypt RSA + form-urlencoded `data` 字段，独立服务端用私钥解密验收；保留实例 receiver，停止录制后对象明文仍可回放； -3. RSA 候选和 AI 上下文只包含 key 类型、位数与加盐指纹，不包含 PEM 或模数； -4. WebCrypto AES-GCM + HMAC，闭包内不可导出 key 和动态 nonce/IV； -5. AES + RSA + HMAC 同请求多来源图，不允许拆分低层调用回放； -6. 动态 key promise，页面刷新后重新捕获； -7. Axios interceptor 中的请求签名； -8. Form URL encode 和 Header signature； -9. 自定义业务 wrapper，低层库调用不足以构造完整报文； -10. 未知函数与多个同分业务 frame，AI 只能补全候选，不能直接执行代码； -11. WASM 导出函数，只能观察输入输出和业务 wrapper； -12. 敏感预览开启时，AI payload、审计和诊断仍不含原始值； -13. 500 事件 / 24,000 evidence 的性能与内存预算。 - -## 17. 目标目录 - -```text -src/features/browser-recording/ - evidence.ts - recorder.ts - -src/features/browser-inference/ - graph.ts - normalize.ts - rules/ - candidates.ts - compiler.ts - ai-context.ts - -src/features/browser-callable/ - registry.ts - execute.ts - lifecycle.ts - -src/features/browser-transform/ - pipeline-v2.ts - profile.ts - replay.ts -``` - -Yak 侧将 `browser_session` 资源解析和 Agent 工具放在独立包中，依赖一个最小的 Bridge caller interface，避免 `common/ai` 直接依赖 gRPC Server。 diff --git a/docs/BROWSER_TRANSFORM_GATEWAY.md b/docs/BROWSER_TRANSFORM_GATEWAY.md deleted file mode 100644 index fd3f782..0000000 --- a/docs/BROWSER_TRANSFORM_GATEWAY.md +++ /dev/null @@ -1,259 +0,0 @@ -# Browser Transform Gateway - -## 1. Product contract - -The Browser Transform Gateway exists for one concrete testing workflow: - -> The operator edits and fuzzes meaningful plaintext in Yakit, while the live authenticated browser page performs the same encryption, signing, serialization, dynamic-parameter generation, or response decryption that the production application performs. - -The result sent on the network must be accepted by the real server. A fixed codec demo, copied JavaScript function, or standalone mock key does not satisfy this contract. - -The primary workflow is: - -```text -real browser operation - -> Recorder correlates user input, crypto calls, and network requests - -> an exact recorded call is retained directly when it already covers the required transform - -> otherwise Deep Capture pauses at the relevant higher-level business call - -> operator retains the real in-scope function as a page callable - -> operator composes callables and typed nodes into a request/response transform profile - -> Yakit Web Fuzzer remains a plaintext editor - -> Yak asks the selected live browser to transform the request immediately before sending - -> Yak sends the resulting wire packet - -> Yak optionally asks the browser to transform the wire response - -> Yakit displays plaintext and preserves a separate wire view -``` - -The browser is therefore an execution environment, not a passive code source. Non-extractable `CryptoKey` objects, closure variables, key promises, runtime tokens, random generators, timestamps, WASM instances, and application serializers remain in the page that already owns them. - -There are two valid discovery outcomes: - -1. **Direct recorded callable.** One observed primitive already accepts the logical plaintext and its output is proven to enter one wire destination. For example, `JSEncrypt.encrypt` with its real instance receiver can map an object body to form field `data`. The operator generates the guided gateway directly; no function expression or debugger pause is required. -2. **Business callable.** A request combines multiple primitives or surrounding serialization/dynamic state. AES ciphertext, RSA-wrapped key, signature, nonce, timestamp, and request canonicalization are treated as one request graph, then Deep Capture retains the higher-level closure. The extension never replays those low-level calls independently merely because each output has an exact field link. - -## 2. Relationship to JS-RPC and JS-Forward - -JS-RPC, JS-Forward, browser-side hook tools, and this gateway share the same basic idea: forward values into a browser JavaScript environment and receive transformed values back. The important product difference is the ownership and workflow around that call. - -| Concern | Traditional forwarding setup | Browser Transform Gateway | -| --- | --- | --- | -| Function discovery | User locates and exposes a function manually | Recorder and Deep Capture lead from a real request to the relevant business frame | -| Runtime environment | Usually a manually maintained browser tab or injected service | Explicitly selected, paired, document-bound authenticated tab | -| Data-plane integration | External HTTP port or custom script modifies packets | Native Web Fuzzer pre-send and post-response hooks in the owning Yak gRPC process | -| Request editing | Often ciphertext-oriented or script-oriented | Plaintext is the canonical editable request | -| Observability | Tool-specific logs | Plaintext request, wire request, wire response, plaintext response, and step timing | -| Lifecycle | Caller must notice stale pages/functions | Navigation and refresh fail with document/origin errors; no silent retargeting | -| Authorization | Commonly a shared local endpoint | Paired device, task, grant, target, scope, and capability schema | - -An external forwarding port can be added later as another Yak data-plane adapter for Burp/Fiddler compatibility. It must reuse the same profile execution contract and must not become a second configuration or authorization system. - -Research notes and comparisons are retained in [`study.md`](study.md). They inform discovery and UX, but the production acceptance criterion is always whether a server accepts the transformed packet. - -## 3. Component responsibilities - -### Browser extension - -- discovers page-side data flow through Recorder; -- captures a real business closure through Chromium Deep Capture; -- stores only document-bound callable metadata and transform profiles; -- keeps an optional replay draft per profile and direction in extension-local storage after the operator saves a gateway; -- validates route, method, origin, document, function binding, paths, and output mappings; -- executes an ordered Pipeline v2 DAG in the live MAIN world; -- sends the complete validated DAG and packet through one extension-to-page round trip instead of crossing the boundary for every node; -- returns bounded URL/body/header mutations plus per-node duration; -- never exports closure bindings or key material. - -The replay draft is deliberately not a field of the transform profile. It may contain a plaintext account, password, -token, request headers, or a selected short capture sample. It is keyed by `profileId + request/response`, stays in -`browser.storage.local`, and is excluded from profile export, Bridge/RPC capabilities, Yak/AI context, audit, and -diagnostics. Deleting a profile deletes both directional drafts. The editor autosaves at most 256 KiB per direction; -larger input remains usable in the current Options page but replaces no persisted value. - -### Yak engine - -- performs profile preflight through the Bridge owned by the current gRPC process; -- composes the browser transform with existing Web Fuzzer hot-patch hooks; -- calls the selected browser immediately before the real request and immediately after the real response; -- fails closed before network transmission if request conversion fails; -- emits an explicit synthetic `598 Browser Transform Failed` response if response conversion fails; -- preserves logical and wire packets separately in every Fuzzer result and history item. - -### Yakit - -- lists only online paired browsers and profiles visible to the active grant; -- provides the full profile editor in Browser Integration; -- lets Web Fuzzer select one browser/profile pair without leaving the request workflow; -- keeps `RequestRaw` and `ResponseRaw` as the canonical plaintext editor/display values; -- exposes `WireRequestRaw` and `WireResponseRaw` through a stable side-by-side comparison; -- restores the selected browser/profile when reopening Fuzzer history. - -## 4. Transform profile - -A profile is intentionally document-bound and contains: - -- a name and enabled state; -- `tabId + frameId + documentId + origin`; -- allowed HTTP methods and a bounded wildcard URL pattern; -- an optional request pipeline; -- an optional response pipeline; -- `failMode: closed`; -- a bounded per-profile concurrency limit from 1 to 8. - -Method, URL, headers, body, captured short sample, and the last local replay result are not profile fields. The first -five can be restored from the separate local-only replay draft; execution results and errors are never persisted. - -At least one direction must be enabled. Every enabled direction contains at least one node and one `output.write` node. - -A path-only URL pattern such as `/api/*` or `*/api/login` is restricted to the bound page origin. Cross-origin APIs must be intentional: use a full pattern such as `https://api.example.test/*`. This prevents a broadly reusable path rule from turning a page-held key into a cross-origin signing oracle. - -### Pipeline v2 nodes - -Each node has a stable ID and may reference only an earlier node. This makes the data flow explicit and prevents cycles or undeclared reads. The supported node kinds are: - -| Node | Purpose | -| --- | --- | -| `context.read` | Read a safe path from the immutable input context | -| `builtin` | Apply one whitelisted JSON/text/URL/Base64/Hex/object/form operation | -| `page.call` | Invoke one document-bound `BrowserPageCallable` with referenced arguments | -| `output.write` | Write a referenced value to an allowed packet destination | - -The normal editor presents these nodes through a three-step guided compiler: choose the plaintext source, choose the -live page callable, and choose the wire destination. The ordered DAG is an implementation detail under “Advanced -Pipeline”; operators do not manually select node references for common request encryption. - -For example, choosing `form field` with the name `encryptedData` compiles to: - -```text -context.read(body) - -> page.call(recorded AES callable) - -> form.compose(keys=["encryptedData"]) - -> output.write(body) - -value.literal("application/x-www-form-urlencoded") - -> output.write(header.Content-Type) -``` - -`value.literal` is a bounded whitelist operation that accepts only a primitive value and no inputs. It exists so the -compiler can express fixed protocol metadata without arbitrary JavaScript. Existing non-canonical DAGs remain in the -advanced editor and are never silently rewritten. - -`context.read` accepts these safe roots: - -```text -method -url -statusCode -headers.content-type -body -body.account -body.password -text -bodyBase64 -query -query.name -``` - -Missing node IDs, forward references, duplicate IDs, malformed paths, and prototype traversal segments are rejected. Arbitrary JavaScript is not a Pipeline node. - -The background validates the profile, route, origin and live document before dispatch. The selected document then evaluates the complete bounded DAG locally, including all `page.call` nodes, and returns one structured result. This keeps multi-node profiles from multiplying `scripting.executeScript` latency and keeps the Pipeline executor out of the always-on Service Worker bundle. - -### Output nodes - -An `output.write` maps a prior node result to exactly one supported destination: - -```text -body replace the complete body -body.password update a JSON body field -header.X-Sign set a header; null/undefined removes it -query.signature set a URL query field; null/undefined removes it -``` - -Output encoding is explicit: `auto`, `text`, `json`, or `base64`. Header names and values reject CR/LF injection. JSON and form field mapping preserve their structured wire format and never mutate object prototypes. The extension can return a query-mutated URL, but Yak independently verifies that scheme, hostname, effective port and path are unchanged before replacing the request target. - -## 5. Ordering - -Request execution order is deliberate: - -```text -plaintext request in Web Fuzzer - -> user beforeRequest hot patch - -> browser request transform - -> actual wire request -``` - -Response execution uses the inverse boundary: - -```text -actual wire response - -> browser response transform - -> user afterRequest hot patch - -> plaintext response in Web Fuzzer -``` - -This allows ordinary Web Fuzzer mutation logic to work on meaningful application data. The browser transform remains the last operation before transmission and the first operation after receipt. - -Redirected requests are transformed independently. A redirect to a route outside the selected profile fails closed instead of leaking a plaintext request to an unintended endpoint. - -## 6. Failure and lifecycle semantics - -The request path never falls back to sending plaintext. Profile lookup failure, offline device, expired grant, stale document, changed origin, unavailable callable, route mismatch, illegal URL mutation, queue overflow, invalid mapping, timeout, and page exception all abort transmission. - -The response path never presents undecoded wire data as if it were plaintext. It returns an explicit transformation failure while preserving the wire response for diagnosis. - -Profiles are not portable secrets. They may remain visible after a navigation so the operator can understand what became stale, but execution requires the exact current document and all referenced page callables. A reload intentionally requires recapture and rebinding. - -Local replay drafts can contain secrets even though profiles do not. Navigation or a temporarily stale callable keeps -the draft intact so the operator does not lose work. The operator can clear the current direction explicitly, and -deleting its owning profile removes both request and response drafts. The draft does not make a stale callable -executable and is never silently rebound to another origin. - -Chromium is required for capturing closure-bound business callables. Firefox keeps Recorder-created callables but does not advertise or display the Transform Gateway and Deep Capture workspaces. - -## 7. Bounds and performance - -- request and response bodies are limited to 8 MiB; -- a profile has at most 64 nodes per direction; -- a builtin or page-call node has at most 64 input references; -- a profile queue is bounded to 128 waiting operations; -- per-profile concurrency is 1, 2, 4, or 8 in the UI; -- page-callable output is lossless within bounds; cycles, functions, symbols, excessive depth/nodes, and oversized values fail explicitly; -- parsed and mapped JSON is limited to 64 levels and 100,000 nodes before a page function is invoked; -- previews may be truncated, execution values are never silently truncated; -- Bridge messages remain under the existing 16 MiB aggregate limit and use chunking above 512 KiB. - -Concurrency must reflect the page function's state model. Use `1` when the application mutates shared counters, nonce state, or token caches. Higher values are appropriate only after verifying that the retained business function is re-entrant. - -## 8. Authorization - -Transform capabilities are separated by intent: - -| Capability | Scope | -| --- | --- | -| `browser.transform.profile.list` | `browser.transform.read` | -| `browser.transform.profile.save/delete` | `browser.transform.manage` | -| `browser.transform.execute` | `browser.transform.execute` | - -The Bridge router revalidates the profile target against the active grant for list, save, delete, and execute. An existing profile ID cannot be rebound to another page document. - -## 9. Acceptance criteria - -The production fixture uses live request and response functions that close over non-extractable AES-GCM and HMAC keys. The request function creates a new timestamp, nonce, and IV per call, encrypts a JSON login payload, and signs the resulting envelope. The server returns a second AES-GCM envelope that only the retained page response function opens. Acceptance requires all of the following: - -1. A plaintext account/password packet is transformed through the retained page closure. -2. The wire packet does not contain the plaintext password. -3. The independent test server verifies HMAC, decrypts AES-GCM, and recovers the original request values. -4. The server returns an encrypted response with no plaintext password, and the retained page closure restores it to JSON. -5. Repeated calls produce different nonce and IV values. -6. A mismatched path or implicit cross-origin URL fails closed. -7. Web Fuzzer preserves and displays both plaintext and wire packets in both directions. -8. Refreshing the bound document invalidates execution rather than silently using a new page. - -Extension browser E2E covers the real page and server boundary. Yak unit tests cover hook ordering, request/response conversion, trace preservation, and failure behavior. Yakit TypeScript verification covers the integrated selector, editor, and packet comparison surfaces. - -The browser E2E suite also covers the direct RSA path independently: the real JSEncrypt browser bundle receives a generated RSA public key, its Base64 ciphertext is linked to `application/x-www-form-urlencoded` field `data`, and a separate HTTP test server holding the private key must decrypt and recover the original JSON. Raw key material must remain absent from the candidate/AI context. After recording stops, both Bridge- and UI-created callables must still invoke the retained receiver with a new structured plaintext value whose ciphertext the server-side decryptor can open. - -## 10. Current boundary - -The first production data-plane integration is Yakit Web Fuzzer. Direct Burp/Fiddler interception, WebSocket frame transformation, streaming bodies, and unattended cross-document callable recovery remain outside the current contract. They should be built as explicit extensions of this gateway, not as hidden fallbacks. - -Automatic Profile inference is now part of the core product path rather than a later convenience. Recorder evidence, retained page callables, deterministic rules and task-bound AI analysis must lead from one real browser operation to an explainable Profile candidate. The architecture, evidence contract, AI boundary and phased implementation are defined in [`AUTO_PROFILE_INFERENCE_ARCHITECTURE.md`](AUTO_PROFILE_INFERENCE_ARCHITECTURE.md). diff --git a/docs/DEEP_CAPTURE_ARCHITECTURE.md b/docs/DEEP_CAPTURE_ARCHITECTURE.md deleted file mode 100644 index 61b817f..0000000 --- a/docs/DEEP_CAPTURE_ARCHITECTURE.md +++ /dev/null @@ -1,149 +0,0 @@ -# Deep Capture Architecture - -## Product boundary - -Deep Capture is for an authorized tester who can reproduce a real browser operation but does not want to rebuild a site's frontend encryption environment in a separate JS-RPC service. - -The user chooses the business action and reproduces it once. Request-level inference selects the capture boundary and, when the evidence is unique, the background selects and retains the relevant business frame automatically. The user chooses a stack frame only when candidates are ambiguous; a function expression is an advanced fallback. The extension supplies the browser-only parts: the live document, lexical scope, non-extractable keys, dynamically generated IV/nonce/timestamp values, function receiver and authenticated session. - -This is deliberately not a promise to autonomously solve QR codes, CAPTCHA, MFA, device confirmation or every obfuscated application. Those steps remain visible human actions. The product goal is to remove avoidable environment reconstruction after the user reaches the real business operation. - -## Workflow - -```text -Real user operation - -> lightweight Recorder discovers a Trace and target operation - -> Deep Capture arms one crypto function or request breakpoint - -> Chromium pauses at the next real invocation - -> call frames become visible immediately - -> local / closure / module scopes are collected in parallel - -> shared stack hints and CDP metadata rank page business frames - -> a pure frame becomes a business closure; a send/DOM frame becomes a request transaction - -> function object + receiver + fixed call-frame arguments stay inside the live document - -> page resumes - -> plaintext maps to formal parameters or matching page controls - -> a request transaction captures the target envelope without sending it - -> extension, Yakit or Yak invokes the page callable with new JSON arguments - -> dynamic browser behavior and server validation remain real -``` - -The Recorder is the discovery/index layer. It records bounded interactions, requests, Beacon/WebSocket/Worker/MessagePort activity, unified WebCrypto/CryptoJS/JSEncrypt/sm-crypto/node-forge crypto calls, transforms, Trace membership, exact value links and explicitly correlated channel links. A recorded-call callable replays one eligible stateless or receiver-bound primitive by replacing its named data argument while retaining the original function, receiver and fixed argument template. Stateful sessions remain evidence and are promoted to their enclosing business closure. - -Deep Capture is the runtime/context layer. It captures a business function from a paused lexical environment, so one business-closure callable may preserve several internal crypto calls, closure variables, key promises, dynamic parameters and serialization steps. If the closest common business function also reads DOM controls, builds the request and calls Fetch/XHR/Beacon/Form, the same frame is retained as a `request-transaction` instead of being skipped in favor of an outer click handler. Both sources use the same registry and execution protocol while retaining distinct provenance and input-slot metadata. - -## Chromium implementation - -The background service uses `chrome.debugger` and these Chrome DevTools Protocol domains: - -- `Runtime` resolves the live wrapper function and reads object properties; -- `Debugger` enables pauses, function-call breakpoints, call frames, scopes and `evaluateOnCallFrame`; -- `DOMDebugger` installs a one-shot XHR/fetch URL breakpoint; -- `Network` prepares the session for later request correlation without intercepting traffic in this phase. - -Crypto capture does not depend on a source `debugger` statement. Production minifiers may remove that statement, and page CSP may block dynamic code construction. Instead, the recorder exposes the exact installed adapter or communication-boundary wrapper by its opaque `wrapperHandleId`. The background sets `Debugger.setBreakpointOnFunctionCall` on that object and removes the breakpoint on the first pause. Request-only unknown code can still use a bounded XHR/fetch URL breakpoint. - -Chrome may omit `callFrame.url` for ESM/module frames. The service therefore maintains a per-tab, 4,096-entry LRU-style `Debugger.scriptParsed` index and resolves the frame source from `location.scriptId`. This makes dynamically named ESM chunks first-class capture targets without scanning a bundler cache or exposing their exports on `window`. - -Request capture uses `DOMDebugger.setXHRBreakpoint` with a bounded URL substring. It is also one-shot. - -The current implementation supports Chromium main documents. Firefox does not request `debugger`, does not advertise Deep Capture Bridge capabilities and continues to provide Recorder-created callables. - -## Pause control plane - -A paused page cannot execute `scripting.executeScript`. Status, keepalive, resume, detach and callable creation must therefore never depend on an injected document probe. - -During a pause, target authorization uses only: - -- the grant's tab/frame/document/origin tuple; -- `tabs` and `webNavigation` state; -- extension session storage owned by the background; -- CDP commands on the already attached target. - -Page execution is used only before the pause to install/resolve a target function and after the pause to list, invoke or delete retained callables. This separation prevents the debugger control plane from deadlocking on the page it controls. - -## Two-stage collection - -The pause event publishes a stack skeleton before reading scope properties. This gives UI and Bridge clients an immediately observable `paused` state and lets them extend the deadline. Scope collection then fills the first eight frames in parallel. - -Current bounds are: - -| Resource | Bound | -| --- | ---: | -| Pause watchdog | 45 seconds | -| Call frames | 14 | -| Frames with scope expansion | 8 | -| Scopes per frame | 6 | -| Variables per scope | 48 | -| Variable preview | 512 characters | -| Expandable variable detail | 4,096 characters per variable | -| Expandable detail per scope | 16,384 characters | -| Page callable arguments | 64 JSON values | -| Function expression | 4,096 characters | - -The extension UI sends keepalive every 10 seconds while paused. Yakit uses `browser.deep_capture.keepalive` as its paused-state poll. If all control surfaces disappear, the alarm watchdog resumes the page automatically. - -Every frame carries an explicit `sourceKind`: `extension-hook`, `page`, or `library`. Exact recorder/debugger wrapper names and extension URLs are classified as extension hooks; dependency/runtime URLs are classified as libraries; remaining frames are page code. Request-level inference contributes bounded common-ancestor hints from multiple source stacks. The background combines those hints with frame depth, CDP script identity, function location and risk inspection; the UI cannot supply trusted source metadata. Options and Yakit display the labels and reasons, and prevent an extension hook or dependency frame from being captured as a business callable. Scope rows are keyboard-operable expanders: the list keeps a compact preview, while the expanded block shows a bounded value or function-source detail with copy actions. This makes injected wrappers visibly different from application functions without exporting unbounded debugger data. - -## Unified page callable - -`browser.callable.create` with `source: deep-capture` has three explicit strategies. `selected-frame` resolves a pure function from the stored current call frame and rejects network/DOM/navigation/storage side effects. `request-transaction` retains the closest request-building business frame and its bounded request contract. `expression` remains an advanced fallback and passes the pure-function inspection gate. Client-provided source URLs and line numbers are not accepted. The returned function object and its frame receiver are placed in the shared `BrowserPageCallable` registry keyed by an opaque UUID. Recorder-created calls use the same registry with `source: recording`. Only metadata crosses the extension boundary: - -- callable ID, name and kind; -- ordered input slots and output type/encoding; -- function name; -- source URL and line; -- recording/Trace/event provenance when available; -- creation time; -- for a request transaction, expected method, URL reference, output destinations and allowed boundary kinds; -- `document` lifecycle. - -Formal parameter names are recovered from bounded function source, including parameters after the first default value, and become ordered input slots. Fixed parameter values and `this` are retained by reading the named parameters from the actual CDP call frame; the debugger evaluation wrapper's `arguments` object is never used as business input. Options may correlate those names with values already present in the authorized paused scope to initialize a local replay Body. That short-lived sample never enters Bridge payloads, audit records, callable metadata or profile storage. - -A request transaction exposes one logical `body` input. Execution snapshots bounded form controls and DOM mutations, maps object fields to matching input names/IDs, and temporarily replaces Fetch, XHR send, Beacon and Form submit boundaries. Exactly one request must match the configured method and URL after resolving relative URLs against the current document. The body must contain every inferred destination, such as `body.encryptedData`, `body.encryptedKey` and `body.encryptedIv`. The real transport is never called; controls and observed DOM mutations are rolled back in `finally`. Multiple requests, another URL, an unsupported/file body, timeout, over-budget data or missing fields fail closed. Ordinary business closures also receive runtime transport guards so a transitive helper cannot silently send a request that shallow source inspection missed. - -The registry does not export closure bindings, `CryptoKey` material or the function source. `browser.callable.execute` calls the retained function in the MAIN world and returns a bounded structured result. ArrayBuffer and typed-array results are normalized to byte metadata plus Base64. Execution results are lossless within the 8 MiB string/byte, 100,000-node and depth-32 bounds; cycles, functions, symbols and oversized structures fail explicitly. Only UI previews are truncated. - -Page callables are the execution primitive used by the [Browser Transform Gateway](BROWSER_TRANSFORM_GATEWAY.md). Deep Capture discovers and retains the real business function; a Pipeline v2 profile reads plaintext request/response context, invokes one or more callables and writes explicit results back to the wire packet. - -Navigation, reload or document destruction removes the registry naturally. Explicit deletion removes one callable. Callable IDs are not portable credentials. - -## Authorization and lifecycle - -Deep Capture adds three independent scopes: - -| Scope | Allows | -| --- | --- | -| `browser.debugger.read` | Read status, call frames and scopes | -| `browser.debugger.control` | Attach, arm, keep alive, resume, detach and capture a function from a paused frame | -| `browser.callable.execute` | Create, execute and delete live-document page callables | - -Remote calls remain bound to the active grant's tab, main frame, document, origin and expiry. A grant cannot control a local or different grant's debugger session. Grant replacement, expiry and revocation detach sessions owned by that grant. Tab closure removes session state. Chrome DevTools and an extension debugger may compete for the same target; the UI reports the attach/detach failure rather than silently changing targets. - -## Real acceptance fixture - -The production E2E fixture uses a local authenticated page with: - -- native WebCrypto rather than a string mock; -- non-extractable AES-GCM and HMAC keys imported inside a closure; -- a local `buildLoginEnvelope` function that is not placed on `window`; -- dynamic timestamp, nonce and IV values; -- encrypted account/password JSON; -- an HMAC over envelope fields; -- server-side HMAC verification and AES-GCM decryption. - -The test records a real operation containing AES-GCM and HMAC, infers their common `buildLoginEnvelope` ancestor, pauses on the earliest confirmed crypto source, automatically captures the selected frame, restores both `password` and defaulted `account` parameters, generates `body.password` and `body.account` bindings from the paused sample, and executes the complete local Pipeline. Independent server validation also invokes the closure with new credentials, asserts different nonce/IV values and accepts the generated envelope. A hash stub or a hard-coded frontend demo does not satisfy this acceptance criterion. - -A second real-browser fixture covers the mixed AES + RSA request transaction at `127.0.0.1:82`. Three exact output links must select `sendDataAesRsa`, not its outer `onclick`. The test supplies new username/password values through the page controls, captures `encryptedData`, `encryptedKey` and `encryptedIv`, proves that neither deep-capture recovery nor callable replay added a browser request, and sends the captured envelope independently to the fixture server for acceptance. - -## Known limits - -- Chromium Deep Capture only; Firefox remains on recording and recorded-call page callables. -- Main document only in the current phase. Cross-frame debugging needs an explicit CDP target/session design rather than silently reusing frame grants. -- Source-map remapping is not implemented; URLs and generated line/column values come from CDP. -- Highly optimized, native, WASM-heavy or deliberately anti-debugging applications may expose incomplete names or scopes. -- Runtime transport interception covers dynamic global Fetch, XHR, Beacon and Form boundaries. A function that captured a private transport reference before interception, sends inside another Worker/realm, performs unconditional direct navigation, or mutates storage through an unobserved helper is not claimed as safely automatic; the current system must block on detected evidence or report the failed/stale transaction. -- DOM rollback is bounded and best-effort. It is not a general browser transaction or a replacement for a disposable test profile. -- The tester may need to select a function-valued scope variable or use the advanced in-scope expression when an anonymous or optimized frame cannot be resolved uniquely. -- The callable intentionally stays document-bound. Portable code generation requires a separate reviewed artifact model and cannot assume captured closure/key objects are serializable. - -References: [Chrome Debugger API](https://developer.chrome.com/docs/extensions/reference/api/debugger), [CDP Debugger domain](https://chromedevtools.github.io/devtools-protocol/tot/Debugger/), and [CDP DOMDebugger domain](https://chromedevtools.github.io/devtools-protocol/tot/DOMDebugger/). diff --git a/docs/ENTERPRISE_POLICY.md b/docs/ENTERPRISE_POLICY.md deleted file mode 100644 index b865165..0000000 --- a/docs/ENTERPRISE_POLICY.md +++ /dev/null @@ -1,38 +0,0 @@ -# Enterprise Deployment - -The extension includes `managed-storage-schema.json`. Managed values are read-only and rechecked by background command handlers, not only reflected in disabled UI controls. - -Supported policies: - -| Key | Type | Effect | -| --- | --- | --- | -| `bridgeTransport` | `native` or `websocket` | Locks transport. | -| `bridgeEndpoint` | string | Locks the explicit loopback WebSocket endpoint. | -| `nativeHost` | string | Locks the Native Messaging host name. | -| `autoConnect` | boolean | Locks startup connection behavior. | -| `disableWebSocket` | boolean | Requires Native Messaging. | -| `floatingPanelEnabled` | boolean | Enables or disables the page panel. | -| `maxGrantMinutes` | integer, 5-1440 | Caps every grant even if the UI requests longer. | -| `grantAllowedOrigins` | origin array | Rejects grants containing any other origin. | -| `allowProgramEval` | boolean | Can prohibit the independent program Eval scope. | - -Example managed policy values: - -```json -{ - "bridgeTransport": "native", - "nativeHost": "com.yaklang.browser_agent", - "autoConnect": true, - "disableWebSocket": true, - "maxGrantMinutes": 60, - "grantAllowedOrigins": ["https://security-lab.example"], - "allowProgramEval": false, - "floatingPanelEnabled": true -} -``` - -Chrome/Edge administrators distribute these values using the platform's extension managed-storage policy and can force-install the signed extension ID. Firefox administrators use the `3rdparty.Extensions` policy for the add-on ID `browser-agent@yaklang.com`. Native Host registration remains an operating-system deployment step; see [native-host/README.md](../native-host/README.md). - -The Chrome enterprise package prefers User Scripts MAIN for CSP-compatible page execution and retains the packaged injected bridge as a fallback when User Scripts is unavailable. Administrators should enable User Scripts for the extension in managed Chrome deployments when strict-site CSP execution is required; the fallback remains suitable for explicitly managed sites whose CSP permits it. - -Device pairing identities are deliberately excluded from managed storage. Pair each extension installation locally through Yakit. The extension keeps its non-extractable private key in IndexedDB and Yak stores only the approved public device identity, so a broadly readable policy backend never becomes a credential store. diff --git a/docs/FRONTEND_CRYPTO_GENERALIZATION_ROADMAP.md b/docs/FRONTEND_CRYPTO_GENERALIZATION_ROADMAP.md deleted file mode 100644 index c6fd0c2..0000000 --- a/docs/FRONTEND_CRYPTO_GENERALIZATION_ROADMAP.md +++ /dev/null @@ -1,615 +0,0 @@ -# 前端密码能力通用化重构与适配器路线 - -> 状态：G0–G4 已完成并通过真实浏览器/独立验证器验收；G5 按真实样本继续推进 -> -> 更新时间：2026-07-21 -> -> 关联文档：[`AUTO_PROFILE_INFERENCE_ARCHITECTURE.md`](AUTO_PROFILE_INFERENCE_ARCHITECTURE.md)、[`DEEP_CAPTURE_ARCHITECTURE.md`](DEEP_CAPTURE_ARCHITECTURE.md)、[`BROWSER_TRANSFORM_GATEWAY.md`](BROWSER_TRANSFORM_GATEWAY.md)、[`study.md`](study.md) - -## 1. 结论 - -当前实现的**数据模型、请求推断和 G4 高价值协议覆盖是通用的；WASM、流式协议与长尾生态仍需由真实样本继续驱动**。 - -现有靶场体验顺滑，主要因为它同时满足了三个有利条件： - -1. 使用全局可访问的 `window.CryptoJS` 或 `window.JSEncrypt`； -2. 加密后通过常规 Fetch/Form 请求发送； -3. 密码调用输出可以和请求字段建立精确值关联。 - -生产代码并没有依赖 `127.0.0.1:82`、`/encrypt/aes.php`、`/encrypt/rsa.php`、固定用户名、固定密码或固定业务字段。请求字段推断也已经支持 JSON、Form、Header、Query 和完整 Body。因此当前实现不是为靶场硬编码的结果。 - -但“没有靶场硬编码”不等于“已经覆盖真实世界”。当前 MAIN-world 录制器通过有界 manifest 为以下可访问对象安装语义 Hook： - -- 当前页面 Realm 的 `SubtleCrypto`； -- `CryptoJS`、`JSEncrypt`、`sm-crypto` 与 `node-forge`； -- `jsrsasign` 的 Signature/JWS/JWT/JWK； -- 页面显式暴露的 `jose` 高层 builder 与 verify/decrypt 函数。 - -没有全局导出的 ESM/Webpack 闭包、Worker 内密码运算、WASM 和完全未知的业务封装不会通过侵入 bundler cache 强行发现；它们继续走请求/消息边界、WebCrypto、证据图和 Deep Capture 业务闭包恢复。这是正式的通用路径，不是失败后的临时兜底。 - -因此本轮重构采用以下产品判断： - -> 已知库适配器是语义加速器，不是产品能力的地基。请求与消息边界、业务函数恢复、文档绑定 callable 和服务端认可的真实回放，才是通用能力的地基。 - -最终验收不是“界面显示识别到 AES/RSA”，而是： - -```text -用户执行一次真实操作 - -> 插件定位明文、页面业务调用和线上目标 - -> 已知库时给出准确算法语义，未知库时仍能定位业务封装 - -> 页面保留 key / IV / nonce / receiver / closure / WASM 状态 - -> Yakit Web Fuzzer 编辑明文 - -> 浏览器生成真实线上报文 - -> 独立服务端成功解密、验签或接受请求 -``` - -算法名称可以暂时未知，业务链路不能因此不可用。 - -## 2. 重构目标与非目标 - -### 2.1 目标 - -- 支持全局库、打包闭包、混淆函数、Worker 消息边界和 WASM 外围业务函数； -- 已知密码库接入同一 adapter contract，不再把逻辑堆入 MAIN-world 录制器； -- 未知库也可以从请求/消息边界进入 Deep Capture，恢复上层业务 callable； -- 自动 Profile 以请求为中心，保留 AES + RSA + HMAC + timestamp 等同一业务上下文； -- 页面秘密始终留在页面对象、闭包、CryptoKey 或 WASM 内存中，不通过协议导出； -- 适配器安装、事件归一化、证据建图、AI 分析和 Profile 执行各自独立； -- 使用随机化、跨打包形态的真实服务端夹具证明没有按图索骥； -- 在录制开启时保持有界开销，录制停止后完整恢复页面 API 且不存在后台轮询。 - -### 2.2 非目标 - -- 不追求穷举所有 JavaScript 密码库； -- 不要求先还原算法、密钥或混淆源码才能使用明文网关； -- 不把页面 key、PEM、CryptoKey、闭包变量或 WASM 内存导出到扩展、Yak 或 AI； -- 不在页面主线程进行全量源码搜索、全局对象枚举或 AST 扫描； -- 不为某个站点、接口路径、字段名或靶场流程维护特殊规则； -- 不保留旧 provider 枚举、旧录制协议或旧适配器目录的迁移兼容层。 - -## 3. 四层通用架构 - -```text -L0 业务边界探针 - Fetch / XHR / Form / sendBeacon / WebSocket / Worker / MessagePort / Navigation - | - | 有界输入输出、调用顺序、同步/异步栈、值关联 - v -L1 通用运行时边界 - WebCrypto / random / encoding / WebAssembly 装载 / serializer 边界 - | - | 原生算法元数据、TypedArray 形态、opaque object - v -L2 已知语义适配器 - CryptoJS / JSEncrypt / sm-crypto / node-forge / jsrsasign / jose / sodium ... - | - | 参数角色、模式、padding、state model、可复跑能力 - v -L3 未知业务函数恢复 - 请求断点 -> 页面业务帧排序 -> closure callable -> 自动 Profile -``` - -四层不是按顺序全部执行的流水线。L0 始终提供兜底证据；L1/L2 提供更强语义和更精确的断点；L3 在低层 primitive 不足、库不可见或业务封装复杂时恢复完整现场。 - -### 3.1 L0：业务边界是最低保证 - -请求和消息边界回答三个最重要的问题： - -1. 哪段值真正离开了页面； -2. 它被写入 Body、字段、Header、Query、WebSocket frame 还是 Worker 消息； -3. 哪个页面调用链在边界之前构造了它。 - -现有 Fetch/XHR/Form/WebSocket 继续保留，并补齐： - -- `navigator.sendBeacon`； -- `Worker.prototype.postMessage`； -- `MessagePort.prototype.postMessage`； -- `SharedWorker.port` 消息边界； -- 有界同步栈和可用时的异步栈来源； -- TypedArray、ArrayBuffer、Blob、FormData 和 transferable 的结构化摘要； -- 同一 Trace 内从输入、消息到请求的精确/归一化值关联。 - -页面侧边界看不到 Worker 内部每一步是事实，不应伪装成已识别。即使 Worker 内部无法安装密码适配器，插件仍可关联“页面明文消息 -> Worker 返回值 -> 请求字段”，并以消息边界或调用 Worker 的页面业务函数作为 callable 捕获入口。 - -Service Worker 内部运算不属于普通页面 MAIN world。第一阶段只保证通过 `webRequest` 和页面消息/请求边界观察真实线上结果；更深的 Worker/Service Worker 调试目标支持需要独立评估 CDP Target 生命周期，不能和页面适配器混为一个实现。 - -### 3.2 L1：通用运行时边界 - -首批运行时探针包括： - -- WebCrypto `SubtleCrypto`； -- `crypto.getRandomValues` 和 `randomUUID` 的调用关系摘要，不记录随机原值； -- `TextEncoder` / `TextDecoder`、Base64、Hex 等有界编码链； -- `WebAssembly.instantiate` / `instantiateStreaming` 的模块与实例身份摘要； -- 请求边界处的 JSON、Form、Query 和 Header 结构化解析。 - -不得全局 Hook 每一次 `JSON.stringify`、`encodeURIComponent` 或遍历所有 WASM exports。高频通用函数只在请求边界归一化，或在已确定的 Trace/Deep Capture 窗口内按需观察，避免让正常页面承担持续成本。 - -WASM 的第一目标不是反编译算法，而是保留调用它的页面业务 wrapper、输入输出关联和实例生命周期。只要该 wrapper 能在原页面复跑，明文网关就不需要导出 WASM 内存或重写算法。 - -### 3.3 L2：已知语义适配器 - -适配器负责把“某个函数被调用”解释成统一语义： - -- provider/adapter 身份； -- symmetric、asymmetric、digest、MAC、signature、KDF 或 key-management family； -- data、key、iv、nonce、aad、signature、options 等参数角色； -- algorithm、mode、padding、input/output encoding； -- stateless、receiver-bound、stateful-session、streaming 或 async-ready 状态模型； -- 是否可以安全保留原函数、receiver 和参数模板作为 recorded-call callable。 - -适配器不负责请求字段推断、UI 文案、AI prompt、Profile 编译或 Bridge RPC。新增库不应修改这些下游层。 - -### 3.4 L3：未知业务函数恢复 - -“不知道是哪一个库”不能成为终点。通用回退流程是： - -```text -请求/消息边界已定位 - -> 武装下一次相同边界 - -> 用户重复一次真实操作 - -> 立即发布有界调用栈 - -> 排除 extension hook 和已知依赖 frame - -> 结合参数相关性、请求接近度、源码位置、同步/异步父栈给业务 frame 排序 - -> 捕获完整业务 closure callable - -> 页面恢复 - -> 用短时样本做页面内回放 -``` - -页面函数叫 `encryptPayload`、`pack`、`request` 或 `_0x3f2a` 都不影响流程。AI 可以解释 frame 和参数语义，但只能返回引用既有 evidence 的候选补丁，不能生成并直接执行任意代码。 - -## 4. 适配器协议重构 - -### 4.1 删除封闭 provider 枚举 - -当前 `BrowserCryptoProvider` 是 `webcrypto | cryptojs | jsencrypt | forge | custom` 的封闭联合。继续添加库会迫使协议、归一化器、UI 和测试重复修改。 - -新协议使用有界 adapter ID 和稳定 provider kind： - -```ts -type BrowserCryptoProviderKind = - | "native" - | "library" - | "business" - | "wasm" - | "unknown" - -interface BrowserRecordingCrypto { - adapterId: string // 受限 slug，例如 "webcrypto"、"sm-crypto" - providerKind: BrowserCryptoProviderKind - family: BrowserCryptoFamily - operation: string // 适配器内部稳定 operation ID - algorithm?: string - mode?: string - padding?: string - inputEncoding?: BrowserPageCallableValueEncoding - outputEncoding?: BrowserPageCallableValueEncoding - state?: { - model: "stateless" | "receiver" | "session" | "stream" | "async-ready" - correlationId?: string - phase?: "create" | "init" | "update" | "final" | "one-shot" - } - key?: { - kind: "public" | "private" | "secret" | "unknown" - bits?: number - fingerprint?: string - } -} -``` - -`adapterId`、`operation` 和所有字符串必须限长并按字符集校验。UI 显示名来自扩展自带的 adapter manifest，不信任页面提供的 HTML 或展示文本。未知 ID 使用安全的纯文本回退标签。 - -Deep Capture 不再依赖 `CryptoJS.AES.encrypt` 这类展示字符串查找函数，而是绑定录制器已经保留的 wrapper handle： - -```text -adapterId + operation + wrapperHandleId + documentId -``` - -这样库被混淆、别名导出或方法名重复时，也不会武装错误函数。 - -### 4.2 统一 adapter contract - -```ts -interface PageCryptoAdapter { - manifest: { - id: string - displayName: string - providerKind: BrowserCryptoProviderKind - dynamic: boolean - } - discover(context: AdapterDiscoveryContext): AdapterTarget[] - install(target: AdapterTarget, host: AdapterHost): AdapterInstallation -} - -interface AdapterInstallation { - id: string - operations: InstalledOperation[] - restore(): void -} - -interface AdapterHost { - wrap(input: WrapOperationInput): InstalledOperation - emit(input: NormalizedCryptoCall): void - retain(input: RetainedCallInput): string | undefined - fingerprint(value: unknown): ValueEvidence[] -} -``` - -公共 `wrap` 基础设施必须统一处理： - -- 原 property descriptor、原函数和原 receiver； -- 同步返回、Promise resolve/reject 和库返回 `false/null` 的语义； -- re-entrancy 防护，避免适配器调用辅助方法时递归记录； -- 参数与输出大小预算； -- wrapper handle 与 Deep Capture 一次性断点； -- 页面后续替换函数时不覆盖页面的新值； -- restore 只恢复自己仍然拥有的 descriptor； -- 停止、清空、导航、grant 撤销和异常安装时的幂等清理。 - -适配器只能使用 host 提供的 evidence、emit 和 retain 能力，不各自维护事件队列、Trace、指纹算法或 callable registry。 - -### 4.3 状态型与流式 API - -不能把所有库都按 `encrypt(data, key) -> ciphertext` 的一次函数处理。 - -例如 node-forge 常见调用链是： - -```text -createCipher -> start -> update -> finish -> output -``` - -jsrsasign 的签名流程可能是： - -```text -new Signature -> init -> updateString/updateHex -> sign -``` - -这些调用需要同一 `correlationId` 和 phase 序列。只有满足以下条件才允许生成 recorded-call callable： - -- 可替换明文输入明确； -- 原 receiver/session 仍有效； -- 重放不会复用已经消费的流状态； -- 输出与请求目标存在 proven link； -- 调用没有网络、DOM、导航等额外副作用。 - -不满足时适配器只提供语义证据，并把候选标记为 `capture-required`，由 Deep Capture 保留上层一次性业务封装。 - -### 4.4 晚加载与打包形态 - -现有每秒扫描动态全局库的方式需要替换为有界调度： - -- 录制开始时立即检查一次已知全局路径； -- 捕获动态 `