BrowserDebugger

【Skill】浏览器调试工具

公开

关注 0 Fork 0 Star 0

返回提交历史

XFEstudio/BrowserDebugger

Add media evidence capture and embedded browser guidance

c884aa1

XFE工作室室长 <mail@xfegzs.com>

提交于 2026-10-15 18:47

父提交 e651e5b

代码差异

7 个文件 +172 -5

Modified SKILL.md +1 -0

@@ -39,6 +39,7 @@ await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });

39 39

40 40

- 网页操作、文件选择、下载、弹窗和权限：按需读 [references/browser-actions.md](references/browser-actions.md)。

41 41

- 网络捕获、接口分析、请求重放和 DevTools 调试：按需读 [references/network-and-devtools.md](references/network-and-devtools.md)。

- 视频画面/字幕采集、嵌套 iframe、自定义字体、现有扩展会话事件分页和 C# WebView2：按需读 [references/media-and-embedded.md](references/media-and-embedded.md)。

42 43

43 44

先观察实际页面（截图、ARIA/DOM、当前 URL、标签页），再定位并操作。对 SPA、WebSocket、SSE 页面，以目标元素、响应或状态变化判断完成，避免把 `networkidle` 当成通用完成条件。监听响应、弹窗、下载和文件选择事件应放在触发动作之前。

44 45

Modified agents/openai.yaml +1 -1

@@ -1,6 +1,6 @@

1 1 interface:

2 2 display_name: "浏览器调试与抓包"

3 short_description: "完整浏览器操作、文件上传、跨标签页网络捕获与 API 调试"

3 short_description: "浏览器操作、上传、视频取证、跨标签页网络捕获与 API 调试"

4 4

  default_prompt: "使用 $browser-debugger 操作 Edge 浏览器，并从操作前开始记录网络请求、响应和 WebSocket。"

5 5

6 6 policy:

Modified package-lock.json +2 -2

@@ -1,12 +1,12 @@

1 1 {

2 2 "name": "browser-debugger-skill",

3 "version": "1.0.0",

3 "version": "1.1.0",

4 4 "lockfileVersion": 3,

5 5 "requires": true,

6 6 "packages": {

7 7 "": {

8 8 "name": "browser-debugger-skill",

9 "version": "1.0.0",

9 "version": "1.1.0",

10 10 "dependencies": {

11 11 "playwright-core": "1.64.1"

12 12 },

Modified package.json +1 -1

@@ -1,6 +1,6 @@

1 1 {

2 2 "name": "browser-debugger-skill",

3 "version": "1.0.0",

3 "version": "1.1.0",

4 4 "private": true,

5 5 "type": "module",

6 6 "engines": { "node": ">=22" },

Added references/media-and-embedded.md +50 -0

@@ -0,0 +1,50 @@

1 # 视频、嵌套页面与内置浏览器

3 ## 视频内容取证

先枚举 `page.frames()`，HTML5 播放器、字幕和题目可能位于不同层 iframe。不能从主文档没有 `<video>` 推断页面没有视频。导航时旧 iframe 可能暂时仍在：等待所选章节/文档身份与任务 iframe 同时更新，再读取新内容。

`scripts/media-evidence.mjs` 提供正常播放期间的画面与字幕取证，兼容跨域视频（使用浏览器截图，不依赖可能被污染的 canvas）：

9 ```javascript

10 var mediaTools = await import(pathToFileURL('C:/ABSOLUTE/SKILL/browser-debugger/scripts/media-evidence.mjs').href);

11 var mediaState = await mediaTools.inspectMedia(page);

12 // 先依据实际页面点击播放按钮；helper 不会自动修改播放状态。

13 var evidence = await mediaTools.captureMediaEvidence(page, {

14 outputDir: 'C:/ABSOLUTE/TASK/work/video-evidence',

15 seconds: 60, intervalSeconds: 10,

16 onSample: async sample => { /* 可按批调用用户配置的视觉 Agent */ }

17 });

18 ```

清单记录真实媒体时间、采集时间、播放器位置、来源、字幕活动 cue、截图和缺失原因。`duration=null` 表示浏览器尚未提供有限时长；不伪造时长。`inspectMedia` 读取所有媒体状态；采集只在媒体时间前进、画面就绪且没有 seek 时进行。截图时点可能比状态读取稍晚，不应当作逐帧精确时间码。读取媒体 URL 不等于已下载完整视频，blob URL 也不能在独立 HTTP 客户端中直接使用。

画面不能代替音轨。读取 textTracks 的活动字幕；网站字幕若由 DOM 渲染，观察实际选择器后另行读取。纯音频/讲解音轨需要真实转写或用户给出的字幕，不能让模型根据少量画面声称听完全文。视觉 Agent 的图片输入、端点和模型名应依据该供应商当前官方文档确认，缺少 API Key 时使用本地 mock 验证连接代码并报告未进行真实 API 联调。

如果题目或正文使用自定义字体，DOM 中可能是错误字符。对比浏览器截图；给视觉 Agent 原样截图，保留 DOM 文本仅作结构/选项标识，不能把乱码当作实际题意。模型建议、UI 填写、提交动作与站点确认是不同阶段；每一步以实际页面或响应判断，身份验证、未知题型、低置信度或缺少内容依据时保留现场。

26 ## 已有扩展浏览器会话

宿主已经提供已登录浏览器时优先复用其标签页和公开的 CDP capability。先阅读当前 capability 文档，不猜测方法签名。此类连接可能只覆盖一个标签页或当前 origin，不能当作 `openSession` 的浏览器级全目标记录器。

若接口提供事件 cursor/sequence，先建立基线，再触发操作；读取后以返回 cursor 继续分页，保留同一过滤条件，直到 `hasMore=false`。一旦返回 `truncated=true`，说明缓冲区丢失过事件，必须报告本次记录不完整。对子 iframe/Worker 的 `Target.attachedToTarget` 读取实际 session/target 标识后再路由命令，不猜 ID。只输出 HTTP/HTTPS 的有限 method、去敏 URL、状态、类型摘要；避免把 data URL、图片 base64 或凭据打印进对话。

32 ## C# / WebView2

34 可由应用在创建自己的 `CoreWebView2Environment` 前启用本机 CDP，并使用独立 userDataFolder：

36 ```csharp

37 var options = new CoreWebView2EnvironmentOptions {

38 AdditionalBrowserArguments = $"--remote-debugging-address=127.0.0.1 --remote-debugging-port={port}"

39 };

40 var environment = await CoreWebView2Environment.CreateAsync(null, profileDirectory, options);

41 await webView.EnsureCoreWebView2Async(environment);

42 using var playwright = await Playwright.CreateAsync();

43 var browser = await playwright.Chromium.ConnectOverCDPAsync($"http://127.0.0.1:{port}");

44 ```

`port` 从当前空闲端口分配，启动有竞争时处理失败，不固定暴露到局域网。不要读取其它个人浏览器 profile 的 Cookie。软件自有 WebView2 登录由用户在可见界面完成；发布包需要包含 Playwright 驱动，安装机器需要 WebView2 Runtime。

Playwright .NET 把 `EvaluateAsync<T>` 直接映射到 C# DTO 时，要验证 JSON 字段名映射，不能因为反序列化不匹配而把默认空列表当作网页无内容。可读取 `JsonElement` 后用明确的 `JsonSerializerOptions` 转换，并用真实浏览器集成测试覆盖。应用的去敏网络诊断日志通常只有 URL/状态，不能称为保存了请求/响应体的完整 HAR。

参考：[Playwright .NET WebView2](https://playwright.dev/dotnet/docs/webview2)、[WebView2 Environment Options](https://learn.microsoft.com/en-us/dotnet/api/microsoft.web.webview2.core.corewebview2environmentoptions)、[HTMLMediaElement](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement)。

Added scripts/media-evidence.mjs +82 -0

@@ -0,0 +1,82 @@

1 import { mkdir, appendFile } from 'node:fs/promises';

2 import path from 'node:path';

3 import { randomUUID } from 'node:crypto';

4 import { setTimeout as delay } from 'node:timers/promises';

6 // Read real playback state in every currently attached frame. Does not play, seek,

7 // change playbackRate, or fabricate the site's progress/heartbeat requests.

8 export async function inspectMedia(page) {

9 const media = [], errors = [];

10 const frames = page.frames();

11 for (let frameIndex = 0; frameIndex < frames.length; frameIndex++) {

12 const frame = frames[frameIndex];

13 try {

14 const items = await frame.locator('video,audio').evaluateAll(elements => elements.map((v, index) => ({

15 index, tag: v.tagName.toLowerCase(), title: v.title || document.title,

16 currentTime: Number.isFinite(v.currentTime) ? v.currentTime : 0,

17 duration: Number.isFinite(v.duration) ? v.duration : null,

18 paused: v.paused, ended: v.ended, seeking: v.seeking, readyState: v.readyState,

19 playbackRate: v.playbackRate, source: v.currentSrc || v.src,

20 width: v.getBoundingClientRect().width, height: v.getBoundingClientRect().height,

21 tracks: Array.from(v.textTracks || []).map(t => ({ kind: t.kind, language: t.language, mode: t.mode,

          activeCues: Array.from(t.activeCues || []).map(c => ({ startTime: c.startTime, endTime: c.endTime, text: c.text || '' })) })),

23 })));

24 media.push(...items.map(item => ({ frameIndex, frameUrl: frame.url(), ...item })));

25 } catch (error) { errors.push({ frameIndex, frameUrl: frame.url(), error: String(error.message) }); }

26 }

27 return { recordedAt: new Date().toISOString(), media, errors };

28 }

30 // Sample the actual rendered video and currently available cues while a page is

31 // playing normally. The caller chooses/starts playback through the observed UI.

32 // Each JSONL entry records media time, wall time, frame scope, and any omissions.

33 export async function captureMediaEvidence(page, options) {

34 const { outputDir, seconds = 60, intervalSeconds = 10, pollMs = 500, signal, onSample } = options ?? {};

35 if (!outputDir) throw new Error('outputDir is required.');

36 if (![seconds, intervalSeconds, pollMs].every(Number.isFinite) || seconds <= 0 || intervalSeconds < .1 || pollMs < 50)

37 throw new Error('seconds > 0, intervalSeconds >= 0.1 and pollMs >= 50 are required.');

38 const directory = path.resolve(outputDir);

39 await mkdir(directory, { recursive: true });

40 const runId = randomUUID();

41 const manifestPath = path.join(directory, `media-${runId}.jsonl`);

42 const last = new Map(), frameIds = new WeakMap();

43 let nextFrame = 0, samples = 0, images = 0;

44 const started = Date.now();

45 while (Date.now() - started < seconds * 1000) {

46 signal?.throwIfAborted();

47 const snapshot = await inspectMedia(page);

48 const frames = page.frames();

49 const evidence = [];

50 for (const item of snapshot.media) {

51 const frame = frames[item.frameIndex];

52 if (!frame || frame.isDetached() || frame.url() !== item.frameUrl) {

53 snapshot.errors.push({ frameIndex: item.frameIndex, error: 'Frame changed after inspection; sample omitted.' });

54 continue;

55 }

56 if (!frameIds.has(frame)) frameIds.set(frame, ++nextFrame);

57 const key = `${frameIds.get(frame)}:${item.index}:${item.source}`;

58 if (item.seeking || item.readyState < 2 || item.currentTime <= 0 ||

59 (last.has(key) && item.currentTime - last.get(key) < intervalSeconds)) continue;

60 const record = { ...item, image: null };

61 if (item.tag === 'video' && item.width > 0 && item.height > 0) {

62 const file = `video-${runId}-${String(images + 1).padStart(5, '0')}.jpg`;

63 try {

64 const element = frame.locator('video,audio').nth(item.index);

65 await element.screenshot({ path: path.join(directory, file), type: 'jpeg', quality: 80, timeout: 3000 });

66 record.image = file; images++;

67 } catch (error) { record.imageError = String(error.message); }

68 }

69 last.set(key, item.currentTime);

70 evidence.push(record);

71 }

72 if (evidence.length || snapshot.errors.length) {

73 const sample = { recordedAt: snapshot.recordedAt, evidence, errors: snapshot.errors };

74 await appendFile(manifestPath, `${JSON.stringify(sample)}\n`, 'utf8'); samples++;

75 if (onSample) await onSample(sample);

76 }

77 if (snapshot.media.length && snapshot.media.every(item => item.ended)) break;

78 const remaining = seconds * 1000 - (Date.now() - started);

79 if (remaining > 0) await delay(Math.min(pollMs, remaining), undefined, { signal });

80 }

81 return { manifestPath, samples, images, elapsedSeconds: (Date.now() - started) / 1000 };

82 }

Modified scripts/self-test.mjs +35 -1

@@ -4,6 +4,7 @@ import { createHash } from 'node:crypto';

4 4 import { mkdir, readFile, writeFile } from 'node:fs/promises';

5 5 import path from 'node:path';

6 6 import { openSession } from './browser-session.mjs';

7 import { inspectMedia, captureMediaEvidence } from './media-evidence.mjs';

7 8

8 9 // Entirely local integration fixture. No external website, account, or upload service.

9 10 const root = path.resolve(process.argv[2] ?? path.join(process.cwd(), 'work', `self-test-${Date.now()}`));

@@ -142,6 +143,39 @@ try {

142 143 assert.ok((await session.cdp('Browser.getVersion')).product);

143 144

  console.log('Verified file input, native chooser event, directory upload, download, screenshot, DOM and Debugger commands.');

144 145

146 const mediaFrame = page.frames().find(frame => frame.url().endsWith('/frame'));

147 await page.locator('iframe').evaluate(frame => { frame.style.height = '300px'; frame.style.width = '500px'; });

148 await mediaFrame.evaluate(async () => {

149 const canvas = document.createElement('canvas'); canvas.width = 320; canvas.height = 180;

150 const context = canvas.getContext('2d'); let tick = 0;

151 const paint = setInterval(() => { context.fillStyle = '#007e75'; context.fillRect(0, 0, 320, 180);

152

      context.fillStyle = 'white'; context.font = '22px sans-serif'; context.fillText(`Media evidence ${++tick}`, 15, 90); }, 50);

153 const stream = canvas.captureStream(10), chunks = [];

154 const recorder = new MediaRecorder(stream, { mimeType: 'video/webm' });

155 recorder.ondataavailable = event => chunks.push(event.data);

156 const stopped = new Promise(resolve => recorder.onstop = resolve);

157 recorder.start(); await new Promise(resolve => setTimeout(resolve, 2200)); recorder.stop(); await stopped;

158 clearInterval(paint); stream.getTracks().forEach(track => track.stop());

159 const video = document.createElement('video'); video.width = 320; video.height = 180; video.muted = true;

160 video.src = URL.createObjectURL(new Blob(chunks, { type: 'video/webm' })); document.body.append(video);

161 const track = video.addTextTrack('captions', 'Local captions', 'en'); track.mode = 'showing';

162 track.addCue(new VTTCue(0, 10, 'media-evidence-caption')); await video.play();

163 });

164 const mediaState = await inspectMedia(page);

165 assert.equal(mediaState.media.length, 1);

166 assert.ok(mediaState.media[0].frameUrl.endsWith('/frame'));

167 assert.equal(mediaState.media[0].playbackRate, 1);

168

  const evidence = await captureMediaEvidence(page, { outputDir: path.join(root, 'media'), seconds: 1.4, intervalSeconds: .3, pollMs: 100 });

169 assert.ok(evidence.images >= 2, `Expected progressing media frames, got ${evidence.images}`);

170 const manifest = await readFile(evidence.manifestPath, 'utf8');

171 assert.ok(manifest.includes('media-evidence-caption'));

172 const snapshots = manifest.trim().split('\n').flatMap(line => JSON.parse(line).evidence);

173 assert.ok(snapshots.at(-1).currentTime > snapshots[0].currentTime);

174

  for (const snapshot of snapshots) if (snapshot.image) assert.ok((await readFile(path.join(root, 'media', snapshot.image))).length > 0);

175 const cancelled = new AbortController(); cancelled.abort();

176

  await assert.rejects(captureMediaEvidence(page, { outputDir: path.join(root, 'media'), signal: cancelled.signal }), { name: 'AbortError' });

177 console.log('Verified nested-frame media inspection, advancing video screenshots/captions, and cancellation.');

178

145 179 attached = await openSession({ endpoint: session.endpoint, workDir: root, outputDir: path.join(root, 'attached') });

146 180 attached.page = attached.context.pages().find(page => page.url() === `${base}/`);

147 181 assert.ok(attached.page, 'Select the actual fixture tab rather than browser-internal pages or the popup.');

@@ -191,7 +225,7 @@ try {

191 225 const report = {

192 226 passed: true, testedAt: new Date().toISOString(), browser: session.version.Browser,

193 227 outputDir: root, requests: requests.length, targets: [...new Set(requests.map(r => r.targetType))],

194

    checks: ['multiple files', 'file chooser', 'directory upload', 'server upload response', 'download', 'screenshot', 'DOM', 'Debugger', 'raw CDP', 'existing browser attach/disconnect', 'fetch', 'XHR', 'redirect hops', 'network failure', 'response body', 'ExtraInfo headers', 'large-body omission', 'dedicated worker', 'shared worker', 'service worker', 'cross-origin iframe', 'cross-target request completion', 'popup', 'WebSocket sent/received', 'SSE', 'console', 'all-target HAR', 'native HAR', 'live protocol schema'],

228

    checks: ['multiple files', 'file chooser', 'directory upload', 'server upload response', 'download', 'screenshot', 'DOM', 'Debugger', 'raw CDP', 'existing browser attach/disconnect', 'fetch', 'XHR', 'redirect hops', 'network failure', 'response body', 'ExtraInfo headers', 'large-body omission', 'dedicated worker', 'shared worker', 'service worker', 'cross-origin iframe', 'cross-target request completion', 'popup', 'WebSocket sent/received', 'SSE', 'console', 'all-target HAR', 'native HAR', 'live protocol schema', 'nested-frame media state', 'video frames and active captions', 'media cancellation'],

195 229 };

196 230 await writeFile(path.join(root, 'test-result.json'), `${JSON.stringify(report, null, 2)}\n`, 'utf8');

197 231 console.log(JSON.stringify(report, null, 2));