Browserose MCP

MCP Tool for Agents to control the Google Chrome browser.

Browserose MCP is an MCP (Model Context Protocol) server that lets AI agents and IDEs control Google Chrome via Playwright, with full iframe support: snapshot and interact inside iframes (e.g. ALM SCORM players, embedded apps).

The toolset is sufficient for full end-to-end automation. You can complete entire flows—login, multi-step navigation, nested iframes, quizzes, and course completion—without manual steps or switching tabs. Use navigation, locators, frame selectors, coordinate fallbacks, and diagnostics as needed.

Requirements

• Node.js 18+
• Chrome installed (or set PLAYWRIGHT_MCP_USE_CHROMIUM=1 to use Chromium)
• Playwright browsers: npx playwright install chromium (or chrome if available)

Install and build

npm install
npx playwright install chromium
npm run build

Cursor / IDE configuration

Add to ~/.cursor/mcp.json (or project .cursor/mcp.json):

{
  "mcpServers": {
    "playwright-chrome": {
      "command": "node",
      "args": ["/absolute/path/to/Browserose-MCP/build/index.js"]
    }
  }
}

Restart Cursor after editing mcp.json.

Environment variables

• PLAYWRIGHT_MCP_HEADLESS — set to 1 or true to run the browser headless (default: headed so you see the window).
• PLAYWRIGHT_MCP_USE_CHROMIUM — set to 1 or true to use Playwright's Chromium instead of system Chrome.
• PLAYWRIGHT_MCP_VIEWPORT_MAXIMIZED — window opens maximized by default. Set to 0 or false to use a fixed size (see width/height below).
• PLAYWRIGHT_MCP_VIEWPORT_WIDTH — viewport width in pixels (default: 1280). Used only when maximized is off.
• PLAYWRIGHT_MCP_VIEWPORT_HEIGHT — viewport height in pixels (default: 800). Used only when maximized is off.

Examples: To use a fixed size instead of maximized, set PLAYWRIGHT_MCP_VIEWPORT_MAXIMIZED=0 and optionally PLAYWRIGHT_MCP_VIEWPORT_WIDTH=1920, PLAYWRIGHT_MCP_VIEWPORT_HEIGHT=1080.

Popup handler

When a link or button opens a new tab (e.g. Go to activity on IBM SkillsBuild opens the SCORM player in a new window), the MCP automatically attaches to that new tab. The browser context listens for the page event; when a new page is created, the internal page reference is updated so that all subsequent tool calls (navigate, click, snapshot, frame_probe, etc.) run in the new tab, not the original one. You do not need to switch tabs manually or use a separate “switch to tab” tool: click Go to activity, wait for the player to load, then continue with the same tools—they will already target the player tab. Implementation: in src/browser.ts, context.on("page", (newPage) => { page = newPage; }) runs on every new page (popup or target=_blank). Restart the MCP server (or Cursor) after pulling changes so the handler is active.

Tools (quick reference)

| Tool | Purpose | |------|--------| | browser_navigate | Go to URL | | browser_go_back / browser_go_forward | History | | browser_snapshot | Accessibility-like tree of the page; use includeFrames: true to include same-origin iframes | | browser_snapshot_frame | Snapshot a single iframe by selector (e.g. iframe, iframe#pplayer_iframe) | | browser_click | Click by ref; optional frameSelector for elements inside an iframe | | browser_type | Type text; optional ref, frameSelector, submit | | browser_type_locator | Type into element by role/text/css; optional frameSelector (omit = main page) | | browser_hover | Hover by ref; optional frameSelector | | browser_select_option | Select option(s) by ref; optional frameSelector | | browser_press_key | Press a key; optional frameSelector | | browser_screenshot | Page or iframe screenshot; optional frameSelector | | browser_click_at | Click at (x, y) relative to a frame's viewport (for canvas/cross-origin when refs fail); requires frameSelector, x, y | | browser_click_locator | Click by locator: role+name, text, or css. Optional: nth (0-based index), enabledOnly (first enabled match), scopeCss (within container). No snapshot needed. | | browser_list_clickables | List clickables in a frame. Requires frameSelector when targeting an iframe (e.g. content frame). Optional filters: role, name, text, css; enabledOnly; scopeCss (within container). Use to discover what to click. | | browser_frame_probe | Diagnostic: run inside frame to get url, title, readyState, button/clickable counts, textSample. If probe fails or counts=0, UI may be canvas. | | browser_frame_bbox | Get frame bounding box (x, y, width, height) in page coordinates. | | browser_click_at_rel | Click at relative (rx, ry) in 0..1 inside the frame (e.g. 0.5, 0.9 = center-bottom). Uses Playwright page.mouse.click. | | browser_frame_inventory | Inside frame: list child iframes (id, name, src, rect), canvas (rect), shadowHosts count, bodyRect. Use to see if UI is in nested iframe or canvas. | | browser_hit_test_rel | elementFromPoint at (rx, ry) in frame; returns tag, id, class, rect, pointerEvents, cursor (and iframe src/name). Confirms where clicks land. | | browser_click_at_rel_debug | Frame screenshot + text with page coords and in-frame pixel where click_at_rel(rx, ry) would click. | | browser_evaluate_click | Click element in frame via DOM .click() (frameSelector + css; optional nth). Bypasses visibility/actionability. Use when quiz SUBMIT or other button fails. | | browser_evaluate_click_by_text | Click by text content in frame (frameSelector + text; optional match exact/contains, scopeCss, nth). Bypasses visibility/overlays. Use when quiz options or labels don’t respond to locator/click_at_rel. | | browser_wait | Sleep (seconds) |

---

Tools reference and use cases

Detailed description of each tool and when to use it. Optional frameSelector uses >> for nested iframes (e.g. iframe#a >> iframe#b).

Navigation

| Tool | What it does | Use cases | |------|----------------|-----------| | browser_navigate | Opens a URL in the current tab. | Starting a flow, opening login pages, course URLs, or any target site. | | browser_go_back | Goes back in history. | Undoing a navigation or returning from a redirect. | | browser_go_forward | Goes forward in history. | Repeating a step after going back. |

Snapshot and ref-based interaction

These tools use an accessibility/DOM snapshot to get refs (e.g. s1e2). You then pass that ref to click, type, hover, or select. Best when the page (or iframe) is same-origin and has a normal DOM.

| Tool | What it does | Use cases | |------|----------------|-----------| | browser_snapshot | Captures an accessibility-like tree of the current page. Use includeFrames: true to include same-origin iframes. | Discovering structure and getting refs for main page and embedded frames in one call. | | browser_snapshot_frame | Snapshots a single iframe by selector. For chained selectors ( >> ), uses CDP when in-frame evaluate isn't possible. | Inspecting one frame's tree; getting refs for elements inside that frame (same-origin or when CDP can provide refs). | | browser_click | Clicks the element identified by ref. Optional frameSelector when the element is inside an iframe. | Buttons, links, checkboxes—any clickable from the snapshot. | | browser_type | Types text into the focused element or the element identified by ref. Optional frameSelector, submit (press Enter). | Text inputs, search boxes, login fields. | | browser_hover | Hovers over the element identified by ref. Optional frameSelector. | Opening dropdowns or tooltips before clicking. | | browser_select_option | Selects option(s) in a dropdown by ref. Optional frameSelector. | Select elements, language pickers, filters. | | browser_press_key | Sends a key (e.g. Enter, Tab, ArrowRight). Optional frameSelector to target a frame. | Submitting forms, keyboard navigation, escaping modals. |

Locator-based interaction (no snapshot)

These tools use Playwright locators (role+name, text, or css) and do not require a snapshot. They work in cross-origin iframes and are the main escape hatch when refs aren't available or snapshot is empty.

| Tool | What it does | Use cases | |------|----------------|-----------| | browser_click_locator | Clicks an element by locator: role+name, text, or css. Optional: nth (0-based index of match), enabledOnly (click first enabled match when several are disabled), scopeCss (resolve only within this container selector). | Cross-origin iframes, login buttons, SCORM "Next", quiz "SUBMIT" when multiple SUBMITs exist (use enabledOnly: true or nth), text inside a card (use scopeCss: ".quiz-card" to avoid sidebar). | | browser_type_locator | Types into an element found by role/text/css. Optional: nth (0-based index), scopeCss (within container). | Login fields, search boxes, any input when snapshot isn't used; when several inputs match, use nth or scopeCss. | | browser_list_clickables | Lists visible buttons/links in a frame. Optional: role, name, text, css (filter list to matches), enabledOnly (list only enabled), scopeCss (list only within container). | Discovering what's clickable; listing only "SUBMIT" buttons (role: "button", name: "SUBMIT") to see which is enabled; listing only clickables inside a quiz area (scopeCss). |

Customizing by situation: Use nth when multiple elements match (e.g. 4th button named "SUBMIT"). Use enabledOnly to click the first enabled match when the DOM has several disabled copies. Use scopeCss to restrict the search to a container (e.g. .quiz-card, [role="dialog"]) so you don't match the same text or role in the sidebar or another panel.

Coordinate-based interaction (escape hatch)

When snapshot and locators both fail (e.g. canvas, custom-rendered UI, or wrong frame depth), use coordinates relative to a frame.

| Tool | What it does | Use cases | |------|----------------|-----------| | browser_frame_bbox | Returns the frame's bounding box (x, y, width, height) in page coordinates. | Converting relative positions to absolute (x, y) for browser_click_at, or understanding frame position. | | browser_click_at | Clicks at pixel (x, y) relative to the frame's viewport. Requires frameSelector, x, y. | Canvas or non-DOM UI when you know exact coordinates (e.g. from a screenshot). | | browser_click_at_rel | Clicks at relative position (rx, ry) in [0..1] inside the frame (e.g. 0.5, 0.9 = center-bottom). Uses Playwright's mouse. | Clicking "bottom-right" or "center" of a frame when you don't have pixel coords; quick fallback for known layout. | | browser_click_at_rel_debug | Returns a screenshot of the frame and the exact page coordinates and in-frame pixel where click_at_rel(rx, ry) would click. | Debugging: confirm that (rx, ry) lands on the right element before using browser_click_at_rel. |

Diagnostics (finding the right frame / layer)

When a frame shows no buttons or empty text in the snapshot, the real UI is often in a child iframe, canvas, or shadow DOM. These tools help you find it.

| Tool | What it does | Use cases | |------|----------------|-----------| | browser_frame_probe | Runs a small script inside the frame: returns url, title, readyState, counts of buttons/clickables, and a short textSample. | Quick check: "Does this frame have any DOM?" If buttons: 0, clickables: 0, textSample: "", the visible UI is likely in a child iframe or canvas. | | browser_frame_inventory | Lists child iframes (id, name, src, rect), canvas elements (rect), count of shadow roots, and bodyRect. | When probe says "no content": find the real content frame (e.g. ALM SCORM's iframe#content-frame) or confirm the UI is canvas. Then extend the frame selector chain and use locators in that frame. | | browser_hit_test_rel | Uses elementFromPoint(rxwidth, ryheight) inside the frame; returns tag, id, class, rect, pointerEvents, cursor; for iframes, src/name. | Verify what element a relative point (rx, ry) hits—e.g. "Is (0.5, 0.92) really the Next button or an overlay?" |

Utility

| Tool | What it does | Use cases | |------|----------------|-----------| | browser_screenshot | Takes a screenshot of the full page or a specific iframe (frameSelector). | Visual verification, debugging layout, or feeding into vision models. | | browser_wait | Pauses for a given number of seconds. | Letting the page or iframe finish loading before snapshot or click. |

---

Use cases in practice

• Normal web automation (main page)

Use browser_snapshot (or browser_snapshot_frame with no/minimal nesting) to get refs, then browser_click, browser_type, browser_hover, browser_select_option with those refs. Optional browser_screenshot for verification.

• Login flows

Often on the main page: browser_type_locator and browser_click_locator with role/name or text (e.g. email → Continue → password → Log in). No snapshot required.

• Single iframe, same-origin

browser_snapshot_frame with frameSelector: "iframe#id" → get refs → browser_click / browser_type with the same frameSelector.

• Cross-origin or "empty" iframe

Snapshot may be empty or refs may not work. Use locators: browser_list_clickables with frameSelector to see what's there, then browser_click_locator and browser_type_locator with the same frameSelector and role/text/css.

• ALM / SCORM (nested iframes)

The visible lesson UI is often in a third-level iframe. If browser_frame_probe on iframe#pplayer_iframe >> iframe#modulePlayerIframe shows clickables: 0, run browser_frame_inventory on that chain; it will list child iframes (e.g. iframe#content-frame). Extend the chain to ... >> iframe#content-frame and use browser_list_clickables and browser_click_locator (e.g. role: "button", name: "Next") there. One-line takeaway: when the frame has no DOM content, use frame_inventory to find the real content iframe, then add it to the chain.

• Canvas or custom-rendered UI

If frame_inventory shows a large canvas and no useful iframe, or locators don't match: use browser_click_at_rel_debug to see where (rx, ry) lands, then browser_click_at_rel with adjusted (rx, ry), or browser_frame_bbox + browser_click_at with computed (x, y).

• Quizzes / multiple identical buttons

When several "SUBMIT" or "Next" buttons exist and only one is enabled, use browser_click_locator with enabledOnly: true (and same role/name) so the first enabled match is clicked. Or use browser_list_clickables with role: "button", name: "SUBMIT" (and optionally enabledOnly: true) to see indices, then click with nth. Use scopeCss (e.g. .quiz-card) to restrict to the current question card and avoid matching the sidebar.

• Debugging "click does nothing"

Check: (1) Correct frame? → browser_frame_probe and browser_frame_inventory. (2) Right element? → browser_list_clickables in that frame; browser_hit_test_rel to see what's under (rx, ry). (3) Right coordinates? → browser_click_at_rel_debug. (4) Multiple matches? → use nth or enabledOnly.

---

End-to-end: IBM SkillsBuild / ALM course

You can run the full course flow with only Browserose MCP tools: from the IBM page through login, learning plan, launching the activity, and completing lessons and quizzes.

1. Start from the IBM page and log in

• Navigate to the course or plan URL (e.g. https://skills.yourlearning.ibm.com/activity/PLAN-...).
• If redirected to login: browser_click_locator with text: "Log in with ibm" (or equivalent). On the IBM login page:
• browser_type_locator with role: "textbox", name: "IBMid", and the email.
• browser_click_locator with role: "button", name: "Continue".
• browser_type_locator with role: "textbox", name: "Password", and the password.
• browser_click_locator with role: "button", name: "Log in".
• Use browser_wait and browser_screenshot as needed to confirm the next page.

2. Open the learning plan and the module

• From the plan page: browser_click_locator with text: "Microcredential 1: Data Classification" (or the right section).
• Then browser_click_locator with text: "Classifying and Sourcing Data" (or the target module).
• Launch the activity: browser_click_locator with role: "button", name: "Go to activity".
• Wait for the player: Clicking Go to activity may open the SCORM player in a new tab. The MCP attaches to new tabs automatically (popup handler), so the next tool calls then run in the player tab. Use browser_wait (e.g. 5–8 seconds), then browser_frame_probe or browser_frame_inventory on iframe#pplayer_iframe (and then iframe#pplayer_iframe >> iframe#modulePlayerIframe if needed) until the content frame is present. The visible lesson UI is in the content frame:

iframe#pplayer_iframe >> iframe#modulePlayerIframe >> iframe#content-frame

3. Content frame: lessons and “Continue”

• Continue / Next (content):

browser_click_locator with frameSelector: "iframe#pplayer_iframe >> iframe#modulePlayerIframe >> iframe#content-frame", css: "button.continue-btn".

• Use browser_list_clickables with that frameSelector to discover buttons (e.g. “Next”, “Continue”). Use browser_screenshot with that frameSelector when you need to see what’s on screen.

4. Practice quiz: two-pass strategy

The fastest way to pass is: (1) first pass—answer arbitrarily, submit, read the correct answer from feedback, memorize it; (2) TAKE AGAIN; (3) second pass—answer with the memorized correct options and submit.

• Start or restart quiz:

browser_click_locator in the content frame with text: "START QUIZ" or text: "TAKE AGAIN".

• Select an option:

If locators like #qmc-X-label are covered by overlays, use browser_click_at_rel in the content frame with e.g. rx: 0.5, and ry roughly: first option ~0.48–0.52, second ~0.56–0.6, third ~0.64–0.68 (tune if layout differs). Or use browser_hit_test_rel / browser_click_at_rel_debug to confirm.

• Submit:

browser_click_locator in the content frame with role: "button", name: "SUBMIT", enabledOnly: true (so the active SUBMIT is clicked).

• Read feedback:

After submit, use browser_frame_probe on the content frame and read textSample (or use browser_screenshot) to get “Correct answer: …” and store it per question index (Q1, Q2, …).

• Next question:

browser_click_at_rel in the content frame with rx: 0.65, ry: 0.85 (NEXT button area). Repeat until the quiz is done (no more SUBMIT or you see completion).

• Second pass:

Click TAKE AGAIN, then for each question click the option that matches the stored correct answer (by ry band or by locator if the correct option text is clickable), SUBMIT, then click (0.65, 0.85) for NEXT until the quiz is complete.

5. After the quiz and finishing the module

• When the quiz is complete, use browser_click_locator in the content frame with css: "button.continue-btn" (or equivalent) to continue to the next lesson or close the module.
• Repeat the same pattern for further lessons and quizzes until the module/course is marked complete.

Summary: All steps—login, plan navigation, “Go to activity”, waiting for the player, lesson Continue, quiz (two-pass with feedback reading and TAKE AGAIN), and completion—can be done with the existing tools (navigate, click_locator, type_locator, list_clickables, frame_probe, frame_inventory, click_at_rel, screenshot, wait). No manual tab switching or external tools are required.

---

Using iframes (including cross-origin / ALM SCORM)

1. Navigate to a page that contains an iframe (e.g. ALM course page).
2. Call browser_snapshot with includeFrames: true to get the main page plus same-origin iframes, or call browser_snapshot_frame with frameSelector: "iframe" (or iframe#id) to get only that frame's tree.
3. Use the returned refs with browser_click, browser_type, etc., and pass the same frameSelector. For nested iframes use a chained selector with >> , e.g. iframe#pplayer_iframe >> iframe#modulePlayerIframe.

Example: click "Next" inside the first iframe:

• browser_snapshot_frame with frameSelector: "iframe" → get ref for the "Next" button (e.g. f1e2).
• browser_click with ref: "f1e2", frameSelector: "iframe".

Escape hatch (cross-origin / SCORM): For frames where snapshot fails, use Playwright locators directly (no AX/DOMSnapshot):

• browser_list_clickables with frameSelector: "iframe#pplayer_iframe >> iframe#modulePlayerIframe" → lists buttons/links with text and enabled/disabled.
• ALM SCORM: The visible lesson UI (e.g. "Next", "Learning objectives") lives in a third-level iframe. Use browser_frame_inventory on iframe#pplayer_iframe >> iframe#modulePlayerIframe to see child iframes; then chain to the content frame: iframe#pplayer_iframe >> iframe#modulePlayerIframe >> iframe#content-frame. Use that selector with browser_list_clickables and browser_click_locator (e.g. role: "button", name: "Next").
• browser_click_locator with the same frameSelector and role: "button", name: "Next" (or text: "Next") → clicks the element. Works because Playwright targets the frame's context directly.

Cross-origin / SCORM (AX tree empty): The server also uses a 3-tier snapshot for frames:

1. Tier A — CDP Accessibility.getFullAXTree (refs with backendDOMNodeId; click via box model).
2. Tier B — If AX is empty, CDP DOMSnapshot.captureSnapshot (refs with viewport coordinates; click via Input.dispatchMouseEvent).
3. Tier C — Use browser_screenshot with frameSelector, then browser_click_at with the same frameSelector and (x, y) to click by coordinates (e.g. canvas or when both AX and DOM snapshot fail).

---

License and author

• License: This project is open source. Use and modify it freely.
• Developer: ETTALBI OMAR