- Add custom type_text tool that types a full string by internally
calling press_key for each character, turning N model round-trips
into 1. Dramatically speeds up text input in complex web apps.
- Move tool-specific usage rules from system prompt to individual
tool descriptions via augmentToolDescription() for better
organization and token efficiency.
- Add terminal failure handling instructions to system prompt
(Chrome connection errors, browser crashes, repeated errors)
with specific remediation steps.
- Add complex web app guidance (spreadsheets, rich editors) to
system prompt, recommending type_text + keyboard navigation.
- Fix augmentToolDescription key ordering so more-specific keys
(fill_form, click_at) match before shorter keys (fill, click).
- Remove non-existent tool references (scroll, type_text as MCP tool)
and add click_at hint for vision tool.
Implement the visual agent using the LocalAgentDefinition pattern:
- VisualAgentDefinition: Agent metadata for coordinate-based visual tasks
- delegateToVisualAgent.ts: Tool for semantic agent to delegate visual tasks
- Uses gemini-2.5-computer-use-preview-10-2025 model for Computer Use capability
The visual agent handles tasks requiring visual identification or precise
coordinate-based actions that cannot be done via the accessibility tree.
Gaurav Ghosh