mirror of
https://github.com/google-gemini/gemini-cli.git
synced 2026-05-13 05:12:55 -07:00
Add experimental built-in visualization tooling
This commit is contained in:
Dmitry Lyalin
@@ -0,0 +1,245 @@
|
||||
# Built-in Native Visualization Tool for Gemini CLI (V1)
|
||||
|
||||
## Summary
|
||||
|
||||
Implement a first-class built-in tool `render_visualization` (no MCP) that
|
||||
supports `bar`, `line`, and `table` outputs in the terminal using Ink-native
|
||||
rendering. The model will be guided to use this tool during natural-language
|
||||
reasoning loops (research -> normalize -> visualize), so users do not need to
|
||||
format input data manually.
|
||||
|
||||
This design directly supports these scenarios:
|
||||
|
||||
- "Fastest 3 BMW + comparative 0-60 chart"
|
||||
- Follow-up in same session: "BMW models per year for last 5 years" with dynamic
|
||||
data volumes and consistent cross-platform rendering.
|
||||
|
||||
## Goals and Success Criteria
|
||||
|
||||
- User asks a natural-language question requiring comparison/trend display.
|
||||
- Model can:
|
||||
|
||||
1. Gather data via existing tools (web/file/etc),
|
||||
2. Transform raw findings into visualization schema,
|
||||
3. Call `render_visualization` correctly,
|
||||
4. Render readable inline chart/table in CLI.
|
||||
|
||||
- Works across modern terminals on macOS, Linux, Windows.
|
||||
- Follow-up turns in same session continue to use tool correctly without user
|
||||
schema knowledge.
|
||||
|
||||
## Product Decisions (Locked)
|
||||
|
||||
- V1 scope: one tool with 3 visualization types: `bar | line | table`.
|
||||
- UI surface: inline tool result panel only.
|
||||
- Rendering stack: pure Ink + custom renderers (no chart library dependency in
|
||||
v1).
|
||||
- Invocation behavior: explicit user visualization intent
|
||||
(chart/show/compare/trend/table).
|
||||
- Prompting: tool description + explicit system-prompt guidance snippet for
|
||||
visualization decisioning.
|
||||
- Data entry: primary structured schema; optional text parsing fallback for
|
||||
convenience.
|
||||
|
||||
## Public API / Interface Changes
|
||||
|
||||
### New built-in tool
|
||||
|
||||
- Name: `render_visualization`
|
||||
- Display: `Render Visualization`
|
||||
- Category: built-in core tool (`Kind.Other`)
|
||||
|
||||
### Tool input schema
|
||||
|
||||
- `chartType` (required): `"bar" | "line" | "table"`
|
||||
- `title` (optional): string
|
||||
- `subtitle` (optional): string
|
||||
- `xLabel` (optional): string
|
||||
- `yLabel` (optional): string
|
||||
- `series` (required): array of series
|
||||
- `series[].name` (required): string
|
||||
- `series[].points` (required): array of `{ label: string, value: number }`
|
||||
- `sort` (optional): `"none" | "asc" | "desc"` (applies to single-series
|
||||
bar/table)
|
||||
- `maxPoints` (optional): number (default 30, cap 200)
|
||||
- `inputText` (optional fallback): string (JSON/table/CSV-like text to parse
|
||||
when `series` omitted)
|
||||
- `unit` (optional): string (e.g., `"s"`)
|
||||
|
||||
### Tool output display type
|
||||
|
||||
Add new `ToolResultDisplay` variant:
|
||||
|
||||
- `VisualizationDisplay`:
|
||||
- `type: "visualization"`
|
||||
- `chartType: "bar" | "line" | "table"`
|
||||
- `title?`, `subtitle?`, `xLabel?`, `yLabel?`, `unit?`
|
||||
- `series`
|
||||
- `meta`:
|
||||
`{ truncated: boolean, originalPointCount: number, fallbackMode?: "unicode"|"ascii" }`
|
||||
|
||||
## Architecture and Data Flow
|
||||
|
||||
## 1. Tool implementation (core)
|
||||
|
||||
- New file: `packages/core/src/tools/render-visualization.ts`
|
||||
- Implements validation + normalization pipeline:
|
||||
|
||||
1. Resolve input source:
|
||||
|
||||
- use `series` if provided,
|
||||
- else parse `inputText`.
|
||||
|
||||
2. Validate numeric values (finite only), normalize labels as strings.
|
||||
3. Apply chart-specific constraints:
|
||||
|
||||
- `line`: preserve chronological order unless explicit sort disabled.
|
||||
- `bar/table`: optional sort.
|
||||
|
||||
4. Apply volume controls (`maxPoints`, truncation metadata).
|
||||
5. Return:
|
||||
|
||||
- `llmContent`: concise factual summary + normalized data preview.
|
||||
- `returnDisplay`: typed `VisualizationDisplay`.
|
||||
|
||||
## 2. Built-in registration
|
||||
|
||||
- Register in `packages/core/src/config/config.ts` (`createToolRegistry()`).
|
||||
- Add tool-name constants and built-in lists:
|
||||
- `packages/core/src/tools/tool-names.ts`
|
||||
- `packages/core/src/tools/definitions/coreTools.ts`
|
||||
|
||||
## 3. Prompt guidance (critical for this scenario)
|
||||
|
||||
- Update prompt snippets so model reliably chooses this tool:
|
||||
- In tool-usage guidance section: "When user asks to compare, chart, trend, or
|
||||
show tabular metrics, gather/compute data first, then call
|
||||
`render_visualization` with structured series."
|
||||
- Keep short and deterministic; do not over-prescribe style.
|
||||
- Include 1 canonical example in tool description: "BMW 0-60 comparison."
|
||||
|
||||
## 4. CLI rendering (Ink)
|
||||
|
||||
- Extend `packages/cli/src/ui/components/messages/ToolResultDisplay.tsx` to
|
||||
handle `VisualizationDisplay`.
|
||||
- Add renderer components:
|
||||
- `VisualizationDisplay.tsx` dispatcher
|
||||
- `BarChartDisplay.tsx`
|
||||
- `LineChartDisplay.tsx`
|
||||
- `TableVizDisplay.tsx`
|
||||
|
||||
### Rendering behavior
|
||||
|
||||
- Inline panel, width-aware, no alternate screen required.
|
||||
- Unicode first (`█`, box chars), ASCII fallback when needed.
|
||||
- Label truncation + right-aligned values.
|
||||
- Height caps to preserve conversational viewport.
|
||||
- Multi-series behavior:
|
||||
- V1: `line` supports multi-series.
|
||||
- `bar/table`: single-series required in v1 (validation error if more than one).
|
||||
|
||||
## 5. Natural-language to schema reliability strategy
|
||||
|
||||
- Primary expectation: model transforms researched data into `series`.
|
||||
- Fallback parser for `inputText` supports:
|
||||
- JSON object map
|
||||
- JSON array records
|
||||
- Markdown table
|
||||
- CSV-like 2-column text
|
||||
- Ambiguous prose parsing intentionally rejected with actionable error +
|
||||
accepted examples.
|
||||
- This avoids silent bad charts and improves reasoning-loop consistency.
|
||||
|
||||
## 6. Dynamic volume strategy
|
||||
|
||||
- Defaults:
|
||||
- `maxPoints = 30`
|
||||
- Hard cap `200`
|
||||
- For larger sets:
|
||||
- `bar/table`: keep top N by absolute value (or chronological when user asks
|
||||
"last N years").
|
||||
- `line`: downsample uniformly while preserving first/last points.
|
||||
- Always annotate truncation in `meta` and human-readable footer.
|
||||
|
||||
## Testing Plan
|
||||
|
||||
## Core tests
|
||||
|
||||
- New: `packages/core/src/tools/render-visualization.test.ts`
|
||||
- Cases:
|
||||
- Valid single-series bar (BMW 0-60 style).
|
||||
- Valid line trend (yearly counts).
|
||||
- Valid table rendering payload.
|
||||
- Multi-series line accepted.
|
||||
- Multi-series bar rejected.
|
||||
- `inputText` parse success for JSON/table/CSV.
|
||||
- Ambiguous prose rejected with guidance.
|
||||
- Sort behavior correctness.
|
||||
- Volume truncation/downsampling correctness.
|
||||
- Unit handling and numeric validation.
|
||||
|
||||
## Registration and schema tests
|
||||
|
||||
- Update `packages/core/src/config/config.test.ts`:
|
||||
- tool registers by default
|
||||
- respects `tools.core` allowlist and `tools.exclude`
|
||||
- Update tool definition snapshots for model function declarations.
|
||||
|
||||
## Prompt tests
|
||||
|
||||
- Update prompt snapshot tests to assert presence of visualization guidance line
|
||||
and tool name substitution.
|
||||
|
||||
## UI tests
|
||||
|
||||
- New:
|
||||
- `packages/cli/src/ui/components/messages/VisualizationDisplay.test.tsx`
|
||||
- `BarChartDisplay.test.tsx`
|
||||
- `LineChartDisplay.test.tsx`
|
||||
- `TableVizDisplay.test.tsx`
|
||||
- Validate:
|
||||
- width adaptation (narrow/normal)
|
||||
- unicode/ascii fallback
|
||||
- long labels
|
||||
- truncation indicators
|
||||
- no overflow crashes
|
||||
|
||||
## Integration tests
|
||||
|
||||
- Add scenario tests for:
|
||||
|
||||
1. Research + bar chart call pattern.
|
||||
2. Same-session follow-up question with different metric and chart type.
|
||||
|
||||
- Validate tool call args shape and final rendered output branch selection.
|
||||
|
||||
## Rollout Plan
|
||||
|
||||
- Feature flag: `experimental.visualizationToolV1` default `false`.
|
||||
- Dogfood + internal beta.
|
||||
- Telemetry:
|
||||
- invocation rate
|
||||
- schema validation failure rate
|
||||
- parse fallback usage
|
||||
- render fallback (unicode->ascii) rate
|
||||
- per-chart-type success.
|
||||
- Flip default to `true` after stability threshold.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
- Risk: model under-calls tool.
|
||||
- Mitigation: explicit prompt guidance + strong tool description examples.
|
||||
- Risk: terminal incompatibilities.
|
||||
- Mitigation: deterministic ASCII fallback and conservative layout.
|
||||
- Risk: oversized datasets.
|
||||
- Mitigation: hard caps + truncation/downsampling metadata.
|
||||
- Risk: noisy prose parsing.
|
||||
- Mitigation: strict parser and explicit rejection path.
|
||||
|
||||
## Assumptions and Defaults
|
||||
|
||||
- Modern terminal baseline supports ANSI color and common Unicode; fallback
|
||||
always available.
|
||||
- V1 interactivity (keyboard-driven selections/buttons) is out of scope.
|
||||
- Browser/WebView rendering is out of scope.
|
||||
- V1 excludes negative-value bars.
|
||||
Reference in New Issue
Block a user