Compare commits

...

2 Commits

Author SHA1 Message Date
Hadi Minooei 1e52da87ce chore: drop packages/dd.md 2026-03-26 23:13:42 -07:00
Hadi Minooei 554d52369d fix(cli): improve simulator action history and fix dialog max items 2026-03-26 23:11:02 -07:00
4 changed files with 34 additions and 75 deletions
+31 -2
View File
@@ -22,6 +22,7 @@ export class UserSimulator {
private interactionsFile: string | null = null;
private knowledgeBase = '';
private actionHistory: string[] = [];
constructor(
private readonly config: Config,
@@ -122,6 +123,10 @@ export class UserSimulator {
? `\nUser Knowledge Base:\nUse this information to answer questions if applicable. If the answer is not here, respond as you normally would.\n${this.knowledgeBase}\n`
: '';
const historyInstruction = this.actionHistory.length > 0
? `\nRecent Simulator Actions (last 10):\n${this.actionHistory.slice(-10).map((a, i) => `${i + 1}. ${JSON.stringify(a)}`).join('\n')}\n`
: '';
const prompt = `You are evaluating a CLI agent by simulating a user sitting at the terminal.
Look carefully at the screen and determine the CLI's current state:
@@ -145,7 +150,7 @@ CRITICAL RULES:
- RULE 2: If there is an "Action Required" or confirmation prompt on the screen, YOU MUST HANDLE IT (State 2). This takes precedence over everything else.
- RULE 3: Output ONLY the raw characters to send, <WAIT>, or <DONE>.
- RULE 4: Do NOT output markdown, explanations of your thought process, or quotes.
${goalInstruction}${knowledgeInstruction}
${goalInstruction}${knowledgeInstruction}${historyInstruction}
Here is the current terminal screen output:
@@ -217,6 +222,13 @@ ${strippedScreen}
debugLogger.log(
'[SIMULATOR] Skipping action (model decided to <WAIT>)',
);
this.actionHistory.push('<WAIT>');
if (this.interactionsFile) {
fs.appendFileSync(
this.interactionsFile,
`[LOG] [SIMULATOR] Action History updated with: "<WAIT>"\n\n`,
);
}
this.lastScreenContent = normalizedScreen;
return;
}
@@ -224,14 +236,31 @@ ${strippedScreen}
if (responseText) {
const keys = responseText.replace(/\\n/g, '\r').replace(/\\r/g, '\r');
debugLogger.log(
`[SIMULATOR] Sending to stdin: ${JSON.stringify(keys)}`,
);
this.actionHistory.push(keys);
if (this.interactionsFile) {
fs.appendFileSync(
this.interactionsFile,
`[LOG] [SIMULATOR] Action History updated with: ${JSON.stringify(keys)}\n\n`,
);
}
this.stdinBuffer.write(keys);
this.lastScreenContent = normalizedScreen;
} else {
debugLogger.log('[SIMULATOR] Skipping (empty response)');
this.actionHistory.push('<EMPTY>');
if (this.interactionsFile) {
fs.appendFileSync(
this.interactionsFile,
`[LOG] [SIMULATOR] Action History updated with: "<EMPTY>"\n\n`,
);
}
this.lastScreenContent = normalizedScreen;
}
} catch (e: unknown) {
@@ -853,7 +853,7 @@ const ChoiceQuestionView: React.FC<ChoiceQuestionViewProps> = ({
listHeight && !isAlternateBuffer
? question.unconstrainedHeight
? Math.max(1, listHeight - selectionItems.length * 2)
: Math.min(15, Math.max(1, listHeight - DIALOG_PADDING))
: Math.min(15, Math.max(1, listHeight - Math.min(selectionItems.length, 5) * 2))
: undefined;
const maxItemsToShow =
@@ -30,6 +30,8 @@ exports[`AskUserDialog > Scroll Arrows (useAlternateBuffer: false) > shows scrol
Description 1
2. Option 2
Description 2
3. Option 3
Description 3
Enter to select · ↑/↓ to navigate · Esc to cancel
-72
View File
@@ -1,72 +0,0 @@
# User Simulation Feature - Design Document
*Branch: `user-simulation`*
## 1. Overview
The User Simulation feature enables an autonomous LLM-driven agent to act as a realistic user in interactive terminal sessions. It aims to support integration testing, evaluation, and background evaluation of system capabilities by effectively observing the terminal interface and supplying contextual standard input (stdin).
This is accomplished by adding a `--simulate-user` option to the CLI, intercepting terminal updates via a simulated stdout reader, computing appropriate responses via an LLM, and piping these directly into a `PassThrough` stream masquerading as `process.stdin`.
## 2. Core Architecture
### 2.1 Configurations and Triggers
**Affected Files:**
- `packages/cli/src/config/config.ts`
- `packages/core/src/config/config.ts`
- `packages/cli/src/interactiveCli.tsx`
**Implementation:**
The feature is controlled via the explicit CLI argument `--simulate-user`. When supplied, the argument flows through the `Config` interface (`config.getSimulateUser()`), modifying how the interactive CLI bootstraps.
- Instead of attaching the raw `process.stdin` directly to the `ink` render environment, the UI engine binds to a Node.js `PassThrough` stream if simulation is enabled.
- The Interactive CLI session begins by initiating the `UserSimulator`, providing it access to the `Config`, the `simulatedStdin` buffer, and an observable snapshot of the `lastFrame` (the current state of the UI render).
### 2.2 The `UserSimulator` Service
**Affected Files:**
- `packages/cli/src/services/UserSimulator.ts`
**Implementation:**
The `UserSimulator` class governs the core observation and input loops. It handles interactions asynchronously outside of the primary command execution flow.
#### Loop Execution
The simulator runs via a repeating timer (3-second intervals `setInterval`), checking `tick()`. It ensures only one evaluation happens sequentially using a mutex lock (`this.isProcessing`).
#### Screen Processing and Environment Context
The engine receives the rendered terminal snapshot (`lastFrame`) and applies sanitizations:
- Removes ANSI terminal colors and cursor properties using regex filtering.
- Strips variable UI animation symbols (Spinners like ⠋, ⠙) and time indicators (e.g., `[ 7s ]`).
These visual stability checks allow the simulator to detect meaningful frame changes instead of being repeatedly triggered by minor clock updates.
#### Simulation Brain & LLM Integration
The simulator utilizes `PREVIEW_GEMINI_FLASH_MODEL`, leveraging the newly configured `thinking: true` capacity (`packages/core/src/config/defaultModelConfigs.ts`) for better reasoning, operating under the distinct telemetry role `LlmRole.UTILITY_SIMULATOR`.
The LLM is provided an extensive context schema encapsulating:
- **Terminal output:** Current stripped terminal representation.
- **Original task objective:** The intention of the CLI session.
- **Action Memory:** A tracked sequence (`actionHistory`) of previously executed actions to avoid getting stuck in inquiry loops.
#### State Machine Interpretation
The model translates the screen context into specific behavioral states:
- **STATE 1 (WAIT):** Triggered if any indication of internal "thinking" or processing exists (active spinners/timers). Returns `<WAIT>`.
- **STATE 2 (ACTION/AUTHORIZATION):** Prompt requests an approval (e.g., "Allow execution"). Supplies strict raw choice input.
- **STATE 3 (DONE):** Fully idle string detection paired with task completion fulfillment. Gracefully types `Thank you\r` and eventually exits indicating `<DONE>`.
- **STATE 4 (INPUT):** Idle state where continuation text instruction is needed, supplying direct commands injected to stdin.
#### Safeguards and Debugging
- **Anti-Stall Mechanism:** The simulator tracks repetitive actions. If the LLM repeats the exact same non-empty interaction `MAX_REPEATS` (3) times, it terminates the process abruptly to prevent infinite loops.
- **Interaction Transcripts:** Binds an audit log, dumping each parsed `[SIMULATOR] Screen Content Seen` block, the constructed `Prompt Used`, and `Raw model response` to discrete contextual `interactions_<timestamp>.txt` files, making it traceable.
### 2.3 Required Ecosystem Changes
- `packages/core/src/telemetry/llmRole.ts`: A new internal trace role `UTILITY_SIMULATOR` has been appended to separate evaluation events from main-thread generations.
- `packages/core/src/config/defaultModelConfigs.ts`: Added dynamic thought logic parameter (`thinking: true`) for `gemini-3-flash-preview` to enhance the simulation depth reasoning.
- Unit Testing Mocks: Updated dummy interface `mockConfig.ts` to ensure compatibility by safely mocking `getSimulateUser`.
## 3. Workflow Summary
1. The user launches `gemini --simulate-user`.
2. Interactive UI bindings swap `process.stdin` for a readable `PassThrough` stream.
3. Every 3 seconds, `UserSimulator.tick()` accesses the newest render frame.
4. The frame is stripped of unstable metrics/spinners. If the clean frame is unchanged since the last tick, it waits.
5. If changed, the simulator invokes Gemini Flash Preview (Thinking).
6. Gemini observes the state—emitting `<WAIT>` if busy, action keys for selections, `<DONE>` if completed, or commands to be resolved.
7. Return text is piped cleanly back to `simulatedStdin`—advancing the ink render state.
*This design effectively encapsulates the user interface flow without imposing side-effects on standard operations.*