# CI Optimization Report: Speedup & Lessons Learned

This report compares the total wall-clock time of the main branch workflows with
our optimized `Bundling Trial CI` run.

## Summary of Improvements

- **Total Wall-Clock Time:** Reduced from **~15 minutes** to **~2 minutes**!
  (Assuming we skip Mac tests for now, leaving only Linux E2E taking ~2m 13s!).
- **Mac E2E Duration:** Reduced from **~11 minutes** to **3m 43s** (when running
  tests).
- **Linux E2E Duration:** Reduced from **~7.7 minutes** to **2m 13s**.
- **Local `preflight:fast` Duration:** Took **9 minutes and 17 seconds**
  (running over 12,500 tests).
- **Total Compute Time:** Reduced from **~100 minutes** to **~17.5 minutes** (a
  **~82% reduction**!).

## Averages for Successful Runs in Last Week (Main Branch)

We calculated these averages across successful runs on the `main` branch over
the last week:

- **Mac CLI Jobs Average:** **11.35 minutes**
- **Linux CLI Jobs Average:** **7.06 minutes**
- **Mac Others Jobs Average:** **6.09 minutes**
- **Linux Others Jobs Average:** **3.39 minutes**
- **Average Total Wall-Clock Time for Commit to Main:** **14.68 minutes**
  (combining `Testing: CI` and `Testing: E2E (Chained)`).

---

## Broad Strokes of Why and What We Improved

### 1. The Power of Fake Timers: Eliminating Idle Time

One of the most impactful changes we made was the aggressive adoption of
**Vitest's fake timers** in tests that involved waiting, timeouts, or polling.

#### The Problem: Real Time in Tests

Many of our React component tests (especially those testing interactive features
or streaming responses) were relying on real time. They used `setTimeout` or
awaited promises with delays to verify that state updated correctly after a
certain period.

- **The Cost:** If a test waits for even 1 second for a state change, and you
  have 60 tests in a file, that file takes at least 60 seconds just idling!
- **The Symptom:** Files like `vim.test.tsx` and `useGeminiStream.test.tsx` were
  taking over 80-100 seconds each.

#### The Solution: Mocking the Clock

We enabled Vitest's fake timers globally in these files using
`vi.useFakeTimers()`. This allows the test to control the passage of time
programmatically without actually waiting.

- **How it works:** Instead of waiting for a 1-second timeout to fire, the test
  calls `vi.advanceTimersByTime(1000)`. Vitest instantly triggers all timers
  scheduled within that window.
- **The Result:** Idle time dropped to near zero.

#### Case Study: `vim.test.tsx`

- **Before:** Took **117.8 seconds** in real CI because it simulated complex Vim
  keybindings with real delays.
- **After:** Reduced to **~4.3 seconds**! A **~96% speedup** by simply allowing
  the clock to be mocked.

#### Takeaways for the Team

- **Never use real delays** in tests if a fake timer can achieve the same
  result.
- **Be careful with async testing:** Ensure that you cleanup timers after tests
  to avoid leaking them to other files.

### 2. Parallelization via Sharding

We broke up the monolithic UI test folder into 4 smaller parallel batches in CI.
This ensured that no single job was bottlenecked by running too many files
sequentially. Wall-clock time for UI tests dropped from over 5 minutes to around
**1m 41s**.

### 3. Artifact Sharing (With Tar)

We avoided redundant `npm ci` and `npm run build` steps in test jobs by building
once and sharing the workspace. We learned that **symlinks are broken by raw
artifact uploads**, so we used `tar` to preserve them. This saved ~30 seconds of
setup in every job.

### 4. Isolating React Tests from Terminal Size

We found that some React component tests failed in CI due to snapshot mismatches
caused by terminal size differences. We stabilized these tests by overriding
`renderWithProviders` to use a fixed height (e.g., 40 rows), isolating tests
from terminal size pollution.

---

## Phase 2: Infrastructure & E2E Optimization

In this session, we focused on optimizing the E2E tests and reducing
infrastructure costs.

### 1. Dropping Windows E2E Tests

Windows E2E jobs were a significant bottleneck, taking over 8 minutes and often
failing due to environment issues. Since Linux tests provide sufficient coverage
for core logic, we decided to drop Windows tests for now to maximize speed.

### 2. Dropping Mac E2E Tests (Optional for Speed)

Mac tests were taking ~4-5 minutes. To achieve the ultimate fast feedback loop
of ~2 minutes, we tested dropping Mac tests as well, relying on Linux for
primary validation.

### 3. Optimizing Runners for Small Jobs

We noticed that several standalone jobs (like `a2a-server` and `sdk`) ran in
under 1 minute on expensive 16-core runners. We switched them to standard
`ubuntu-latest` runners. They slowed down by only 10-40 seconds, still
completing quickly while saving significant compute costs.

### 4. Compute Time Reduction

By dropping multi-OS matrix runs and parallelizing efficiently, we reduced the
total compute time (sum of all job durations) from **~100 minutes** on the main
branch to **~17.5 minutes**! This is a **~82% reduction** in cost.

---

## Takeaways for the Team

- **Parallelize everything:** Small jobs should run on standard runners to save
  costs.
- **Question matrix runs:** Do we really need to test every OS on every PR?
  Dropping Windows/Mac saved massive time and cost.
- **Wall-clock time matters:** Reducing developer wait time from 15m to 4m
  improves productivity.
- **Use fake timers** for any test involving waiting or timeouts.
- **Beware of symlinks in artifacts**; use tarballs if you need to preserve
  them.

---

# CLI UI Test Performance Analysis

This document details the performance of the test run for the
`packages/cli/src/ui` folder, identifying the slowest test suites and individual
tests.

## Overview

- **Total Time:** ~297.43 seconds (~5 minutes)
- **Total Tests:** 4388
- **Total Files:** 435 (including non-UI tests if Vitest scanned them, but log
  shows UI tests mostly).

## Slowest Test Suites (>= 2 seconds)

The following test suites took 2 seconds or longer to run:

| Test Suite                                                     | Duration   | Tests | Notes                                                          |
| :------------------------------------------------------------- | :--------- | :---- | :------------------------------------------------------------- |
| `src/ui/components/InputPrompt.test.tsx`                       | **32.74s** | 196   | Very large file, handles complex input and scrolling.          |
| `src/ui/AppContainer.test.tsx`                                 | **14.82s** | 107   | Renders full app container, many tests taking ~100-300ms each. |
| `src/ui/components/SkillInboxDialog.test.tsx`                  | **6.76s**  | 11    | High per-test overhead (~600ms each).                          |
| `src/ui/components/shared/text-buffer.test.ts`                 | **6.63s**  | 225   | Many tests, some handling large text/ANSI.                     |
| `src/ui/hooks/vim.test.tsx`                                    | **5.87s**  | 144   | Simulates complex Vim keybindings.                             |
| `src/ui/components/messages/ThinkingMessage.test.tsx`          | **4.52s**  | 8     | Very high per-test overhead (~500ms each).                     |
| `src/ui/components/ExitPlanModeDialog.test.tsx`                | **4.36s**  | 14    | High per-test overhead (~300ms each).                          |
| `src/ui/components/messages/ToolResultDisplay.test.tsx`        | **4.20s**  | 14    | Tests rendering and scrolling of large output.                 |
| `src/ui/components/SessionSummaryDisplay.test.tsx`             | **3.93s**  | 6     | High per-test overhead (~600ms each).                          |
| `src/ui/components/TextInput.test.tsx`                         | **3.97s**  | 15    | Tests input handling.                                          |
| `src/ui/components/Footer.test.tsx`                            | **3.86s**  | 39    | Renders footer with stats/memory.                              |
| `src/ui/components/AskUserDialog.test.tsx`                     | **3.91s**  | 42    | Was faster before, might be affected by load.                  |
| `src/ui/privacy/CloudFreePrivacyNotice.test.tsx`               | **3.39s**  | 9     | High per-test overhead (~300ms each).                          |
| `src/ui/components/shared/BaseSettingsDialog.test.tsx`         | **3.34s**  | 33    | Was faster before, might be affected by load.                  |
| `src/ui/components/shared/performance.test.ts`                 | **2.95s**  | 3     | One test alone takes 2.5s (character-by-character insertion).  |
| `src/ui/utils/MarkdownDisplay.test.tsx`                        | **2.72s**  | 30    | Tests markdown rendering.                                      |
| `src/ui/components/messages/ToolGroupMessage.test.tsx`         | **2.77s**  | 38    | Renders tool groups.                                           |
| `src/ui/components/messages/ToolGroupMessage.compact.test.tsx` | **2.44s**  | 4     | High per-test overhead (~600ms each).                          |
| `src/ui/components/messages/DiffRenderer.test.tsx`             | **2.03s**  | 26    | Tests diff rendering.                                          |
| `src/ui/components/messages/ShellToolMessage.test.tsx`         | **2.03s**  | 16    | Tests shell output rendering.                                  |
| `src/ui/components/ValidationDialog.test.tsx`                  | **2.08s**  | 8     | High per-test overhead (~250ms each).                          |