diff --git a/.gemini/skills/experimentation/SKILL.md b/.gemini/skills/experimentation/SKILL.md index 29119bdc83..84f8483efb 100644 --- a/.gemini/skills/experimentation/SKILL.md +++ b/.gemini/skills/experimentation/SKILL.md @@ -5,69 +5,67 @@ description: Guide developers through the process of adding new remote experimen # Experimentation Skill -This skill assists developers in adding net-new remote experiments (feature flags) to the Gemini CLI codebase. It acts as an interactive guide to ensure all necessary scaffolding, telemetry, command-line overrides, settings, and code placement are considered. +This skill assists developers in adding net-new remote experiments (feature flags) to the Gemini CLI codebase. You MUST follow these steps sequentially to ensure consistency, type safety, and proper validation. ## Key Files - `packages/core/src/code_assist/experiments/flagNames.ts`: Definitions and metadata for all experiment flags. - `packages/core/src/config/config.ts`: Unified configuration object where strongly typed wrappers for experiments are added. - `packages/cli/src/config/settingsSchema.ts`: Schema definitions for `settings.json`. +- `packages/core/src/config/config.experiment.test.ts`: Regression tests for experiment resolution. ## Core Pattern: Unified Configuration Gemini CLI uses a unified `Config` object as the source of truth. Every experiment should be accessed via the `config.getExperimentValue(flagId)` method, which internally handles the priority of overrides: **Command Line Argument (--experiment-*) > Local Setting (experimental.*) > Remote Experiment > Default Value.** +--- + ## Workflow: Adding a New Experiment -When a user asks to add a new experiment, follow these steps sequentially: - -### 0. Research and Discovery -- **Locate Existing Flags:** Read `packages/core/src/code_assist/experiments/flagNames.ts` to see current IDs and naming conventions. -- **Check for Duplicates:** Ensure the proposed experiment doesn't already exist or overlap with another flag. -- **Analyze Usage:** Search for `getExperimentValue` in `packages/core/src/config/config.ts` to see how similar experiments are wrapped and used. - -### 1. Define Behavior and Intent -- **CRITICAL:** Before writing any code, you MUST ask the user: "What should the behavior be when the flag is enabled vs. disabled?" -- Discuss the architectural impact: Will this change a routing strategy? Enable a new tool? Modify a prompt? -- Document the intended logic to ensure the implementation aligns with the experiment's goals. - -### 2. Scaffolding the Config Entry +### 1. List Current Experiments - **File:** `packages/core/src/code_assist/experiments/flagNames.ts` -- Add the new experiment ID to `ExperimentFlags`. +- Read this file and list all existing experiments to the user. This ensures you pick a unique ID and avoid duplicate names. + +### 2. Ask for Details (Behavior & Intent) +- **CRITICAL:** You MUST ask the user: + 1. "What is the name of the new flag?" (e.g., `enable-new-thing`) + 2. "What is the data type?" (boolean, number, or string) + 3. "What should the behavior be when the flag is enabled vs. disabled?" + 4. "What are the architectural impacts (routing, tools, prompt construction, etc.)?" +- Do not proceed until the behavior is clearly defined. + +### 3. Add Experiment ID and Metadata +- **File:** `packages/core/src/code_assist/experiments/flagNames.ts` +- Add a new constant to the `ExperimentFlags` object with a unique numeric ID. - Add a corresponding entry to `ExperimentMetadata`, including `description`, `type`, and `defaultValue`. -- **Note:** The key in `ExperimentFlags` (converted to kebab-case) will be the name used for CLI flags and Settings. For example, `MY_NEW_FEATURE` becomes `my-new-feature`. -### 3. Update Settings Schema -- **File:** `packages/cli/src/config/settingsSchema.ts` -- To ensure IDE autocompletion and validation work for the new experiment in `settings.json`, you MUST add it to the `experimental.properties` section of the schema. -- Match the kebab-case name used in step 2. -- **Action:** After updating the file, run `npm run schema:settings` to regenerate `schemas/settings.schema.json`. - -### 4. Usage in Code -- **Generic Method:** `config.getExperimentValue(ExperimentFlags.YOUR_FLAG_ID)` -- **Preferred Pattern (Strongly Typed Wrappers):** To maintain a clean and discoverable interface, you should add a strongly typed wrapper method in `packages/core/src/config/config.ts`. This allows other developers to easily find and use your experiment with proper type safety. - - Example in `Config` class: +### 4. Add Config Entry and Schema +- **Step A (Wrapper):** In `packages/core/src/config/config.ts`, add a strongly typed wrapper method: ```typescript - isNewFeatureEnabled(): boolean { - return this.getExperimentValue(ExperimentFlags.MY_NEW_FEATURE) ?? false; + isNewThingEnabled(): boolean { + return this.getExperimentValue(ExperimentFlags.ENABLE_NEW_THING) ?? false; } ``` +- **Step B (Schema):** In `packages/cli/src/config/settingsSchema.ts`, add the kebab-case name to the `experimental.properties` section. +- **Step C (Regenerate):** Run `npm run schema:settings` to update `schemas/settings.schema.json`. -- **Testing:** Add or update tests in `packages/core/src/config/config.experiment.test.ts` to verify the flag resolves correctly across all priority levels (CLI, Settings, Remote, Default). -- **CLI Override:** Users can override via `--experiment your-flag-name=value`. -- **Settings Override:** Users can override in their `settings.json`: - ```json - "experimental": { - "your-flag-name": value - } +### 5. Implement Code Change with Debug Logging +- **Usage:** Implement the logic defined in Step 2. +- **Logging:** At the point of usage, add a debug log to confirm the flag's state: + ```typescript + debugLogger.debug('New thing enabled:', config.isNewThingEnabled()); ``` -### 5. Telemetry and Metrics (Crucial) -- Prompt the user to think deeply about what metrics are necessary to evaluate the success or failure of the experiment. -- Ask questions like: "How will we know if this feature is working as intended?" or "What telemetry events should be fired when this new code path is executed?" -- Help them add the necessary telemetry calls to the codebase to capture these insights. +### 6. Update Tests +- **File:** `packages/core/src/config/config.experiment.test.ts` +- Add a test case to verify that the flag resolves correctly (e.g., `expect(config.isNewThingEnabled()).toBe(true)` when passed via `experimentalCliArgs`). -### 6. Branching and PR Preparation -- If not already on a dedicated branch, help the user create a new git branch for this experiment (e.g., `git checkout -b exp/feature-name`). -- Remind them to run local tests and linting (`npm run lint:all`, `npm test` or `npm run preflight`) before preparing a Pull Request. +### 7. Final Validation +- **Action:** Run the CLI with debugging enabled and the new flag set via the command line: + ```bash + npm run start -- --debug --experiment your-flag-name=value --prompt "test" + ``` +- **Verification:** Check the console output for: + 1. The "Experimental CLI args" normalization log. + 2. The custom debug log you added in Step 5. + 3. The intended behavior change in the CLI.