MediaMetz/gemini-cli
1
0
Fork 0
mirror of https://github.com/google-gemini/gemini-cli.git synced 2026-05-22 17:53:04 -07:00
Code Issues Packages Projects Releases Wiki Activity

feat(skills): pivot experimentation skill to guide new experiment creation

Browse Source
This commit is contained in:
mkorwel
2026-02-19 16:19:32 -06:00
committed by Matt Korwel
parent 50f71a8df5
commit 2e1b556764
2 changed files with 23 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gemini/skills/experimentation/SKILL.md
+23 -33
View File
@@ -1,46 +1,36 @@
---
name: experimentation
description: Manage, view, and override Gemini CLI remote experiments and feature flags locally.
description: Guide developers through the process of adding new remote experiments (feature flags) to the Gemini CLI codebase.
---
# Experimentation Skill
This skill allows you to safely manage, view, and locally override remote Gemini CLI experiments (feature flags).
This skill assists developers in adding net-new remote experiments (feature flags) to the Gemini CLI codebase. It acts as an interactive guide to ensure all necessary scaffolding, telemetry, command-line overrides, settings, and code placement are considered.
## Core Concepts
## Workflow: Adding a New Experiment
Remote experimentation is enabled by default. `packages/core/src/code_assist/experiments/flagNames.ts` contains the active `ExperimentFlags` and `ExperimentMetadata` which describe each flag's purpose, type, and default value.
When a user asks to add a new experiment, follow these steps sequentially:
Currently, Gemini CLI supports local overrides using the `GEMINI_EXP` environment variable pointing to a JSON file.
### 1. Scaffolding the Config Entry
- Guide the user to add the new experiment ID to `ExperimentFlags` in `packages/core/src/code_assist/experiments/flagNames.ts`.
- Ensure a corresponding entry is added to `ExperimentMetadata` in the same file. This must include a clear `description`, the `type` (`boolean`, `number`, `string`), and a sensible `defaultValue`.
## Workflow: Viewing Experiments
### 2. Command Line and Settings Overrides (Crucial Pattern)
Every remote flag must be overridable via local settings and command-line arguments.
- **Settings:** Guide the user to add a corresponding property to the settings schema (e.g., in `schemas/settings.schema.json` and the corresponding TypeScript interfaces).
- **Command Line:** Guide the user to add a corresponding CLI argument in the appropriate command definitions.
- **The Config Object:** The source of truth for the rest of the application should be the unified `Config` object. Guide the user to wrap the remote experiment value, the local setting, and the command-line argument into a single property on the `Config` object. The typical priority order is: Command Line > Local Setting > Remote Experiment > Default Value.
When a user asks what experiments are active or available:
1. Search `packages/core/src/code_assist/experiments/flagNames.ts` for `ExperimentMetadata`.
2. Extract the descriptions, types, and defaults to answer the user's questions.
3. Check if there is an active local override file at `.gemini/experiments.json`.
### 3. Code Placement and Usage
- Discuss with the user *where* in the codebase this experiment should take effect.
- Guide them on how to correctly fetch and evaluate the experiment value using the unified `Config` object.
- Help them write the necessary conditional logic around the experimental feature.
## Workflow: Local Overrides & Opt-Out
### 4. Telemetry and Metrics (Crucial)
- Prompt the user to think deeply about what metrics are necessary to evaluate the success or failure of the experiment.
- Ask questions like: "How will we know if this feature is working as intended?" or "What telemetry events should be fired when this new code path is executed?"
- Help them add the necessary telemetry calls to the codebase to capture these insights.
When a user wants to override a flag locally (e.g., to turn off a preview feature) or opt-out:
1. Use the `scripts/override_experiment.cjs` script bundled with this skill to safely update or create `.gemini/experiments.json`.
2. When the user asks to "opt out of experiments", use the script to set `experimentIds` to an empty array and clear flags, ensuring a sensible baseline.
3. **Important:** After updating the JSON file, instruct the user to run the CLI with `GEMINI_EXP` set, e.g.:
`GEMINI_EXP=.gemini/experiments.json gemini <command>`
## Using the Scripts
You have access to `scripts/override_experiment.cjs` to manage the local overrides safely without disrupting the file structure required by the CLI backend.
Example usage:
```bash
# Enable an experiment locally
node .gemini/skills/experimentation/scripts/override_experiment.cjs set 45740196 true
# Remove an override
node .gemini/skills/experimentation/scripts/override_experiment.cjs unset 45740196
# Opt out of all experiments (clear everything)
node .gemini/skills/experimentation/scripts/override_experiment.cjs clear
```
### 5. Branching and PR Preparation
- If not already on a dedicated branch, help the user create a new git branch for this experiment (e.g., `git checkout -b exp/feature-name`).
- Remind them to run local tests and linting (`npm run lint:all`, `npm test` or `npm run preflight`) before preparing a Pull Request.
.gemini/skills/experimentation/scripts/override_experiment.cjs
-72
View File
@@ -1,72 +0,0 @@
const fs = require('fs');
const path = require('path');
const filePath = path.join(process.cwd(), '.gemini', 'experiments.json');
function readExperiments() {
if (fs.existsSync(filePath)) {
try {
const data = JSON.parse(fs.readFileSync(filePath, 'utf8'));
return {
flags: data.flags || [],
experimentIds: data.experimentIds || []
};
} catch (e) {
console.error('Failed to parse existing experiments.json, starting fresh.');
}
}
return { flags: [], experimentIds: [] };
}
function writeExperiments(data) {
const dir = path.dirname(filePath);
if (!fs.existsSync(dir)) {
fs.mkdirSync(dir, { recursive: true });
}
fs.writeFileSync(filePath, JSON.stringify(data, null, 2), 'utf8');
console.log(`Updated local experiments at ${filePath}`);
}
const args = process.argv.slice(2);
const command = args[0];
if (!command) {
console.error('Usage: override_experiment.cjs <set|unset|clear> [flagId] [value]');
process.exit(1);
}
const data = readExperiments();
if (command === 'clear') {
data.flags = [];
data.experimentIds = [];
writeExperiments(data);
} else if (command === 'unset') {
const flagId = parseInt(args[1], 10);
data.flags = data.flags.filter(f => f.flagId !== flagId);
writeExperiments(data);
} else if (command === 'set') {
const flagId = parseInt(args[1], 10);
const rawValue = args[2];
if (isNaN(flagId) || rawValue === undefined) {
console.error('Invalid arguments for set: requires numeric flagId and value');
process.exit(1);
}
// Remove existing flag
data.flags = data.flags.filter(f => f.flagId !== flagId);
// Parse value
const flag = { flagId };
if (rawValue === 'true') flag.boolValue = true;
else if (rawValue === 'false') flag.boolValue = false;
else if (!isNaN(Number(rawValue))) flag.numberValue = Number(rawValue);
else flag.stringValue = rawValue;
data.flags.push(flag);
writeExperiments(data);
} else {
console.error('Unknown command');
process.exit(1);
}