Replaced eslint-disable-next-line comments with explicit type guarding (checking instanceof Error and 'code') and used process.stderr.write instead of console.warn to comply with project linting rules natively.
Moved the isSubpath and resolveToRealPath imports to the top of the file to satisfy the static analysis bot, which failed to detect the existing import statement further down in the file.
- Reverted the 'isIdle' guard in AppContainer.tsx to ensure slash commands entered while the agent is busy are correctly processed or queued, preventing them from falling through as regular chat text.
- Enhanced the physical path validation in config.ts to gracefully handle 'mkdirSync' failures (e.g. EACCES). The CLI will now log a warning and return the lexically-validated path instead of throwing a misleading 'Security violation' via 'resolveToRealPath'.
If the user's home directory or global `.gemini` config directory has
restrictive permissions that prevent `realpathSync` from succeeding, the
CLI should not crash. Instead, it now gracefully degrades by omitting
the global config directory from the valid security boundaries for plan
directories, allowing the CLI to continue operating securely within the
project root.
Resolves security review findings:
- Reordered resolveToRealPath before mkdirSync to fully eliminate TOCTOU risks with symlink injection.
- Fail closed by re-throwing 'Security violation' errors instead of swallowing them.
- Replaced lint-disabler with process.stderr.write for legitimate fallback warnings.
- Used direct context string as LRUCache key to avoid collision with an extension potentially named 'default'.
This change addresses a critical security review finding regarding a Time-of-Check to Time-of-Use (TOCTOU) vulnerability.
Previously, plan directory paths were validated using `isSubpath` before creation. However, an attacker could potentially replace a path component with a symlink pointing outside the project root exactly between validation and creation.
By resolving the physical path *after* `fs.mkdirSync` using `resolveToRealPath` and then verifying it with `isSubpath`, we ensure that the actual directory created on disk resides safely within the workspace. Any violation results in a warning, and the malicious path is prevented from being added to the agent's `workspaceContext`.
This commit addresses the final performance and usability review comments:
- **Performance:** Introduced `LRUCache` for `plansDirCache` and `initializedPlanDirs` to prevent redundant, synchronous filesystem calls to `Storage.getPlansDir` on every turn.
- **Performance:** Cached the resolved `realProjectRoot` in the `Storage` constructor, eliminating expensive synchronous symlink resolution calls during active command routing.
- **Usability:** Replaced hard `throw` with `console.warn` when `fs.mkdirSync` fails (e.g., `EACCES`, `EEXIST`), allowing the CLI to gracefully degrade and continue functioning rather than crashing the entire process.
- **Validation:** Updated `config.test.ts` to verify the exact warning messages emitted during filesystem failures.
This commit addresses several critical findings from the review bot:
- **Security:** Implemented defense-in-depth symlink resolution. Removed insecure string-based fallbacks in `Storage.getPlansDir` and added a mandatory `isSubpath` validation AFTER directory creation in `Config.getPlansDir` to prevent TOCTOU traversal attacks.
- **Architecture:** Fixed a race condition where active extension context was mutated synchronously in `AppContainer`, potentially corrupting concurrent background tasks. Mutation now occurs within the command execution pipeline.
- **Robustness:** Switched to canonical path checking for `plan` command detection to support aliases and subcommands.
- **Regressions:** Added a `planEnabled` guard to prevent unwanted directory creation when the planning feature is disabled.
- **Validation:** Added exhaustive unit tests covering sequential context switching, shared directory deduplication, and symlink security edge cases.
This removes the insecure ENOENT fallback in `Storage.getPlansDir` that could be exploited to bypass the `isSubpath` check via symlinks. The fallback was unnecessary because the underlying `resolveToRealPath` function (via `robustRealpath`) was recently updated to gracefully handle and resolve symlinks for non-existent target paths.
This addresses a potential TOCTOU vulnerability and edge case identified during review. The redundant `fs.existsSync` check in `getPlansDir` has been removed, allowing `fs.mkdirSync(..., { recursive: true })` to safely handle directory idempotency.
By relying directly on `mkdirSync`, we ensure that if a non-directory file already exists at the target path, the system will correctly throw an `EEXIST` error rather than silently treating the file as a directory and crashing later during workspace registration.
This fixes a bug where the active extension context would remain sticky when a user switched from an extension command to a standard non-plan command, or to an extension without a plan directory.
The context is now correctly reset to undefined when an extension command without a plan directory is executed, preventing subsequent plan mode invocations from incorrectly targeting the previous extension's folder.
This commit addresses two bugs identified during review:
1. Cleared the sticky `activeExtensionContext` when the standard `/plan` command is executed, ensuring subsequent prompts correctly target the default global plan directory.
2. Fixed a path resolution regression in `Storage.getPlansDir()` by constructing the fallback ENOENT path directly against the real project root. This prevents `isSubpath` validation failures and potential traversal vulnerabilities when the project root is a symlink.
Extracts the extension context from slash commands based on their registered metadata and sets it as the active context in the Config before execution. This enables the backend to dynamically route plan directories based on the extension that owns the invoked command.
Updates prompts and tool implementations (edit, write-file, enter/exit plan mode) to route through Config.getPlansDir() instead of Storage.getPlansDir(). This ensures the plan directory is lazily created exactly when these features attempt to use it, preventing ENOENT failures.
Introduces active extension context tracking in config to support dynamic switching of plan directories. Resolves circular dependency in storage by deferring plan directory creation until on-demand use, preventing ENOENT errors on non-existent paths.
description="Check status of nightly evals, fix failures for key models, and re-run."
prompt="""
You are an expert at fixing behavioral evaluations.
1. **Investigate**:
- Use 'gh' cli to fetch the results from the latest run from the main branch: https://github.com/google-gemini/gemini-cli/actions/workflows/evals-nightly.yml.
- DO NOT push any changes or start any runs. The rest of your evaluation will be local.
- Evals are in evals/ directory and are documented by evals/README.md.
- The test case trajectory logs will be logged to evals/logs.
- You should also enable and review the verbose agent logs by setting the GEMINI_DEBUG_LOG_FILE environment variable.
- Identify the relevant test. Confine your investigation and validation to just this test.
- Proactively add logging that will aid in gathering information or validating your hypotheses.
2. **Fix**:
- If a relevant test is failing, locate the test file and the corresponding prompt/code.
- It's often helpful to make an extreme, brute force change to see if you are changing the right place to make an improvement and then scope it back iteratively.
- Your **final** change should be **minimal and targeted**.
- Keep in mind the following:
- The prompt has multiple configurations and pieces. Take care that your changes
end up in the final prompt for the selected model and configuration.
- The prompt chosen for the eval is intentional. It's often vague or indirect
to see how the agent performs with ambiguous instructions. Changing it should
be a last resort.
- When changing the test prompt, carefully consider whether the prompt still tests
the same scenario. We don't want to lose test fidelity by making the prompts too
direct (i.e.: easy).
- Your primary mechanism for improving the agent's behavior is to make changes to
tool instructions, prompt.ts, and/or modules that contribute to the prompt.
- If prompt and description changes are unsuccessful, use logs and debugging to
confirm that everything is working as expected.
- If unable to fix the test, you can make recommendations for architecture changes
that might help stablize the test. Be sure to THINK DEEPLY if offering architecture guidance.
Some facts that might help with this are:
- Agents may be composed of one or more agent loops.
- AgentLoop == 'context + toolset + prompt'. Subagents are one type of agent loop.
- Agent loops perform better when:
- They have direct, unambiguous, and non-contradictory prompts.
- They have fewer irrelevant tools.
- They have fewer goals or steps to perform.
- They have less low value or irrelevant context.
- You may suggest compositions of existing primitives, like subagents, or
propose a new one.
- These recommendations should be high confidence and should be grounded
in observed deficient behaviors rather than just parroting the facts above.
Investigate as needed to ground your recommendations.
3. **Verify**:
- Run just that one test if needed to validate that it is fixed. Be sure to run vitest in non-interactive mode.
- Running the tests can take a long time, so consider whether you can diagnose via other means or log diagnostics before committing the time. You must minimize the number of test runs needed to diagnose the failure.
- After the test completes, check whether it seems to have improved.
- You will need to run the test 3 times for Gemini 3.0, Gemini 3 flash, and Gemini 2.5 pro to ensure that it is truly stable. Run these runs in parallel, using scripts if needed.
- Some flakiness is expected; if it looks like a transient issue or the test is inherently unstable but passes 2/3 times, you might decide it cannot be improved.
4. **Report**:
- Provide a summary of the test success rate for each of the tested models.
- Success rate is calculated based on 3 runs per model (e.g., 3/3 = 100%).
- If you couldn't fix it due to persistent flakiness, explain why.
In addition to the code context, you MUST strictly adhere to the following frontend-specific development guidelines while adding code to packages/cli.
## Testing Standards
* **Async Testing**: ALWAYS use `waitFor` from `packages/cli/src/test-utils/async.ts` instead of `vi.waitFor` to prevent flakiness and `act` warnings. NEVER use fixed waits (e.g., `await delay(100)`).
* **State Changes**: Wrap all blocks that change component state in `act`.
* **Snapshots**: Use `toMatchSnapshot` to verify rendering.
* **Rendering**: Use `render` or `renderWithProviders` from `packages/cli/src/test-utils/render.tsx` instead of `ink-testing-library` directly.
* **Mocking**:
* Reuse existing mocks/fakes where possible.
* **Parameterized Tests**: Use parameterized tests with explicit types to reduce duplication and ensure type safety.
* Avoid mocking the file system, os, or child_process if at all possible; if you have to mock the Mock critical dependencies (`fs`, `os`, `child_process`) do so ONLY at the top of the file.
## React & Ink Architecture
* **Keyboard Handling**: You MUST use `useKeyPress.ts` from Gemini CLI for keyboard handling, NOT the standard ink library. This is critical for supporting slow terminals and multiple events per frame.
* Handle multiple events gracefully (often requires a reducer pattern, see `text-buffer.ts`).
* **State Management**:
* NEVER trigger side effects from within the body of a `setState` callback. Use a reducer or `useRef` if necessary.
* Initialize state explicitly (e.g., use `undefined` rather than `true` if unknown).
* **Performance**:
* Avoid synchronous file I/O in components.
* Do not introduce excessive property drilling; leverage or extend existing providers.
## Configuration & Settings
* **Settings vs Args**: Use settings for user-configurable options; do not add new CLI arguments.
* **Schema**: Add new settings to `packages/cli/src/config/settingsSchema.ts`.
* **Documentation**:
* If `showInDialog: true`, document in `docs/get-started/configuration.md`.
* Ensure `requiresRestart` is correctly set.
## Keyboard Shortcuts
* **Registration**: Define new shortcuts in `packages/cli/src/config/keyBindings.ts`.
* **Documentation**: Document all new shortcuts in `docs/cli/keyboard-shortcuts.md`.
* **Compatibility**: Avoid function keys and common VSCode shortcuts. Be cautious with `Meta` key (Mac-limited support).
## General
* **Logging**: Use `debugLogger` for errors. NEVER leave `console.log/warn/error` in the code.
* **TypeScript**: Avoid the non-null assertion operator (`!`).
In addition to the code context, you MUST strictly adhere to the following frontend-specific development guidelines when writing code in packages/cli.
## Testing Standards
* **Async Testing**: ALWAYS use `waitFor` from `packages/cli/src/test-utils/async.ts` instead of `vi.waitFor` to prevent flakiness and `act` warnings. NEVER use fixed waits (e.g., `await delay(100)`).
* **State Changes**: Wrap all blocks that change component state in `act`.
* **Snapshots**: Use `toMatchSnapshot` to verify rendering.
* **Rendering**: Use `render` or `renderWithProviders` from `packages/cli/src/test-utils/render.tsx` instead of `ink-testing-library` directly.
* **Mocking**:
* Reuse existing mocks/fakes where possible.
* **Parameterized Tests**: Use parameterized tests with explicit types to reduce duplication and ensure type safety.
* Avoid mocking the file system, os, or child_process if at all possible; if you have to mock the Mock critical dependencies (`fs`, `os`, `child_process`) do so ONLY at the top of the file.
## React & Ink Architecture
* **Keyboard Handling**: You MUST use `useKeyPress.ts` from Gemini CLI for keyboard handling, NOT the standard ink library. This is critical for supporting slow terminals and multiple events per frame.
* Handle multiple events gracefully (often requires a reducer pattern, see `text-buffer.ts`).
* **State Management**:
* NEVER trigger side effects from within the body of a `setState` callback. Use a reducer or `useRef` if necessary.
* Initialize state explicitly (e.g., use `undefined` rather than `true` if unknown).
* **Performance**:
* Avoid synchronous file I/O in components.
* Do not introduce excessive property drilling; leverage or extend existing providers.
## Configuration & Settings
* **Settings vs Args**: Use settings for user-configurable options; do not add new CLI arguments.
* **Schema**: Add new settings to `packages/cli/src/config/settingsSchema.ts`.
* **Documentation**:
* If `showInDialog: true`, document in `docs/get-started/configuration.md`.
* Ensure `requiresRestart` is correctly set.
## Keyboard Shortcuts
* **Registration**: Define new shortcuts in `packages/cli/src/config/keyBindings.ts`.
* **Documentation**: Document all new shortcuts in `docs/cli/keyboard-shortcuts.md`.
* **Compatibility**: Avoid function keys and common VSCode shortcuts. Be cautious with `Meta` key (Mac-limited support).
## General
* **Logging**: Use `debugLogger` for errors. NEVER leave `console.log/warn/error` in the code.
* **TypeScript**: Avoid the non-null assertion operator (`!`).
@@ -22,132 +22,9 @@ Follow these steps to conduct a thorough review:
4. Search the codebase if required.
5. Write a concise review of the changes, keeping in mind to encourage strong code quality and best practices. Pay particular attention to the Gemini MD file in the repo.
6. Consider ways the code may not be consistent with existing code in the repo. In particular it is critical that the react code uses patterns consistent with existing code in the repo.
7. Evaluate all tests on the changes and make sure that they are doing the following:
* Using `waitFor` from @{packages/cli/src/test-utils/async.ts} rather than
using `vi.waitFor` for all `waitFor` calls within `packages/cli`. Even if
tests pass, using the wrong `waitFor` could result in flaky tests as `act`
warnings could show up if timing is slightly different.
* Using `act` to wrap all blocks in tests that change component state.
* Using `toMatchSnapshot` to verify that rendering works as expected rather
than matching against the raw content of the output.
* If snapshots were changed as part of the changes, review the snapshots
changes to ensure they are intentional and comment if any look at all
suspicious. Too many snapshot changes that indicate bugs have been approved
in the past.
* Use `render` or `renderWithProviders` from
@{packages/cli/src/test-utils/render.tsx} rather than using `render` from
`ink-testing-library` directly. This is needed to ensure that we do not get
warnings about spurious `act` calls. If test cases specify providers
directly, consider whether the existing `renderWithProviders` should be
modified to support that use case.
* Ensure the test cases are using parameterized tests where that might reduce
the number of duplicated lines significantly.
* NEVER use fixed waits (e.g. 'await delay(100)'). Always use 'waitFor' with
a predicate to ensure tests are stable and fast.
* Ensure mocks are properly managed:
* Critical dependencies (fs, os, child_process) should only be mocked at
the top of the file. Ideally avoid mocking these dependencies altogether.
* Check to see if there are existing mocks or fakes that can be used rather
than creating new ones for the new tests added.
* Try to avoid mocking the file system whenever possible. If using the real
file system is difficult consider whether the test should be an
integration test rather than a unit test.
* `vi.restoreAllMocks()` should be called in `afterEach` to prevent test
pollution.
* Use `vi.useFakeTimers()` for tests involving time-based logic to avoid
flakiness.
* When creating parameterized tests, give the parameters types to ensure
that the tests are type-safe.
8. Evaluate all react logic carefully keeping in mind that the author of the
changes is not likely an expert on React. Key areas to audit carefully are:
* Whether `setState` calls trigger side effects from within the body of the
`setState` callback. If so, you *must* propose an alternate design using
reducers or other ways the code might be modified to not have to modify
state from within a `setState`. Make sure to comment about absolutely
every case like this as these cases have introduced multiple bugs in the
past. Typically these cases should be resolved using a reducer although
occassionally other techniques such as useRef are appropriate. Consider
suggesting that jacob314@ be tagged on the code review if the solution is
not 100% obvious.
* Whether code might introduce an infinite rendering loop in React.
* Whether keyboard handling is robust. Keyboard handling must go through
`useKeyPress.ts` from the Gemini CLI package rather than using the
standard ink library used by most keyboard handling. Unlike the standard
ink library, the keyboard handling library in Gemini CLI may report
multiple keyboard events one after another in the same React frame. This
is needed to support slow terminals but introduces complexity in all our
code that handles keyboard events. Handling this correctly often means
that reducers must be used or other mechanisms to ensure that multiple
state updates one after another are handled gracefully rather than
overriding values from the first update with the second update. Refer to
text-buffer.ts as a canonical example of using a reducer for this sort of
case.
* Ensure code does not use `console.log`, `console.warn`, or `console.error`
as these indicate debug logging that was accidentally left in the code.
* Avoid synchronous file I/O in React components as it will hang the UI.
* Ensure state initialization is explicit (e.g., use 'undefined' rather than
'true' as a default if the state is truly unknown initially).
* Carefully manage 'useEffect' dependencies. Prefer to use a reducer
whenever practical to resolve the issues. If that is not practical it is
ok to use 'useRef' to access the latest value of a prop or state inside an
effect without adding it to the dependency array if re-running the effect
is undesirable (common in event listeners).
* NEVER disable 'react-hooks/exhaustive-deps'. Fix the code to correctly
declare dependencies. Disabling this lint rule will almost always lead to
hard to detect bugs.
* Avoid making types nullable unless strictly necessary, as it hurts
readability.
* Do not introduce excessive property drilling. There are multiple providers
that can be leveraged to avoid property drilling. Make sure one of them
cannot be used. Do suggest a provider that might make sense to be extended
to include the new property or propose a new provider to add if the
property drilling is excessive. Only use providers for properties that are
* Ensure services are implemented as classes with clear lifecycle management (e.g., `initialize()` methods).
* Verify that `debugLogger` from `packages/core/src/utils/debugLogger.ts` is used for internal logging instead of `console`.
* Ensure all shell operations use `spawnAsync` from `packages/core/src/utils/shell-utils.ts` for consistent error handling and promise management.
* Check that filesystem errors are handled gracefully using `isNodeError` from `packages/core/src/utils/errors.ts`.
* Verify that new tools are added to `packages/core/src/tools/` and registered in `packages/core/src/tools/tool-registry.ts`.
* Ensure all new public services, utilities, and types are exported from `packages/core/src/index.ts`.
* Check that services are stateless where possible, or use the centralized `Storage` service for persistence.
* **Cross-Service Communication**: Prefer using the `coreEvents` bus (from `packages/core/src/utils/events.ts`) for asynchronous communication between services or to notify the UI of state changes. Avoid tight coupling between services.
10. Architectural Audit (Package Boundaries):
* **Logic Placement**: Non-UI logic (e.g., model orchestration, tool implementation, git/filesystem operations) MUST reside in `packages/core`. `packages/cli` should only contain UI/Ink components, command-line argument parsing, and user interaction logic.
* **Environment Isolation**: Core logic should not assume a TUI environment. Use the `ConfirmationBus` or `Output` abstractions for communicating with the user from Core.
* **Decoupling**: Actively look for opportunities to decouple services by using `coreEvents`. If a service is importing another service just to notify it of a change, it should probably be using an event instead.
11. General Gemini CLI design principles:
* Make sure that settings are only used for options that a user might
consider changing.
* Do not add new command line arguments and suggest settings instead.
* New settings must be added to packages/cli/src/config/settingsSchema.ts.
* If a setting has 'showInDialog: true', it MUST be documented in
docs/get-started/configuration.md.
* Ensure 'requiresRestart' is correctly set for new settings.
* Use 'debugLogger' for rethrown errors to avoid duplicate logging.
* All new keyboard shortcuts MUST be documented in
docs/cli/keyboard-shortcuts.md.
* Ensure new keyboard shortcuts are defined in
packages/cli/src/config/keyBindings.ts.
* If new keyboard shortcuts are added, remind the user to test them in
VSCode, iTerm2, Ghostty, and Windows to ensure they work for all
users.
* Be careful of keybindings that require the meta key as only certain
meta key shortcuts are supported on Mac.
* Be skeptical of function keys and keyboard shortcuts that are commonly
bound in VSCode as they may conflict.
12. TypeScript Best Practices:
* Use 'checkExhaustive' in the 'default' clause of 'switch' statements to
ensure all cases are handled.
* Avoid using the non-null assertion operator ('!') unless absolutely
necessary and you are confident the value is not null.
* **STRICT TYPING**: Strictly forbid 'any' and 'unknown' in both CLI and Core
packages. 'unknown' is only allowed if it is immediately narrowed using
type guards or Zod validation. Reject any code that uses 'any' or
'unknown' without narrowing.
13. **Ruthless Cleanup**:
* If you identify significant code duplication, technical debt, or "AISlop" (boilerplate, redundant comments), explicitly suggest initiating a `ruthless-refactorer` loop to clean it up.
14. Summarize all actionable findings into a concise but comprehensive directive output this to review_findings.md and advance to phase 2.
description: Trigger this skill when the user wants to start an asynchronous PR review, run background checks on a PR, or check the status of a previously started async PR review.
---
# Async PR Review
This skill provides a set of tools to asynchronously review a Pull Request. It will create a background job to run the project's preflight checks, execute Gemini-powered test plans, and perform a comprehensive code review using custom prompts.
This skill is designed to showcase an advanced "Agentic Asynchronous Pattern":
1.**Native Background Shells vs Headless Inference**: While Gemini CLI can natively spawn and detach background shell commands (using the `run_shell_command` tool with `is_background: true`), a standard bash background job cannot perform LLM inference. To conduct AI-driven code reviews and test generation in the background, the shell script *must* invoke the `gemini` executable headlessly using `-p`. This offloads the AI tasks to independent worker agents.
2.**Dynamic Git Scoping**: The review scripts avoid hardcoded paths. They use `git rev-parse --show-toplevel` to automatically resolve the root of the user's current project.
3.**Ephemeral Worktrees**: Instead of checking out branches in the user's main workspace, the skill provisions temporary git worktrees in `.gemini/tmp/async-reviews/pr-<number>`. This prevents git lock conflicts and namespace pollution.
4.**Agentic Evaluation (`check-async-review.sh`)**: The check script outputs clean JSON/text statuses for the main agent to parse. The interactive agent itself synthesizes the final assessment dynamically from the generated log files.
## Workflow
1.**Determine Action**: Establish whether the user wants to start a new async review or check the status of an existing one.
* If the user says "start an async review for PR #123" or similar, proceed to **Start Review**.
* If the user says "check the status of my async review for PR #123" or similar, proceed to **Check Status**.
### Start Review
If the user wants to start a new async PR review:
1. Ask the user for the PR number if they haven't provided it.
2. Execute the `async-review.sh` script, passing the PR number as the first argument. Be sure to run it with the `is_background` flag set to true to ensure it immediately detaches.
3. **Evaluate Output**: Read the output from the script.
* If the output contains `STATUS: IN_PROGRESS`, tell the user which tasks are still running.
* If the output contains `STATUS: COMPLETE`, use your file reading tools (`read_file`) to retrieve the contents of `final-assessment.md`, `review.md`, `pr-diff.diff`, `npm-test.log`, and `test-execution.log` files from the `LOG_DIR` specified in the output.
***Final Assessment**: Read those files, synthesize their results, and give the user a concise recommendation on whether the PR builds successfully, passes tests, and if you recommend they approve it based on the review.
{{ npm run clean && npm ci && npm run format && npm run build && npm run lint:ci && npm run typecheck;} > "${log_dir}/build-and-lint.log" 2>&1;echo$? > "${log_dir}/build-and-lint.exit";}&
# Dynamically resolve gemini binary (fallback to your nightly path)
echo"CI checks passed. Skipping local npm tests." > "${log_dir}/npm-test.log"
echo0 > "${log_dir}/npm-test.exit"
elif[["${ci_status}" -eq 8]];then
echo"CI checks are still pending. Skipping local npm tests to avoid duplicate work. Please check GitHub for final results." > "${log_dir}/npm-test.log"
echo"Found failing test files from CI:" >> "${log_dir}/npm-test.log"
for f in ${failed_files};doecho" - ${f}" >> "${log_dir}/npm-test.log";done
echo"Running ONLY failing tests locally..." >> "${log_dir}/npm-test.log"
exit_code=0
for file in ${failed_files};do
if[["${file}"== packages/* ]];then
ws_dir="$(echo"${file}"| cut -d'/' -f1,2)"
else
ws_dir="$(echo"${file}"| cut -d'/' -f1)"
fi
rel_file="${file#"${ws_dir}"/}"
echo"--- Running ${rel_file} in workspace ${ws_dir} ---" >> "${log_dir}/npm-test.log"
if ! npm run test:ci -w "${ws_dir}" -- "${rel_file}" >> "${log_dir}/npm-test.log" 2>&1;then
exit_code=1
fi
done
echo"${exit_code}" > "${log_dir}/npm-test.exit"
else
echo"Could not extract specific failing files. Skipping full local test suite as it takes too long. Please check CI logs manually." >> "${log_dir}/npm-test.log"
echo1 > "${log_dir}/npm-test.exit"
fi
fi
else
echo"Skipped due to build-and-lint failure" > "${log_dir}/npm-test.log"
echo1 > "${log_dir}/npm-test.exit"
fi
}&
echo" ↳ [5/5] Starting Gemini test execution (waiting for build and lint)..."
"${GEMINI_CMD}" --policy "${POLICY_PATH}" -p "Analyze the diff for PR ${pr_number} using 'gh pr diff ${pr_number}'. Instead of running the project's automated test suite (like 'npm test'), physically exercise the newly changed code in the terminal (e.g., by writing a temporary script to call the new functions, or testing the CLI command directly). Verify the feature's behavior works as expected. IMPORTANT: Do NOT modify any source code to fix errors. Just exercise the code and log the results, reporting any failures clearly. Do not ask for user confirmation." > "${log_dir}/test-execution.log" 2>&1;echo$? > "${log_dir}/test-execution.exit"
else
echo"Skipped due to build-and-lint failure" > "${log_dir}/test-execution.log"
echo1 > "${log_dir}/test-execution.exit"
fi
}&
echo"✅ All tasks dispatched!"
echo"You can monitor progress with: tail -f ${log_dir}/*.log"
echo"Read your review later at: ${log_dir}/review.md"
# Polling loop to wait for all background tasks to finish
echo"⏳ Tasks complete! Synthesizing final assessment..."
if ! "${GEMINI_CMD}" --policy "${POLICY_PATH}" -p "Read the review at ${log_dir}/review.md, the automated test logs at ${log_dir}/npm-test.log, and the manual test execution logs at ${log_dir}/test-execution.log. Summarize the results, state whether the build and tests passed based on ${log_dir}/build-and-lint.exit and ${log_dir}/npm-test.exit, and give a final recommendation for PR ${pr_number}." > "${log_dir}/final-assessment.md" 2>&1;then
echo$? > "${log_dir}/final-assessment.exit"
echo"❌ Final assessment synthesis failed!"
echo"Check ${log_dir}/final-assessment.md for details."
description: Guidance for creating, running, fixing, and promoting behavioral evaluations. Use when verifying agent decision logic, debugging failures, debugging prompt steering, or adding workspace regression tests.
---
# Behavioral Evals
## Overview
Behavioral evaluations (evals) are tests that validate the **agent's decision-making** (e.g., tool choice) rather than pure functionality. They are critical for verifying prompt changes, debugging steerability, and preventing regressions.
> [!NOTE]
> **Single Source of Truth**: For core concepts, policies, running tests, and general best practices, always refer to **[evals/README.md](file:///Users/abhipatel/code/gemini-cli/docs/evals/README.md)**.
---
## 🔄 Workflow Decision Tree
1.**Does a prompt/tool change need validation?**
**No* -> Normal integration tests.
**Yes* -> Continue below.
2.**Is it UI/Interaction heavy?**
**Yes* -> Use `appEvalTest` (`AppRig`). See **[creating.md](references/creating.md)**.
**No* -> Use `evalTest` (`TestRig`). See **[creating.md](references/creating.md)**.
3.**Is it a new test?**
**Yes* -> Set policy to `USUALLY_PASSES`.
**No* -> `ALWAYS_PASSES` (locks in regression).
4.**Are you fixing a failure or promoting a test?**
**Fixing* -> See **[fixing.md](references/fixing.md)**.
**Promoting* -> See **[promoting.md](references/promoting.md)**.
---
## 📋 Quick Checklist
### 1. Setup Workspace
Seed the workspace with necessary files using the `files` object to simulate a realistic scenario (e.g., NodeJS project with `package.json`).
**Details in **[creating.md](references/creating.md)***
### 2. Write Assertions
Audit agent decisions using `rig.setBreakpoint()` (AppRig only) or index verification on `rig.readToolLogs()`.
**Details in **[creating.md](references/creating.md)***
### 3. Verify
Run single tests locally with Vitest. Confirm stability locally before relying on CI workflows.
**See **[evals/README.md](file:///Users/abhipatel/code/gemini-cli/docs/evals/README.md)** for running commands.*
| **`evalTest`** | `./test-helper.js` | **Subprocess**. Runs the CLI in a separate process + waits for exit. | Standard workspace tests. **Do not use `setBreakpoint`**; auditing history (`readToolLogs`) is safer. |
| **`appEvalTest`** | `./app-test-helper.js` | **In-Process**. Runs directly inside the runner loop. | UI/Ink rendering. Safe for `setBreakpoint` triggers. |
---
## 🏗️ Scenario Design
Evals must simulate realistic agent environments to effectively test
decision-making.
- **Workspace State**: Seed with standard project anchors if testing general
2.**Isolate**: DO NOT push changes or start remote runs. Confine investigation
to the local workspace.
3.**Read Logs**:
- Eval logs live in `evals/logs/<test_name>.log`.
- Enable verbose debugging via `export GEMINI_DEBUG_LOG_FILE="debug.log"`.
4.**Diagnose**: Audit tool logs and telemetry. Note if due to setup/assert.
- **Tip**: Proactively add custom logging/diagnostics to check hypotheses.
---
## 2. 🛠️ Fix Strategy
1.**Targeted Location**: Locate the test case and the corresponding
prompt/code.
2.**Iterative Scope**: Make extreme change first to verify scope, then refine
to a minimal, targeted change.
3.**Assertion Fidelity**:
- Changing the test prompt is a **last resort** (prompts are often vague by
design).
- **Warning**: Do not lose test fidelity by making prompts too direct/easy.
- **Primary Fix Trigger**: Adjust tool descriptions, system prompts
(`snippets.ts`), or **modules that contribute to the prompt template**.
- Fixes should generally try to improve the prompt `@packages/core/src/prompts/snippets.ts` first.
- **Instructional Generality**: Changes to the system prompt should aim to be as general as possible while still accomplishing the goal. Specificity should be added only as needed.
- **Principle**: Instead of creating "forbidden lists" for specific syntax (e.g., "Don't use `Object.create()`"), formulate a broader engineering principle that covers the underlying issue (e.g., "Prioritize explicit composition over hidden prototype manipulation"). This improves steerability across a wider range of similar scenarios.
- *Low Specificity*: "Follow ecosystem best practices"
- *Medium Specificity*: "Utilize OOP and functional best practices, as applicable"
- *High Specificity*: Provide ecosystem-specific hints as examples of a broader principle rather than direct instructions. e.g., "NEVER use hacks like bypassing the type system or employing 'hidden' logic (e.g.: reflection, prototype manipulation). Instead, use explicit and idiomatic language features (e.g.: type guards, explicit class instantiation, or object spread) that maintain structural integrity."
- **Prompt Simplification**: Once the test is passing, use `ask_user` to determine if prompt simplification is desired.
- **Criteria**: Simplification should be attempted only if there are related clauses that can be de-duplicated or reparented under a single heading.
- **Verification**: As part of simplification, you MUST identify and run any behavioral eval tests that might be affected by the changes to ensure no regressions are introduced.
- Test fixes should not "cheat" by changing a test's `GEMINI.md` file or by updating the test's prompt to instruct it to not repro the bug.
- **Warning**: Prompts have multiple configurations; ensure your fix targets
the correct config for the model in question.
4.**Architecture Options**: If prompt or instruction tuning triggers no
improvement, analyze loop composition.
- **AgentLoop**: Defined by `context + toolset + prompt`.
- **Enhancements**: Loops perform best with direct prompts, fewer irrelevant
tools, low goal density, and minimal low-value/irrelevant context.
- **Modifications**: Compose subagents or isolate tools. Ground in observed
traces.
- **Warning**: Think deeply before offering recommendations; avoid parroting
abstract design guidelines.
---
## 3. ✅ Verify
1.**Run Local**: Run Vitest in non-interactive mode on just the file.
2.**Log Audit**: Prioritize diagnosing failures via log comparison before
triggering heavy test runs.
3.**Stability Limit**: Run the test **3 times** locally on key models (can use
scripts to run in parallel for speed):
- **Gemini 3.0**
- **Gemini 3 Flash**
- **Gemini 2.5 Pro**
4.**Flakiness Rule**: If it passes 2/3 times, it may be inherent noise
difficult to improve without a structural split.
---
## 4. 📊 Report
Provide a summary of:
- Test success rate for each tested model (e.g., 3/3 = 100%).
- Root cause identification and fix explanation.
- If unfixed, provide high-confidence architecture recommendations.
description: Use this skill if the user asks you to help them address GitHub PR comments for their current branch of the Gemini CLI. Requires `gh` CLI tool.
---
You are helping the user address comments on their Pull Request. These comments may have come from an automated review agent or a team member.
OBJECTIVE: Help the user review and address comments on their PR.
# Comment Review Procedure
1. Run the `scripts/fetch-pr-info.js` script to get PR info and state. MAKE SURE you read the entire output of the command, even if it gets truncated.
2. Summarize the review status by analyzing the diff, commit log, and comments to see which still need to be addressed. Pay attention to the current user's comments. For resolved threads, summarize as a single line with a ✅. For open threads, provide a reference number e.g. [1] and the comment content.
3. Present your summary of the feedback and current state and allow the user to guide you as to what to fix/address/skip. DO NOT begin fixing issues automatically.
description: Use this skill during code reviews to proactively investigate the codebase for duplicated functionality, reinvented wheels, or failure to reuse existing project best practices and shared utilities.
---
# Review Duplication
## Overview
This skill provides a structured workflow for investigating a codebase during a code review to identify duplicated logic, reinvented utilities, and missed opportunities to reuse established patterns. By executing this workflow, you ensure that new code integrates seamlessly with the existing project architecture.
## Workflow: Investigating for Duplication
When reviewing code, perform the following steps before finalizing your review:
### 1. Extract Core Logic
Analyze the new code to identify the core algorithms, utility functions, generic data structures, or UI components being introduced. Look beyond the specific business logic to see the underlying mechanics.
- **Core Logic:** Call out `packages/core/` if functionality does not appear React UI specific.
**Trace Third-Party Dependencies:** If the PR introduces a new import for a utility library (e.g., `lodash.merge`, `date-fns`), trace how and where the project currently uses that library. There is likely an existing wrapper or shared utility.
**Check Package Files:** Before flagging a custom implementation of a complex algorithm, check `package.json` to see if a standard library (like `lodash` or `uuid`) is already installed that provides this functionality.
### 3. Investigate the Codebase (Sub-Agent Delegation)
Delegate the heavy lifting of codebase investigation to specialized sub-agents. They are optimized to perform deep searches and semantic mapping without bloating your session history.
To ensure a comprehensive review, you MUST formulate highly specific objectives for the sub-agents, providing them with the "scents" you discovered in Step 1.
- **Codebase Investigator:** Use the `codebase_investigator` as your primary researcher. When delegating, formulate an objective that asks specific, investigative questions about the codebase, explicitly including these search vectors:
- **Structural Similarity:** Ask if existing code uses the same underlying APIs (e.g., "Does any existing code use `Intl.DateTimeFormat` or `setTimeout` for similar purposes?").
- **Naming Conventions:** Ask if there are existing symbols with similar naming patterns (e.g., "Are there existing symbols with naming patterns like `*Format*` or `*Debounce*`?").
- **Comments & Documentation:** Ask if keywords from the PR's comments or JSDoc exist in describing similar behavior elsewhere.
- **Architectural Fit:** Ask where this type of logic is currently centralized (e.g., "Where is centralized date formatting logic located?").
- **Refactoring Guidance:** Crucially, ask the sub-agent to explain *how* the new code could be refactored to use any existing logic it finds.
- **Generalist Agent:** Use the `generalist` for detailed, turn-intensive comparisons. For example: "Review the implementation of `MyNewComponent` in the PR and compare it semantically against all components in `packages/ui/src`. Are there any existing components that could be extended or used instead?"
- **Retain Fast Path for Simple Searches:** For extremely simple, unambiguous checks (e.g., "Does `package.json` include `lodash`?"), perform a direct search to save time. Default to delegation for any open-ended "investigations."
### 4. Evaluate Best Practices
Check if the new code aligns with the project's established conventions.
- **Error Handling:** Does it use the project's standard error classes or logging mechanisms?
- **State Management:** Does it bypass established stores or contexts?
- **Styling:** Does it hardcode colors or spacing instead of using theme variables?
If the PR introduces a new pattern, compare it against the documented standards and explicitly confirm if an existing project pattern should have been used instead.
### 5. Formulate Constructive Feedback
If you discover that the PR duplicates existing functionality or ignores a best practice:
- Provide a clear review comment.
- **Identify the Source:** Explicitly mention the absolute or project-relative file path and the specific symbol (function, component, class) that should be reused.
- **Implementation Guidance:** Provide a brief code snippet or a clear explanation showing **how** to integrate the existing code to fulfill the task's requirements.
- **Explain the Value:** Briefly explain why reusing the existing code is beneficial (e.g., maintainability, consistency, built-in edge case handling).
Example comment:
> "It looks like this PR introduces a new `formatDate` utility. We already have a robust, tested `formatDate` function in `src/utils/dateHelpers.ts`.
>
> You can replace your implementation by importing it like this:
> ```typescript
> import { formatDate } from '../utils/dateHelpers';
echo "Dry run: Would have added tag '${{ inputs.channel }}' to version '${{ inputs.version }}' for ${{ inputs.cli-package-name }}, ${{ inputs.core-package-name }}, and ${{ inputs.a2a-package-name }}."
echo "Dry run: Would have added tag '${INPUTS_CHANNEL}' to version '${INPUTS_VERSION}' for ${INPUTS_CLI_PACKAGE_NAME}, ${INPUTS_CORE_PACKAGE_NAME}, and ${INPUTS_A2A_PACKAGE_NAME}."
if [[ "${{ steps.detect.outputs.STEERING_DETECTED }}" == "true" ]]; then
echo "### 🧠 Model Steering Guidance"
echo ""
echo "This PR modifies files that affect the model's behavior (prompts, tools, or instructions)."
echo ""
if [[ "${{ steps.analysis.outputs.MISSING_EVALS }}" == "true" ]]; then
echo "- ⚠️ **Consider adding Evals:** No behavioral evaluations (\`evals/*.eval.ts\`) were added or updated in this PR. Consider [adding a test case](https://github.com/google-gemini/gemini-cli/blob/main/evals/README.md#creating-an-evaluation) to verify the new behavior and prevent regressions."
fi
if [[ "${{ steps.analysis.outputs.IS_MAINTAINER }}" == "true" ]]; then
echo "- 🚀 **Maintainer Reminder:** Please ensure that these changes do not regress results on benchmark evals before merging."
fi
fi
echo ""
echo "---"
echo "*This is an automated guidance message triggered by steering logic signatures.*"
echo "<!-- eval-pr-report -->"
} > full_comment.md
# 2. Find if a comment with our unique tag already exists
# We extract the numeric ID from the URL to ensure compatibility with the REST API
- Description: Issues related to user-facing documentation and other content on the documentation website.
- Example Issues:
"A typo in a README file."
"DOCS: A command is not working as described in the documentation."
"A request for a new documentation page."
"Instructions missing for skills feature"
area/unknown
- Description: Issues that do not clearly fit into any other defined area/ category, or where information is too limited to make a determination. Use this when no other area is appropriate.
@@ -284,8 +296,21 @@ jobs:
return;
}
} else {
core.setFailed(`Output is not valid JSON and does not contain a JSON markdown block.\nRaw output: ${rawOutput}`);
return;
// If no markdown block, try to find a raw JSON object in the output.
// The CLI may include debug/log lines (e.g. telemetry init, YOLO mode)
// Closure Policy: No help-wanted label = Close after 14 days
if (pr.state === 'open' && !hasHelpWanted && !hasMaintainerOnly) {
const prCreatedAt = new Date(pr.created_at);
// We give a 14-day grace period for non-help-wanted PRs to be manually reviewed/labeled by an EM
if (prCreatedAt > fourteenDaysAgo) {
core.info(`PR #${pr.number} is new and lacks 'help wanted'. Giving 14-day grace period for EM review.`);
continue;
}
core.info(`PR #${pr.number} is older than 14 days and lacks 'help wanted' association. Closing.`);
if (!dryRun) {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: pr.number,
body: "Hi there! Thank you for your contribution to Gemini CLI. \n\nTo improve our contribution process and better track changes, we now require all pull requests to be associated with an existing issue, as announced in our [recent discussion](https://github.com/google-gemini/gemini-cli/discussions/16706) and as detailed in our [CONTRIBUTING.md](https://github.com/google-gemini/gemini-cli/blob/main/CONTRIBUTING.md#1-link-to-an-existing-issue).\n\nThis pull request is being closed because it is not currently linked to an issue. **Once you have updated the description of this PR to link an issue (e.g., by adding `Fixes #123` or `Related to #123`), it will be automatically reopened.**\n\n**How to link an issue:**\nAdd a keyword followed by the issue number (e.g., `Fixes #123`) in the description of your pull request. For more details on supported keywords and how linking works, please refer to the [GitHub Documentation on linking pull requests to issues](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue).\n\nThank you for your understanding and for being a part of our community!"
body: "Hi there! Thank you for your interest in contributing to Gemini CLI. \n\nTo ensure we maintain high code quality and focus on our prioritized roadmap, we have updated our contribution policy (see [Discussion #17383](https://github.com/google-gemini/gemini-cli/discussions/17383)). \n\n**We only *guarantee* review and consideration of pull requests for issues that are explicitly labeled as 'help wanted'.** All other community pull requests are subject to closure after 14 days if they do not align with our current focus areas. For this reason, we strongly recommend that contributors only submit pull requests against issues explicitly labeled as **'help-wanted'**. \n\nThis pull request is being closed as it has been open for 14 days without a 'help wanted' designation. We encourage you to find and contribute to existing 'help wanted' issues in our backlog! Thank you for your understanding and for being part of our community!"
});
await github.rest.pulls.update({
owner: context.repo.owner,
@@ -152,64 +192,55 @@ jobs:
continue;
}
// Staleness check (Scheduled runs only)
if (pr.state === 'open' && context.eventName !== 'pull_request') {
core.info(`PR #${pr.number} is stale but has a protected label. Skipping closure.`);
continue;
}
core.info(`PR #${pr.number} is stale (no maintainer activity for 30+ days). Closing.`);
if (!dryRun) {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: pr.number,
body: "Hi there! Thank you for your contribution to Gemini CLI. We really appreciate the time and effort you've put into this pull request.\n\nTo keep our backlog manageable and ensure we're focusing on current priorities, we are closing pull requests that haven't seen maintainer activity for 30 days. Currently, the team is prioritizing work associated with **🔒 maintainer only** or **help wanted** issues.\n\nIf you believe this change is still critical, please feel free to comment with updated details. Otherwise, we encourage contributors to focus on open issues labeled as **help wanted**. Thank you for your understanding!"
body: "Hi there! Thank you for your contribution. To keep our backlog manageable, we are closing pull requests that haven't seen maintainer activity for 30 days. If you're still working on this, please let us know!"
body: `👋 @${commenter}, thanks for your interest in this issue! We're reserving self-assignment for issues that have been marked with the \`help wanted\` label. Feel free to check out our list of [issues that need attention](https://github.com/google-gemini/gemini-cli/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22).`
});
return;
}
// Search for open issues already assigned to the commenter in this repo
body: `👋 @${commenter}, you've been assigned to this issue! Thank you for taking the time to contribute. Make sure to check out our [contributing guidelines](https://github.com/google-gemini/gemini-cli/blob/main/CONTRIBUTING.md).`
**⏳ Status:** The patch creation workflow has been triggered and is waiting for deployment approval. Please visit the specific workflow links below and approve the runs.
**⏳ Status:** The patch creation workflow has been triggered and is waiting for deployment approval. Please visit the workflow history link below and approve the runs.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.