Compare commits

...

7 Commits

Author SHA1 Message Date
Adam Weidman 3d76c80da5 test(core): validate external AbortSignal propagation through LocalSubagentSession
The existing abort test mocked the executor to always reject, meaning it
would pass even if the abort wiring was broken. Add a test that actually
fires the caller's AbortSignal and asserts it propagates through the
abortListener -> session.abort() -> internal AbortController -> executor
chain.
2026-03-26 16:28:01 -04:00
Adam Weidman d034c4dd96 test(core): add unit tests for LocalSubagentProtocol and RemoteSubagentProtocol
Adds 58 tests covering lifecycle events, activity translation, config
buffering, session state persistence, auth setup, error handling, abort,
and concurrent send() guard for both new protocol implementations.

Also fixes a timing bug in RemoteSubagentProtocol where _sessionState was
persisted in a finally block, causing it to be written after the result
promise settled. Moved _saveSessionState() calls to run synchronously
before each _resultResolve/_resultReject so callers see updated
contextId/taskId immediately after getResult() settles.
2026-03-26 15:56:14 -04:00
Adam Weidman f1a3387160 feat(core): wrap local and remote subagents behind AgentProtocol/AgentSession
Introduce LocalSubagentProtocol and RemoteSubagentProtocol that implement
AgentProtocol, wrapping LocalAgentExecutor and A2A client streaming
respectively. LocalSubagentSession and RemoteSubagentSession extend
AgentSession and are the public entry points.

- LocalSubagentProtocol: translates SubagentActivityEvent -> AgentEvent
  (THOUGHT_CHUNK->message/thought, TOOL_CALL_START->tool_request,
  TOOL_CALL_END->tool_response, ERROR->error). Accepts optional
  rawActivityCallback for rich SubagentProgress display without losing
  displayName/errorType detail that AgentEvent types do not carry.

- RemoteSubagentProtocol: wraps A2A sendMessageStream, maintains
  contextId/taskId session state, tracks SubagentProgress per chunk
  for error recovery, and returns a ToolResult with proper
  SubagentProgress as returnDisplay.

- LocalSubagentInvocation: now uses LocalSubagentSession internally,
  preserving all existing SubagentProgress display logic via
  rawActivityCallback. External AbortSignal wired through session.abort().

- RemoteAgentInvocation: now uses RemoteSubagentSession, subscribing
  to message events for live progress updates.

- SubagentToolWrapper and SubagentTool: add optional onAgentEvent
  callback for future parent session observability (currently unused,
  wired through invocation constructors to avoid a second pass later).

- index.ts: export LocalSubagentSession and RemoteSubagentSession.

No behavioral change to SubagentProgress display or ToolResult output.
2026-03-26 12:09:47 -04:00
Sehoon Shon 49534209f2 fix(cli): prioritize primary name matches in slash command search (#23850) 2026-03-26 12:18:57 +00:00
Chris Williams 9e7f52b8f5 Merge examples of use into quickstart documentation (#23319) 2026-03-26 02:57:23 +00:00
Gal Zahavi 30e0ab102a feat(sandbox): dynamic Linux sandbox expansion and worktree support (#23692)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-03-26 01:58:45 +00:00
Alisa 2e03e3aed5 feat(evals): add reliability harvester and 500/503 retry support (#23626) 2026-03-26 01:48:45 +00:00
38 changed files with 4048 additions and 949 deletions
+12
View File
@@ -334,8 +334,20 @@ jobs:
if: "${{ steps.check_evals.outputs.should_run == 'true' }}"
env:
GEMINI_API_KEY: '${{ secrets.GEMINI_API_KEY }}'
GEMINI_MODEL: 'gemini-3-pro-preview'
# Disable Vitest internal retries to avoid double-retrying;
# custom retry logic is handled in evals/test-helper.ts
VITEST_RETRY: 0
run: 'npm run test:always_passing_evals'
- name: 'Upload Reliability Logs'
if: "always() && steps.check_evals.outputs.should_run == 'true'"
uses: 'actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02' # ratchet:actions/upload-artifact@v4
with:
name: 'eval-logs-${{ github.run_id }}-${{ github.run_attempt }}'
path: 'evals/logs/api-reliability.jsonl'
retention-days: 7
e2e:
name: 'E2E'
if: |
+2
View File
@@ -61,6 +61,8 @@ jobs:
GEMINI_MODEL: '${{ matrix.model }}'
RUN_EVALS: "${{ github.event.inputs.run_all != 'false' }}"
TEST_NAME_PATTERN: '${{ github.event.inputs.test_name_pattern }}'
# Disable Vitest internal retries to avoid double-retrying;
# custom retry logic is handled in evals/test-helper.ts
VITEST_RETRY: 0
run: |
CMD="npm run test:all_evals"
-141
View File
@@ -1,141 +0,0 @@
# Gemini CLI examples
Gemini CLI helps you automate common engineering tasks by combining AI reasoning
with local system tools. This document provides examples of how to use the CLI
for file management, code analysis, and data transformation.
<!-- prettier-ignore -->
> [!NOTE]
> These examples demonstrate potential capabilities. Your actual
> results can vary based on the model used and your project environment.
## Rename your photographs based on content
You can use Gemini CLI to automate file management tasks that require visual
analysis. In this example, Gemini CLI renames images based on their actual
subject matter.
Scenario: You have a folder containing the following files:
```bash
photos/photo1.png
photos/photo2.png
photos/photo3.png
```
Give Gemini the following prompt:
```cli
Rename the photos in my "photos" directory based on their contents.
```
Result: Gemini asks for permission to rename your files.
Select **Allow once** and your files are renamed:
```bash
photos/yellow_flowers.png
photos/antique_dresser.png
photos/green_android_robot.png
```
## Explain a repository by reading its code
Gemini CLI is effective for rapid codebase exploration. The following example
shows how to ask Gemini CLI to fetch, analyze, and summarize a remote project.
Scenario: You want to understand how a popular open-source utility works by
inspecting its code, not just its README.
Give Gemini CLI the following prompt:
```cli
Clone the 'chalk' repository from https://github.com/chalk/chalk, read its key source files, and explain how it works.
```
Result: Gemini performs a sequence of actions to answer your request.
1. First, it asks for permission to run `git clone` to download the repository.
2. Next, it finds the important source files and asks for permission to read
them.
3. Finally, after analyzing the code, it provides a summary.
Gemini CLI returns an explanation based on the actual source code:
```markdown
The `chalk` library is a popular npm package for styling terminal output with
colors. After analyzing the source code, here's how it works:
- **Core functionality:** The main file sets up a chainable API. Each color or
modifier (like `bold` or `italic`) is a getter that appends the corresponding
ANSI escape code to an internal stack.
- **Chainable API:** When you call `chalk.red.bold('Hello')`, you are accessing
getters. The `red` getter adds the red color code, and the `bold` getter adds
the bold code.
- **Output generation:** When the chain is treated as a string (e.g., in
`console.log`), a final `toString()` method is called. This method joins all
the stored ANSI codes, wraps them around the input string ('Hello'), and adds
a reset code at the end. This produces the final, styled string that the
terminal can render.
```
## Combine two spreadsheets into one spreadsheet
Gemini CLI can process and transform data across multiple files. Use this
capability to merge reports or reformat data sets without manual copying.
Scenario: You have two .csv files: `Revenue - 2023.csv` and
`Revenue - 2024.csv`. Each file contains monthly revenue figures.
Give Gemini CLI the following prompt:
```cli
Combine the two .csv files into a single .csv file, with each year a different column.
```
Result: Gemini CLI reads each file and then asks for permission to write a new
file. Provide your permission and Gemini CLI provides the combined data:
```csv
Month,2023,2024
January,0,1000
February,0,1200
March,0,2400
April,900,500
May,1000,800
June,1000,900
July,1200,1000
August,1800,400
September,2000,2000
October,2400,3400
November,3400,1800
December,2100,9000
```
## Run unit tests
Gemini CLI can generate boilerplate code and tests based on your existing
implementation. This example demonstrates how to request code coverage for a
JavaScript component.
Scenario: You've written a simple login page. You wish to write unit tests to
ensure that your login page has code coverage.
Give Gemini CLI the following prompt:
```cli
Write unit tests for Login.js.
```
Result: Gemini CLI asks for permission to write a new file and creates a test
for your login page.
## Next steps
- Follow the [File management](../cli/tutorials/file-management.md) guide to
start working with your codebase.
- Follow the [Quickstart](./index.md) to start your first session.
- See the [Cheatsheet](../cli/cli-reference.md) for a quick reference of
available commands.
+127 -1
View File
@@ -62,7 +62,133 @@ Once installed and authenticated, you can start using Gemini CLI by issuing
commands and prompts in your terminal. Ask it to generate code, explain files,
and more.
To explore the power of Gemini CLI, see [Gemini CLI examples](./examples.md).
<!-- prettier-ignore -->
> [!NOTE]
> These examples demonstrate potential capabilities. Your actual
> results can vary based on the model used and your project environment.
### Rename your photographs based on content
You can use Gemini CLI to automate file management tasks that require visual
analysis. In this example, Gemini CLI renames images based on their actual
subject matter.
Scenario: You have a folder containing the following files:
```bash
photos/photo1.png
photos/photo2.png
photos/photo3.png
```
Give Gemini the following prompt:
```cli
Rename the photos in my "photos" directory based on their contents.
```
Result: Gemini asks for permission to rename your files.
Select **Allow once** and your files are renamed:
```bash
photos/yellow_flowers.png
photos/antique_dresser.png
photos/green_android_robot.png
```
### Explain a repository by reading its code
Gemini CLI is effective for rapid codebase exploration. The following example
shows how to ask Gemini CLI to fetch, analyze, and summarize a remote project.
Scenario: You want to understand how a popular open-source utility works by
inspecting its code, not just its README.
Give Gemini CLI the following prompt:
```cli
Clone the 'chalk' repository from https://github.com/chalk/chalk, read its key source files, and explain how it works.
```
Result: Gemini performs a sequence of actions to answer your request.
1. First, it asks for permission to run `git clone` to download the repository.
2. Next, it finds the important source files and asks for permission to read
them.
3. Finally, after analyzing the code, it provides a summary.
Gemini CLI returns an explanation based on the actual source code:
```markdown
The `chalk` library is a popular npm package for styling terminal output with
colors. After analyzing the source code, here's how it works:
- **Core functionality:** The main file sets up a chainable API. Each color or
modifier (like `bold` or `italic`) is a getter that appends the corresponding
ANSI escape code to an internal stack.
- **Chainable API:** When you call `chalk.red.bold('Hello')`, you are accessing
getters. The `red` getter adds the red color code, and the `bold` getter adds
the bold code.
- **Output generation:** When the chain is treated as a string (e.g., in
`console.log`), a final `toString()` method is called. This method joins all
the stored ANSI codes, wraps them around the input string ('Hello'), and adds
a reset code at the end. This produces the final, styled string that the
terminal can render.
```
### Combine two spreadsheets into one spreadsheet
Gemini CLI can process and transform data across multiple files. Use this
capability to merge reports or reformat data sets without manual copying.
Scenario: You have two .csv files: `Revenue - 2023.csv` and
`Revenue - 2024.csv`. Each file contains monthly revenue figures.
Give Gemini CLI the following prompt:
```cli
Combine the two .csv files into a single .csv file, with each year a different column.
```
Result: Gemini CLI reads each file and then asks for permission to write a new
file. Provide your permission and Gemini CLI provides the combined data:
```csv
Month,2023,2024
January,0,1000
February,0,1200
March,0,2400
April,900,500
May,1000,800
June,1000,900
July,1200,1000
August,1800,400
September,2000,2000
October,2400,3400
November,3400,1800
December,2100,9000
```
### Run unit tests
Gemini CLI can generate boilerplate code and tests based on your existing
implementation. This example demonstrates how to request code coverage for a
JavaScript component.
Scenario: You've written a simple login page. You wish to write unit tests to
ensure that your login page has code coverage.
Give Gemini CLI the following prompt:
```cli
Write unit tests for Login.js.
```
Result: Gemini CLI asks for permission to write a new file and creates a test
for your login page.
## Check usage and quota
-2
View File
@@ -19,8 +19,6 @@ Jump in to Gemini CLI.
on your system.
- **[Authentication](./get-started/authentication.md):** Setup instructions for
personal and enterprise accounts.
- **[Examples](./get-started/examples.md):** Practical examples of Gemini CLI in
action.
- **[CLI cheatsheet](./cli/cli-reference.md):** A quick reference for common
commands and options.
- **[Gemini 3 on Gemini CLI](./get-started/gemini-3.md):** Learn about Gemini 3
+1
View File
@@ -13,6 +13,7 @@
"/docs/faq": "/docs/resources/faq",
"/docs/get-started/configuration": "/docs/reference/configuration",
"/docs/get-started/configuration-v1": "/docs/reference/configuration",
"/docs/get-started/examples": "/docs/get-started/index",
"/docs/index": "/docs",
"/docs/quota-and-pricing": "/docs/resources/quota-and-pricing",
"/docs/tos-privacy": "/docs/resources/tos-privacy",
-1
View File
@@ -12,7 +12,6 @@
"label": "Authentication",
"slug": "docs/get-started/authentication"
},
{ "label": "Examples", "slug": "docs/get-started/examples" },
{ "label": "CLI cheatsheet", "slug": "docs/cli/cli-reference" },
{
"label": "Gemini 3 on Gemini CLI",
+207
View File
@@ -0,0 +1,207 @@
/**
* @license
* Copyright 2026 Google LLC
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import fs from 'node:fs';
import path from 'node:path';
import { internalEvalTest } from './test-helper.js';
import { TestRig } from '@google/gemini-cli-test-utils';
// Mock TestRig to control API success/failure
vi.mock('@google/gemini-cli-test-utils', () => {
return {
TestRig: vi.fn().mockImplementation(() => ({
setup: vi.fn(),
run: vi.fn(),
cleanup: vi.fn(),
readToolLogs: vi.fn().mockReturnValue([]),
_lastRunStderr: '',
})),
};
});
describe('evalTest reliability logic', () => {
const LOG_DIR = path.resolve(process.cwd(), 'evals/logs');
const RELIABILITY_LOG = path.join(LOG_DIR, 'api-reliability.jsonl');
beforeEach(() => {
vi.clearAllMocks();
if (fs.existsSync(RELIABILITY_LOG)) {
fs.unlinkSync(RELIABILITY_LOG);
}
});
afterEach(() => {
if (fs.existsSync(RELIABILITY_LOG)) {
fs.unlinkSync(RELIABILITY_LOG);
}
});
it('should retry 3 times on 500 INTERNAL error and then SKIP', async () => {
const mockRig = new TestRig() as any;
(TestRig as any).mockReturnValue(mockRig);
// Simulate permanent 500 error
mockRig.run.mockRejectedValue(new Error('status: INTERNAL - API Down'));
// Execute the test function directly
await internalEvalTest({
name: 'test-api-failure',
prompt: 'do something',
assert: async () => {},
});
// Verify retries: 1 initial + 3 retries = 4 setups/runs
expect(mockRig.run).toHaveBeenCalledTimes(4);
// Verify log content
const logContent = fs
.readFileSync(RELIABILITY_LOG, 'utf-8')
.trim()
.split('\n');
expect(logContent.length).toBe(4);
const entries = logContent.map((line) => JSON.parse(line));
expect(entries[0].status).toBe('RETRY');
expect(entries[0].attempt).toBe(0);
expect(entries[3].status).toBe('SKIP');
expect(entries[3].attempt).toBe(3);
expect(entries[3].testName).toBe('test-api-failure');
});
it('should fail immediately on non-500 errors (like assertion failures)', async () => {
const mockRig = new TestRig() as any;
(TestRig as any).mockReturnValue(mockRig);
// Simulate a real logic error/bug
mockRig.run.mockResolvedValue('Success');
const assertError = new Error('Assertion failed: expected foo to be bar');
// Expect the test function to throw immediately
await expect(
internalEvalTest({
name: 'test-logic-failure',
prompt: 'do something',
assert: async () => {
throw assertError;
},
}),
).rejects.toThrow('Assertion failed');
// Verify NO retries: only 1 attempt
expect(mockRig.run).toHaveBeenCalledTimes(1);
// Verify NO reliability log was created (it's not an API error)
expect(fs.existsSync(RELIABILITY_LOG)).toBe(false);
});
it('should recover if a retry succeeds', async () => {
const mockRig = new TestRig() as any;
(TestRig as any).mockReturnValue(mockRig);
// Fail once, then succeed
mockRig.run
.mockRejectedValueOnce(new Error('status: INTERNAL'))
.mockResolvedValueOnce('Success');
await internalEvalTest({
name: 'test-recovery',
prompt: 'do something',
assert: async () => {},
});
// Ran twice: initial (fail) + retry 1 (success)
expect(mockRig.run).toHaveBeenCalledTimes(2);
// Log should only have the one RETRY entry
const logContent = fs
.readFileSync(RELIABILITY_LOG, 'utf-8')
.trim()
.split('\n');
expect(logContent.length).toBe(1);
expect(JSON.parse(logContent[0]).status).toBe('RETRY');
});
it('should retry 3 times on 503 UNAVAILABLE error and then SKIP', async () => {
const mockRig = new TestRig() as any;
(TestRig as any).mockReturnValue(mockRig);
// Simulate permanent 503 error
mockRig.run.mockRejectedValue(
new Error('status: UNAVAILABLE - Service Busy'),
);
await internalEvalTest({
name: 'test-api-503',
prompt: 'do something',
assert: async () => {},
});
expect(mockRig.run).toHaveBeenCalledTimes(4);
const logContent = fs
.readFileSync(RELIABILITY_LOG, 'utf-8')
.trim()
.split('\n');
const entries = logContent.map((line) => JSON.parse(line));
expect(entries[0].errorCode).toBe('503');
expect(entries[3].status).toBe('SKIP');
});
it('should throw if an absolute path is used in files', async () => {
const mockRig = new TestRig() as any;
(TestRig as any).mockReturnValue(mockRig);
mockRig.testDir = path.resolve(process.cwd(), 'test-dir-tmp');
if (!fs.existsSync(mockRig.testDir)) {
fs.mkdirSync(mockRig.testDir, { recursive: true });
}
try {
await expect(
internalEvalTest({
name: 'test-absolute-path',
prompt: 'do something',
files: {
'/etc/passwd': 'hacked',
},
assert: async () => {},
}),
).rejects.toThrow('Invalid file path in test case: /etc/passwd');
} finally {
if (fs.existsSync(mockRig.testDir)) {
fs.rmSync(mockRig.testDir, { recursive: true, force: true });
}
}
});
it('should throw if directory traversal is detected in files', async () => {
const mockRig = new TestRig() as any;
(TestRig as any).mockReturnValue(mockRig);
mockRig.testDir = path.resolve(process.cwd(), 'test-dir-tmp');
// Create a mock test-dir
if (!fs.existsSync(mockRig.testDir)) {
fs.mkdirSync(mockRig.testDir, { recursive: true });
}
try {
await expect(
internalEvalTest({
name: 'test-traversal',
prompt: 'do something',
files: {
'../sensitive.txt': 'hacked',
},
assert: async () => {},
}),
).rejects.toThrow('Invalid file path in test case: ../sensitive.txt');
} finally {
if (fs.existsSync(mockRig.testDir)) {
fs.rmSync(mockRig.testDir, { recursive: true, force: true });
}
}
});
});
+171 -71
View File
@@ -39,87 +39,34 @@ export * from '@google/gemini-cli-test-utils';
export type EvalPolicy = 'ALWAYS_PASSES' | 'USUALLY_PASSES';
export function evalTest(policy: EvalPolicy, evalCase: EvalCase) {
const fn = async () => {
runEval(
policy,
evalCase.name,
() => internalEvalTest(evalCase),
evalCase.timeout,
);
}
export async function internalEvalTest(evalCase: EvalCase) {
const maxRetries = 3;
let attempt = 0;
while (attempt <= maxRetries) {
const rig = new TestRig();
const { logDir, sanitizedName } = await prepareLogDir(evalCase.name);
const activityLogFile = path.join(logDir, `${sanitizedName}.jsonl`);
const logFile = path.join(logDir, `${sanitizedName}.log`);
let isSuccess = false;
try {
rig.setup(evalCase.name, evalCase.params);
// Symlink node modules to reduce the amount of time needed to
// bootstrap test projects.
symlinkNodeModules(rig.testDir || '');
if (evalCase.files) {
const acknowledgedAgents: Record<string, Record<string, string>> = {};
const projectRoot = fs.realpathSync(rig.testDir!);
for (const [filePath, content] of Object.entries(evalCase.files)) {
const fullPath = path.join(rig.testDir!, filePath);
fs.mkdirSync(path.dirname(fullPath), { recursive: true });
fs.writeFileSync(fullPath, content);
// If it's an agent file, calculate hash for acknowledgement
if (
filePath.startsWith('.gemini/agents/') &&
filePath.endsWith('.md')
) {
const hash = crypto
.createHash('sha256')
.update(content)
.digest('hex');
try {
const agentDefs = await parseAgentMarkdown(fullPath, content);
if (agentDefs.length > 0) {
const agentName = agentDefs[0].name;
if (!acknowledgedAgents[projectRoot]) {
acknowledgedAgents[projectRoot] = {};
}
acknowledgedAgents[projectRoot][agentName] = hash;
}
} catch (error) {
console.warn(
`Failed to parse agent for test acknowledgement: ${filePath}`,
error,
);
}
}
}
// Write acknowledged_agents.json to the home directory
if (Object.keys(acknowledgedAgents).length > 0) {
const ackPath = path.join(
rig.homeDir!,
'.gemini',
'acknowledgments',
'agents.json',
);
fs.mkdirSync(path.dirname(ackPath), { recursive: true });
fs.writeFileSync(
ackPath,
JSON.stringify(acknowledgedAgents, null, 2),
);
}
const execOptions = { cwd: rig.testDir!, stdio: 'inherit' as const };
execSync('git init', execOptions);
execSync('git config user.email "test@example.com"', execOptions);
execSync('git config user.name "Test User"', execOptions);
// Temporarily disable the interactive editor and git pager
// to avoid hanging the tests. It seems the the agent isn't
// consistently honoring the instructions to avoid interactive
// commands.
execSync('git config core.editor "true"', execOptions);
execSync('git config core.pager "cat"', execOptions);
execSync('git config commit.gpgsign false', execOptions);
execSync('git add .', execOptions);
execSync('git commit --allow-empty -m "Initial commit"', execOptions);
await setupTestFiles(rig, evalCase.files);
}
symlinkNodeModules(rig.testDir || '');
// If messages are provided, write a session file so --resume can load it.
let sessionId: string | undefined;
if (evalCase.messages) {
@@ -188,6 +135,37 @@ export function evalTest(policy: EvalPolicy, evalCase: EvalCase) {
await evalCase.assert(rig, result);
isSuccess = true;
return; // Success! Exit the retry loop.
} catch (error: unknown) {
const errorMessage =
error instanceof Error ? error.message : String(error);
const errorCode = getApiErrorCode(errorMessage);
if (errorCode) {
const status = attempt < maxRetries ? 'RETRY' : 'SKIP';
logReliabilityEvent(
evalCase.name,
attempt,
status,
errorCode,
errorMessage,
);
if (attempt < maxRetries) {
attempt++;
console.warn(
`[Eval] Attempt ${attempt} failed with ${errorCode} Error. Retrying...`,
);
continue; // Retry
}
console.warn(
`[Eval] '${evalCase.name}' failed after ${maxRetries} retries due to persistent API errors. Skipping failure to avoid blocking PR.`,
);
return; // Gracefully exit without failing the test
}
throw error; // Real failure
} finally {
if (isSuccess) {
await fs.promises.unlink(activityLogFile).catch((err) => {
@@ -206,9 +184,131 @@ export function evalTest(policy: EvalPolicy, evalCase: EvalCase) {
);
await rig.cleanup();
}
}
}
function getApiErrorCode(message: string): '500' | '503' | undefined {
if (
message.includes('status: UNAVAILABLE') ||
message.includes('code: 503') ||
message.includes('Service Unavailable')
) {
return '503';
}
if (
message.includes('status: INTERNAL') ||
message.includes('code: 500') ||
message.includes('Internal error encountered')
) {
return '500';
}
return undefined;
}
/**
* Log reliability event for later harvesting.
*
* Note: Uses synchronous file I/O to ensure the log is persisted even if the
* test process is abruptly terminated by a timeout or CI crash. Performance
* impact is negligible compared to long-running evaluation tests.
*/
function logReliabilityEvent(
testName: string,
attempt: number,
status: 'RETRY' | 'SKIP',
errorCode: '500' | '503',
errorMessage: string,
) {
const reliabilityLog = {
timestamp: new Date().toISOString(),
testName,
model: process.env.GEMINI_MODEL || 'unknown',
attempt,
status,
errorCode,
error: errorMessage,
};
runEval(policy, evalCase.name, fn, evalCase.timeout);
try {
const relDir = path.resolve(process.cwd(), 'evals/logs');
fs.mkdirSync(relDir, { recursive: true });
fs.appendFileSync(
path.join(relDir, 'api-reliability.jsonl'),
JSON.stringify(reliabilityLog) + '\n',
);
} catch (logError) {
console.error('Failed to write reliability log:', logError);
}
}
/**
* Helper to setup test files and git repository.
*
* Note: While this is an async function (due to parseAgentMarkdown), it
* intentionally uses synchronous filesystem and child_process operations
* for simplicity and to ensure sequential environment preparation.
*/
async function setupTestFiles(rig: TestRig, files: Record<string, string>) {
const acknowledgedAgents: Record<string, Record<string, string>> = {};
const projectRoot = fs.realpathSync(rig.testDir!);
for (const [filePath, content] of Object.entries(files)) {
if (filePath.includes('..') || path.isAbsolute(filePath)) {
throw new Error(`Invalid file path in test case: ${filePath}`);
}
const fullPath = path.join(projectRoot, filePath);
if (!fullPath.startsWith(projectRoot)) {
throw new Error(`Path traversal detected: ${filePath}`);
}
fs.mkdirSync(path.dirname(fullPath), { recursive: true });
fs.writeFileSync(fullPath, content);
if (filePath.startsWith('.gemini/agents/') && filePath.endsWith('.md')) {
const hash = crypto.createHash('sha256').update(content).digest('hex');
try {
const agentDefs = await parseAgentMarkdown(fullPath, content);
if (agentDefs.length > 0) {
const agentName = agentDefs[0].name;
if (!acknowledgedAgents[projectRoot]) {
acknowledgedAgents[projectRoot] = {};
}
acknowledgedAgents[projectRoot][agentName] = hash;
}
} catch (error) {
console.warn(
`Failed to parse agent for test acknowledgement: ${filePath}`,
error,
);
}
}
}
if (Object.keys(acknowledgedAgents).length > 0) {
const ackPath = path.join(
rig.homeDir!,
'.gemini',
'acknowledgments',
'agents.json',
);
fs.mkdirSync(path.dirname(ackPath), { recursive: true });
fs.writeFileSync(ackPath, JSON.stringify(acknowledgedAgents, null, 2));
}
const execOptions = { cwd: rig.testDir!, stdio: 'inherit' as const };
execSync('git init --initial-branch=main', execOptions);
execSync('git config user.email "test@example.com"', execOptions);
execSync('git config user.name "Test User"', execOptions);
// Temporarily disable the interactive editor and git pager
// to avoid hanging the tests. It seems the the agent isn't
// consistently honoring the instructions to avoid interactive
// commands.
execSync('git config core.editor "true"', execOptions);
execSync('git config core.pager "cat"', execOptions);
execSync('git config commit.gpgsign false', execOptions);
execSync('git add .', execOptions);
execSync('git commit --allow-empty -m "Initial commit"', execOptions);
}
/**
-4
View File
@@ -16,10 +16,6 @@ export default defineConfig({
},
test: {
testTimeout: 300000, // 5 minutes
// Retry in CI but not nightly to avoid blocking on API error.
retry: process.env['VITEST_RETRY']
? parseInt(process.env['VITEST_RETRY'], 10)
: 3,
reporters: ['default', 'json'],
outputFile: {
json: 'evals/logs/report.json',
@@ -691,6 +691,40 @@ describe('useSlashCompletion', () => {
});
unmount();
});
it('should rank primary name prefix matches higher than alias prefix matches', async () => {
const slashCommands = [
createTestCommand({
name: 'footer',
altNames: ['statusline'],
description: 'Configure footer',
}),
createTestCommand({
name: 'stats',
altNames: ['usage'],
description: 'Check stats',
}),
];
const { result, unmount } = await renderHook(() =>
useTestHarnessForSlashCompletion(
true,
'/stat',
slashCommands,
mockCommandContext,
),
);
await resolveMatch();
await waitFor(() => {
// 'stats' should be first because 'stat' is a prefix match on its name
// while 'footer' only matches 'stat' via its alias 'statusline'
expect(result.current.suggestions[0].label).toBe('stats');
expect(result.current.suggestions[1].label).toBe('footer');
});
unmount();
});
});
describe('Sub-Commands', () => {
@@ -272,13 +272,45 @@ function useCommandSuggestions(
}
if (!signal.aborted) {
// Sort potentialSuggestions so that exact match (by name or altName) comes first
// Sort potentialSuggestions so that exact name/prefix match comes first,
// prioritizing primary name over altNames.
const lowerPartial = partial.toLowerCase();
const sortedSuggestions = [...potentialSuggestions].sort((a, b) => {
const aIsExact = matchesCommand(a, partial);
const bIsExact = matchesCommand(b, partial);
if (aIsExact && !bIsExact) return -1;
if (!aIsExact && bIsExact) return 1;
return 0;
// 1. Exact name match
const aNameExact = a.name.toLowerCase() === lowerPartial;
const bNameExact = b.name.toLowerCase() === lowerPartial;
if (aNameExact && !bNameExact) return -1;
if (!aNameExact && bNameExact) return 1;
// 2. Exact altName match
const aAltExact =
a.altNames?.some((alt) => alt.toLowerCase() === lowerPartial) ||
false;
const bAltExact =
b.altNames?.some((alt) => alt.toLowerCase() === lowerPartial) ||
false;
if (aAltExact && !bAltExact) return -1;
if (!aAltExact && bAltExact) return 1;
// 3. Prefix name match
const aNamePrefix = a.name.toLowerCase().startsWith(lowerPartial);
const bNamePrefix = b.name.toLowerCase().startsWith(lowerPartial);
if (aNamePrefix && !bNamePrefix) return -1;
if (!aNamePrefix && bNamePrefix) return 1;
// 4. Prefix altName match
const aAltPrefix =
a.altNames?.some((alt) =>
alt.toLowerCase().startsWith(lowerPartial),
) || false;
const bAltPrefix =
b.altNames?.some((alt) =>
alt.toLowerCase().startsWith(lowerPartial),
) || false;
if (aAltPrefix && !bAltPrefix) return -1;
if (!aAltPrefix && bAltPrefix) return 1;
return 0; // Maintain FZF score order for other matches
});
const finalSuggestions = sortedSuggestions.map((cmd) => {
@@ -200,7 +200,10 @@ describe('LocalSubagentInvocation', () => {
}),
);
expect(mockExecutorInstance.run).toHaveBeenCalledWith(params, signal);
expect(mockExecutorInstance.run).toHaveBeenCalledWith(
params,
expect.any(AbortSignal),
);
expect(result.llmContent).toEqual([
{
@@ -495,10 +498,50 @@ describe('LocalSubagentInvocation', () => {
expect(mockExecutorInstance.run).toHaveBeenCalledWith(
params,
controller.signal,
expect.any(AbortSignal),
);
});
it('external AbortSignal propagates through session to abort executor', async () => {
// This test validates the wiring: caller signal → abortListener → session.abort()
// → internal AbortController → executor's signal aborts.
// The previous test mocks the executor to always reject, so it would pass even
// if the abort wiring was broken. This test requires the signal to actually fire.
const controller = new AbortController();
let executorSignal: AbortSignal | undefined;
mockExecutorInstance.run.mockImplementation(
(_p: unknown, sig: AbortSignal) => {
executorSignal = sig;
return new Promise((_resolve, reject) => {
sig.addEventListener('abort', () => {
const err = new Error('AbortError');
err.name = 'AbortError';
reject(err);
});
});
},
);
const executePromise = invocation.execute(
controller.signal,
updateOutput,
);
// Wait for the executor to start so executorSignal is populated
await vi.waitFor(() => {
expect(executorSignal).toBeDefined();
});
expect(executorSignal!.aborted).toBe(false);
// Fire the external signal — this must propagate through to abort the executor
controller.abort();
await expect(executePromise).rejects.toThrow();
expect(executorSignal!.aborted).toBe(true);
});
it('should throw an error and bubble cancellation when execution returns ABORTED', async () => {
const mockOutput = {
result: 'Cancelled by user',
+184 -158
View File
@@ -5,7 +5,6 @@
*/
import { type AgentLoopContext } from '../config/agent-loop-context.js';
import { LocalAgentExecutor } from './local-executor.js';
import {
BaseToolInvocation,
type ToolResult,
@@ -30,6 +29,8 @@ import {
sanitizeToolArgs,
sanitizeErrorMessage,
} from '../utils/agent-sanitization-utils.js';
import { LocalSubagentSession } from './local-subagent-protocol.js';
import type { AgentEvent } from '../agent/types.js';
const INPUT_PREVIEW_MAX_LENGTH = 50;
const DESCRIPTION_MAX_LENGTH = 200;
@@ -39,11 +40,10 @@ const MAX_RECENT_ACTIVITY = 3;
* Represents a validated, executable instance of a subagent tool.
*
* This class orchestrates the execution of a defined agent by:
* 1. Initializing the {@link LocalAgentExecutor}.
* 2. Running the agent's execution loop.
* 3. Bridging the agent's streaming activity (e.g., thoughts) to the tool's
* live output stream.
* 4. Formatting the final result into a {@link ToolResult}.
* 1. Using {@link LocalSubagentSession} as the execution engine.
* 2. Bridging the agent's streaming activity (e.g., thoughts) to the tool's
* live output stream via the session's rawActivityCallback.
* 3. Formatting the final result into a {@link ToolResult}.
*/
export class LocalSubagentInvocation extends BaseToolInvocation<
AgentInputs,
@@ -54,6 +54,9 @@ export class LocalSubagentInvocation extends BaseToolInvocation<
* @param context The agent loop context.
* @param params The validated input parameters for the agent.
* @param messageBus Message bus for policy enforcement.
* @param _toolName Optional override for the tool name.
* @param _toolDisplayName Optional override for the tool display name.
* @param _onAgentEvent Optional callback for parent session observability.
*/
constructor(
private readonly definition: LocalAgentDefinition,
@@ -62,6 +65,7 @@ export class LocalSubagentInvocation extends BaseToolInvocation<
messageBus: MessageBus,
_toolName?: string,
_toolDisplayName?: string,
private readonly _onAgentEvent?: (event: AgentEvent) => void,
) {
super(
params,
@@ -101,9 +105,170 @@ export class LocalSubagentInvocation extends BaseToolInvocation<
): Promise<ToolResult> {
let recentActivity: SubagentActivityItem[] = [];
// Raw SubagentActivityEvent handler — preserves all existing progress display logic.
// Passed as rawActivityCallback to LocalSubagentSession so the protocol can call it
// before translating to AgentEvents.
const onActivity = (activity: SubagentActivityEvent): void => {
if (!updateOutput) return;
let updated = false;
switch (activity.type) {
case 'THOUGHT_CHUNK': {
const text = String(activity.data['text']);
const lastItem = recentActivity[recentActivity.length - 1];
if (
lastItem &&
lastItem.type === 'thought' &&
lastItem.status === 'running'
) {
lastItem.content = sanitizeThoughtContent(text);
} else {
recentActivity.push({
id: randomUUID(),
type: 'thought',
content: sanitizeThoughtContent(text),
status: 'running',
});
}
updated = true;
break;
}
case 'TOOL_CALL_START': {
const name = String(activity.data['name']);
const displayName = activity.data['displayName']
? sanitizeErrorMessage(String(activity.data['displayName']))
: undefined;
const description = activity.data['description']
? sanitizeErrorMessage(String(activity.data['description']))
: undefined;
const args = JSON.stringify(sanitizeToolArgs(activity.data['args']));
recentActivity.push({
id: randomUUID(),
type: 'tool_call',
content: name,
displayName,
description,
args,
status: 'running',
});
updated = true;
break;
}
case 'TOOL_CALL_END': {
const name = String(activity.data['name']);
const data = activity.data['data'];
const isError = isToolActivityError(data);
for (let i = recentActivity.length - 1; i >= 0; i--) {
if (
recentActivity[i].type === 'tool_call' &&
recentActivity[i].content === name &&
recentActivity[i].status === 'running'
) {
recentActivity[i].status = isError ? 'error' : 'completed';
updated = true;
break;
}
}
break;
}
case 'ERROR': {
const error = String(activity.data['error']);
const errorType = activity.data['errorType'];
const sanitizedError = sanitizeErrorMessage(error);
const isCancellation =
errorType === SubagentActivityErrorType.CANCELLED ||
error === SUBAGENT_CANCELLED_ERROR_MESSAGE;
const isRejection =
errorType === SubagentActivityErrorType.REJECTED ||
error.startsWith(SUBAGENT_REJECTED_ERROR_PREFIX);
const toolName = activity.data['name']
? String(activity.data['name'])
: undefined;
if (toolName && (isCancellation || isRejection)) {
for (let i = recentActivity.length - 1; i >= 0; i--) {
if (
recentActivity[i].type === 'tool_call' &&
recentActivity[i].content === toolName &&
recentActivity[i].status === 'running'
) {
recentActivity[i].status = 'cancelled';
updated = true;
break;
}
}
} else if (toolName) {
// Mark non-rejection/non-cancellation errors as 'error'
for (let i = recentActivity.length - 1; i >= 0; i--) {
if (
recentActivity[i].type === 'tool_call' &&
recentActivity[i].content === toolName &&
recentActivity[i].status === 'running'
) {
recentActivity[i].status = 'error';
updated = true;
break;
}
}
}
recentActivity.push({
id: randomUUID(),
type: 'thought',
content:
isCancellation || isRejection
? sanitizedError
: `Error: ${sanitizedError}`,
status: isCancellation || isRejection ? 'cancelled' : 'error',
});
updated = true;
break;
}
default:
break;
}
if (updated) {
// Keep only the last N items
if (recentActivity.length > MAX_RECENT_ACTIVITY) {
recentActivity = recentActivity.slice(-MAX_RECENT_ACTIVITY);
}
const progress: SubagentProgress = {
isSubagentProgress: true,
agentName: this.definition.name,
recentActivity: [...recentActivity], // Copy to avoid mutation issues
state: 'running',
};
updateOutput(progress);
}
};
// Create session with the raw activity callback for rich progress display
const session = new LocalSubagentSession(
this.definition,
this.context,
this.messageBus,
onActivity,
);
// Subscribe for parent session observability (future use)
let unsubscribeParent: (() => void) | undefined;
if (this._onAgentEvent) {
unsubscribeParent = session.subscribe(this._onAgentEvent);
}
// Wire external abort signal to session abort
const abortListener = () => void session.abort();
signal.addEventListener('abort', abortListener, { once: true });
try {
if (updateOutput) {
// Send initial state
const initialProgress: SubagentProgress = {
isSubagentProgress: true,
agentName: this.definition.name,
@@ -113,158 +278,16 @@ export class LocalSubagentInvocation extends BaseToolInvocation<
updateOutput(initialProgress);
}
// Create an activity callback to bridge the executor's events to the
// tool's streaming output.
const onActivity = (activity: SubagentActivityEvent): void => {
if (!updateOutput) return;
// Buffer non-query params, then send query as message to start execution
const query = String(this.params['query'] ?? '');
const otherParams = { ...this.params } as Record<string, unknown>;
delete otherParams['query'];
if (Object.keys(otherParams).length > 0) {
await session.send({ update: { config: otherParams } });
}
await session.send({ message: [{ type: 'text', text: query }] });
let updated = false;
switch (activity.type) {
case 'THOUGHT_CHUNK': {
const text = String(activity.data['text']);
const lastItem = recentActivity[recentActivity.length - 1];
if (
lastItem &&
lastItem.type === 'thought' &&
lastItem.status === 'running'
) {
lastItem.content = sanitizeThoughtContent(text);
} else {
recentActivity.push({
id: randomUUID(),
type: 'thought',
content: sanitizeThoughtContent(text),
status: 'running',
});
}
updated = true;
break;
}
case 'TOOL_CALL_START': {
const name = String(activity.data['name']);
const displayName = activity.data['displayName']
? sanitizeErrorMessage(String(activity.data['displayName']))
: undefined;
const description = activity.data['description']
? sanitizeErrorMessage(String(activity.data['description']))
: undefined;
const args = JSON.stringify(
sanitizeToolArgs(activity.data['args']),
);
recentActivity.push({
id: randomUUID(),
type: 'tool_call',
content: name,
displayName,
description,
args,
status: 'running',
});
updated = true;
break;
}
case 'TOOL_CALL_END': {
const name = String(activity.data['name']);
const data = activity.data['data'];
const isError = isToolActivityError(data);
for (let i = recentActivity.length - 1; i >= 0; i--) {
if (
recentActivity[i].type === 'tool_call' &&
recentActivity[i].content === name &&
recentActivity[i].status === 'running'
) {
recentActivity[i].status = isError ? 'error' : 'completed';
updated = true;
break;
}
}
break;
}
case 'ERROR': {
const error = String(activity.data['error']);
const errorType = activity.data['errorType'];
const sanitizedError = sanitizeErrorMessage(error);
const isCancellation =
errorType === SubagentActivityErrorType.CANCELLED ||
error === SUBAGENT_CANCELLED_ERROR_MESSAGE;
const isRejection =
errorType === SubagentActivityErrorType.REJECTED ||
error.startsWith(SUBAGENT_REJECTED_ERROR_PREFIX);
const toolName = activity.data['name']
? String(activity.data['name'])
: undefined;
if (toolName && (isCancellation || isRejection)) {
for (let i = recentActivity.length - 1; i >= 0; i--) {
if (
recentActivity[i].type === 'tool_call' &&
recentActivity[i].content === toolName &&
recentActivity[i].status === 'running'
) {
recentActivity[i].status = 'cancelled';
updated = true;
break;
}
}
} else if (toolName) {
// Mark non-rejection/non-cancellation errors as 'error'
for (let i = recentActivity.length - 1; i >= 0; i--) {
if (
recentActivity[i].type === 'tool_call' &&
recentActivity[i].content === toolName &&
recentActivity[i].status === 'running'
) {
recentActivity[i].status = 'error';
updated = true;
break;
}
}
}
recentActivity.push({
id: randomUUID(),
type: 'thought',
content:
isCancellation || isRejection
? sanitizedError
: `Error: ${sanitizedError}`,
status: isCancellation || isRejection ? 'cancelled' : 'error',
});
updated = true;
break;
}
default:
break;
}
if (updated) {
// Keep only the last N items
if (recentActivity.length > MAX_RECENT_ACTIVITY) {
recentActivity = recentActivity.slice(-MAX_RECENT_ACTIVITY);
}
const progress: SubagentProgress = {
isSubagentProgress: true,
agentName: this.definition.name,
recentActivity: [...recentActivity], // Copy to avoid mutation issues
state: 'running',
};
updateOutput(progress);
}
};
const executor = await LocalAgentExecutor.create(
this.definition,
this.context,
onActivity,
);
const output = await executor.run(this.params, signal);
const output = await session.getResult();
if (output.terminate_reason === AgentTerminateMode.ABORTED) {
const progress: SubagentProgress = {
@@ -359,6 +382,9 @@ ${output.result}`;
// We omit the 'error' property so that the UI renders our rich returnDisplay
// instead of the raw error message. The llmContent still informs the agent of the failure.
};
} finally {
signal.removeEventListener('abort', abortListener);
unsubscribeParent?.();
}
}
}
@@ -0,0 +1,776 @@
/**
* @license
* Copyright 2026 Google LLC
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { LocalSubagentSession } from './local-subagent-protocol.js';
import { LocalAgentExecutor } from './local-executor.js';
import {
AgentTerminateMode,
type LocalAgentDefinition,
type SubagentActivityEvent,
} from './types.js';
import { makeFakeConfig } from '../test-utils/config.js';
import { createMockMessageBus } from '../test-utils/mock-message-bus.js';
import type { AgentLoopContext } from '../config/agent-loop-context.js';
import type { AgentEvent } from '../agent/types.js';
import type { MessageBus } from '../confirmation-bus/message-bus.js';
import type { z } from 'zod';
import type { Mocked } from 'vitest';
vi.mock('./local-executor.js');
const MockLocalAgentExecutor = vi.mocked(LocalAgentExecutor);
// Captures the onActivity callback passed to LocalAgentExecutor.create().
// Set via create.mockImplementation in beforeEach to avoid mock.calls index fragility.
let capturedOnActivity: ((activity: SubagentActivityEvent) => void) | undefined;
const testDefinition: LocalAgentDefinition = {
kind: 'local',
name: 'TestProtocolAgent',
description: 'A test agent for protocol tests.',
inputConfig: {
inputSchema: {
type: 'object',
properties: {
task: { type: 'string' },
priority: { type: 'number' },
},
},
},
modelConfig: { model: 'test', generateContentConfig: {} },
runConfig: { maxTimeMinutes: 1 },
promptConfig: { systemPrompt: 'test' },
};
const GOAL_OUTPUT = {
result: 'Analysis complete.',
terminate_reason: AgentTerminateMode.GOAL,
};
describe('LocalSubagentSession (protocol)', () => {
let mockContext: AgentLoopContext;
let mockMessageBus: MessageBus;
let mockExecutorInstance: Mocked<LocalAgentExecutor<z.ZodUnknown>>;
beforeEach(() => {
vi.clearAllMocks();
capturedOnActivity = undefined;
mockContext = makeFakeConfig() as unknown as AgentLoopContext;
mockMessageBus = createMockMessageBus();
mockExecutorInstance = {
run: vi.fn().mockResolvedValue(GOAL_OUTPUT),
definition: testDefinition,
} as unknown as Mocked<LocalAgentExecutor<z.ZodUnknown>>;
// Use mockImplementation (not mockResolvedValue) so we can capture onActivity.
MockLocalAgentExecutor.create.mockImplementation(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
async (_def: any, _ctx: any, onActivity: any) => {
capturedOnActivity = onActivity;
return mockExecutorInstance as unknown as LocalAgentExecutor<z.ZodTypeAny>;
},
);
});
afterEach(() => {
vi.restoreAllMocks();
});
// ---------------------------------------------------------------------------
// Lifecycle events
// ---------------------------------------------------------------------------
describe('lifecycle events', () => {
it('emits agent_start then agent_end(completed) for a GOAL run', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await session.send({ message: [{ type: 'text', text: 'query' }] });
await session.getResult();
expect(events[0].type).toBe('agent_start');
expect(events[events.length - 1].type).toBe('agent_end');
const endEvent = events[events.length - 1];
if (endEvent.type === 'agent_end') {
expect(endEvent.reason).toBe('completed');
}
});
it('emits agent_start exactly once even if ensureAgentStart called twice internally', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await session.send({ message: [{ type: 'text', text: 'query' }] });
await session.getResult();
const startEvents = events.filter((e) => e.type === 'agent_start');
expect(startEvents).toHaveLength(1);
});
it('emits agent_end exactly once on error path', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
mockExecutorInstance.run.mockRejectedValue(new Error('executor failed'));
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await session.send({ message: [{ type: 'text', text: 'query' }] });
await expect(session.getResult()).rejects.toThrow('executor failed');
const endEvents = events.filter((e) => e.type === 'agent_end');
expect(endEvents).toHaveLength(1);
});
it('all events share the same streamId', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await session.send({ message: [{ type: 'text', text: 'query' }] });
await session.getResult();
const streamIds = new Set(events.map((e) => e.streamId));
expect(streamIds.size).toBe(1);
});
});
// ---------------------------------------------------------------------------
// Config buffering (update + message pattern)
// ---------------------------------------------------------------------------
describe('config buffering', () => {
it('merges buffered config with message query', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
await session.send({
update: { config: { task: 'analyze', priority: 5 } },
});
await session.send({ message: [{ type: 'text', text: 'my query' }] });
await session.getResult();
expect(mockExecutorInstance.run).toHaveBeenCalledWith(
{ task: 'analyze', priority: 5, query: 'my query' },
expect.any(AbortSignal),
);
});
it('omits query key when message text is empty', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
await session.send({ update: { config: { task: 'no-query-task' } } });
await session.send({ message: [{ type: 'text', text: '' }] });
await session.getResult();
const callArgs = mockExecutorInstance.run.mock.calls[0][0];
expect(callArgs).not.toHaveProperty('query');
expect(callArgs).toEqual({ task: 'no-query-task' });
});
it('sends only query when no prior update', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
await session.send({ message: [{ type: 'text', text: 'just a query' }] });
await session.getResult();
expect(mockExecutorInstance.run).toHaveBeenCalledWith(
{ query: 'just a query' },
expect.any(AbortSignal),
);
});
it('multiple update calls are merged', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
await session.send({ update: { config: { field1: 'a' } } });
await session.send({ update: { config: { field2: 'b' } } });
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult();
expect(mockExecutorInstance.run).toHaveBeenCalledWith(
{ field1: 'a', field2: 'b', query: 'q' },
expect.any(AbortSignal),
);
});
it('update returns streamId: null; message returns a streamId', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
const updateResult = await session.send({ update: { config: {} } });
expect(updateResult.streamId).toBeNull();
const messageResult = await session.send({
message: [{ type: 'text', text: 'q' }],
});
expect(messageResult.streamId).not.toBeNull();
expect(typeof messageResult.streamId).toBe('string');
// Await completion to prevent dangling execution affecting subsequent tests
await session.getResult();
});
});
// ---------------------------------------------------------------------------
// Activity translation
// ---------------------------------------------------------------------------
describe('activity translation', () => {
function makeSession() {
const activityEvents: SubagentActivityEvent[] = [];
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
return { session, activityEvents };
}
async function runWithActivities(
session: LocalSubagentSession,
activities: SubagentActivityEvent[],
) {
mockExecutorInstance.run.mockImplementation(async () => {
// capturedOnActivity is set by the create.mockImplementation in beforeEach
// and updated whenever create() is called. By the time run() is called,
// capturedOnActivity holds the onActivity closure for the most-recently
// created executor — which is the one associated with this session.
for (const act of activities) {
capturedOnActivity?.(act);
}
return GOAL_OUTPUT;
});
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult();
return events;
}
it('THOUGHT_CHUNK → message event with thought content', async () => {
const { session } = makeSession();
const events = await runWithActivities(session, [
{
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
type: 'THOUGHT_CHUNK',
data: { text: 'I am thinking...' },
},
]);
const msgEvent = events.find((e) => e.type === 'message');
expect(msgEvent).toBeDefined();
if (msgEvent?.type === 'message') {
expect(msgEvent.role).toBe('agent');
expect(msgEvent.content).toContainEqual({
type: 'thought',
thought: 'I am thinking...',
});
}
});
it('TOOL_CALL_START → tool_request event', async () => {
const { session } = makeSession();
const events = await runWithActivities(session, [
{
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
type: 'TOOL_CALL_START',
data: { callId: 'call-123', name: 'read_file', args: { path: '/a' } },
},
]);
const reqEvent = events.find((e) => e.type === 'tool_request');
expect(reqEvent).toBeDefined();
if (reqEvent?.type === 'tool_request') {
expect(reqEvent.requestId).toBe('call-123');
expect(reqEvent.name).toBe('read_file');
expect(reqEvent.args).toEqual({ path: '/a' });
}
});
it('TOOL_CALL_END → tool_response event', async () => {
const { session } = makeSession();
const events = await runWithActivities(session, [
{
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
type: 'TOOL_CALL_END',
data: { id: 'call-123', name: 'read_file', output: 'file contents' },
},
]);
const respEvent = events.find((e) => e.type === 'tool_response');
expect(respEvent).toBeDefined();
if (respEvent?.type === 'tool_response') {
expect(respEvent.requestId).toBe('call-123');
expect(respEvent.name).toBe('read_file');
expect(respEvent.content).toContainEqual({
type: 'text',
text: 'file contents',
});
}
});
it('ERROR activity → error event with INTERNAL status, fatal: false', async () => {
const { session } = makeSession();
const events = await runWithActivities(session, [
{
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
type: 'ERROR',
data: { error: 'something went wrong' },
},
]);
const errEvent = events.find((e) => e.type === 'error');
expect(errEvent).toBeDefined();
if (errEvent?.type === 'error') {
expect(errEvent.status).toBe('INTERNAL');
expect(errEvent.message).toBe('something went wrong');
expect(errEvent.fatal).toBe(false);
}
});
it('unknown activity type → no events emitted', async () => {
const { session } = makeSession();
const events = await runWithActivities(session, [
{
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
// eslint-disable-next-line @typescript-eslint/no-explicit-any
type: 'UNKNOWN_TYPE' as any,
data: {},
},
]);
// Only agent_start and agent_end should be present
const nonLifecycle = events.filter(
(e) => e.type !== 'agent_start' && e.type !== 'agent_end',
);
expect(nonLifecycle).toHaveLength(0);
});
it('TOOL_CALL_START with non-object args defaults to {}', async () => {
const { session } = makeSession();
const events = await runWithActivities(session, [
{
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
type: 'TOOL_CALL_START',
data: { callId: 'x', name: 'tool', args: null },
},
]);
const reqEvent = events.find((e) => e.type === 'tool_request');
if (reqEvent?.type === 'tool_request') {
expect(reqEvent.args).toEqual({});
}
});
});
// ---------------------------------------------------------------------------
// getResult() promise
// ---------------------------------------------------------------------------
describe('getResult()', () => {
it('resolves with OutputObject on GOAL termination', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
await session.send({ message: [{ type: 'text', text: 'q' }] });
const output = await session.getResult();
expect(output.result).toBe('Analysis complete.');
expect(output.terminate_reason).toBe(AgentTerminateMode.GOAL);
});
it('rejects when executor throws', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
mockExecutorInstance.run.mockRejectedValue(new Error('executor error'));
await session.send({ message: [{ type: 'text', text: 'q' }] });
await expect(session.getResult()).rejects.toThrow('executor error');
});
});
// ---------------------------------------------------------------------------
// rawActivityCallback
// ---------------------------------------------------------------------------
describe('rawActivityCallback', () => {
it('receives raw SubagentActivityEvent before AgentEvent translation', async () => {
const rawActivities: SubagentActivityEvent[] = [];
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
(activity) => rawActivities.push(activity),
);
const thoughtActivity: SubagentActivityEvent = {
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
type: 'THOUGHT_CHUNK',
data: { text: 'raw thought' },
};
mockExecutorInstance.run.mockImplementation(async () => {
const onActivity = MockLocalAgentExecutor.create.mock.calls[0]?.[2];
onActivity?.(thoughtActivity);
return GOAL_OUTPUT;
});
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult();
expect(rawActivities).toHaveLength(1);
expect(rawActivities[0]).toBe(thoughtActivity);
});
it('is called before AgentEvent translation (raw arrives first)', async () => {
const callOrder: string[] = [];
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
() => callOrder.push('raw'),
);
session.subscribe((e) => {
if (e.type === 'message') callOrder.push('translated');
});
mockExecutorInstance.run.mockImplementation(async () => {
const onActivity = MockLocalAgentExecutor.create.mock.calls[0]?.[2];
onActivity?.({
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
type: 'THOUGHT_CHUNK',
data: { text: 'thought' },
});
return GOAL_OUTPUT;
});
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult();
expect(callOrder).toEqual(['raw', 'translated']);
});
it('is optional — no callback causes no error', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
// no rawActivityCallback
);
await session.send({ message: [{ type: 'text', text: 'q' }] });
await expect(session.getResult()).resolves.toBeDefined();
});
});
// ---------------------------------------------------------------------------
// Subscription
// ---------------------------------------------------------------------------
describe('subscription', () => {
it('unsubscribe stops event delivery', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
const received: AgentEvent[] = [];
const unsub = session.subscribe((e) => received.push(e));
unsub();
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult();
expect(received).toHaveLength(0);
});
it('multiple subscribers all receive events', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
const received1: AgentEvent[] = [];
const received2: AgentEvent[] = [];
session.subscribe((e) => received1.push(e));
session.subscribe((e) => received2.push(e));
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult();
expect(received1.length).toBeGreaterThan(0);
expect(received1).toEqual(received2);
});
it('events array accumulates all emitted events', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult();
expect(session.events.length).toBeGreaterThanOrEqual(2); // at least agent_start + agent_end
expect(session.events[0].type).toBe('agent_start');
});
});
// ---------------------------------------------------------------------------
// Terminate mode mapping
// ---------------------------------------------------------------------------
describe('terminate mode → StreamEndReason mapping', () => {
const cases: Array<[AgentTerminateMode, string]> = [
[AgentTerminateMode.GOAL, 'completed'],
[AgentTerminateMode.TIMEOUT, 'max_time'],
[AgentTerminateMode.MAX_TURNS, 'max_turns'],
[AgentTerminateMode.ABORTED, 'aborted'],
[AgentTerminateMode.ERROR, 'failed'],
[AgentTerminateMode.ERROR_NO_COMPLETE_TASK_CALL, 'failed'],
];
for (const [terminateMode, expectedReason] of cases) {
it(`${terminateMode} → agent_end(reason:'${expectedReason}')`, async () => {
mockExecutorInstance.run.mockResolvedValue({
result: 'done',
terminate_reason: terminateMode,
});
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult().catch(() => {
// ABORTED results in rejection — catch to let test complete
});
const endEvent = events.find((e) => e.type === 'agent_end');
expect(endEvent).toBeDefined();
if (endEvent?.type === 'agent_end') {
expect(endEvent.reason).toBe(expectedReason);
}
});
}
});
// ---------------------------------------------------------------------------
// Abort
// ---------------------------------------------------------------------------
describe('abort()', () => {
it('abort() causes agent_end(reason:aborted)', async () => {
// Make run() wait until aborted
let abortSignal: AbortSignal | undefined;
mockExecutorInstance.run.mockImplementation(
(_params: unknown, signal: AbortSignal) => {
abortSignal = signal;
return new Promise((_resolve, reject) => {
signal.addEventListener('abort', () => {
const err = new Error('AbortError');
err.name = 'AbortError';
reject(err);
});
});
},
);
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
void session.send({ message: [{ type: 'text', text: 'q' }] });
// Wait for executor to be created and run started
await vi.waitFor(() => {
expect(abortSignal).toBeDefined();
});
await session.abort();
await expect(session.getResult()).rejects.toThrow();
const endEvent = events.find((e) => e.type === 'agent_end');
expect(endEvent).toBeDefined();
if (endEvent?.type === 'agent_end') {
expect(endEvent.reason).toBe('aborted');
}
});
});
// ---------------------------------------------------------------------------
// Full event sequence
// ---------------------------------------------------------------------------
describe('full event sequence', () => {
it('emits agent_start → message(thought) → tool_request → tool_response → agent_end in order', async () => {
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
mockExecutorInstance.run.mockImplementation(async () => {
const onActivity = MockLocalAgentExecutor.create.mock.calls[0]?.[2];
onActivity?.({
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
type: 'THOUGHT_CHUNK',
data: { text: 'thinking' },
});
onActivity?.({
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
type: 'TOOL_CALL_START',
data: { callId: 'c1', name: 'tool', args: {} },
});
onActivity?.({
isSubagentActivityEvent: true,
agentName: 'TestProtocolAgent',
type: 'TOOL_CALL_END',
data: { id: 'c1', name: 'tool', output: 'result' },
});
return GOAL_OUTPUT;
});
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await session.send({ message: [{ type: 'text', text: 'go' }] });
await session.getResult();
const types = events.map((e) => e.type);
expect(types).toEqual([
'agent_start',
'message',
'tool_request',
'tool_response',
'agent_end',
]);
});
});
// ---------------------------------------------------------------------------
// Concurrent send() guard
// ---------------------------------------------------------------------------
describe('concurrent send() guard', () => {
it('calling send() while a stream is active throws', async () => {
let abortSignal: AbortSignal | undefined;
mockExecutorInstance.run.mockImplementation(
(_params: unknown, signal: AbortSignal) => {
abortSignal = signal;
return new Promise((_resolve, reject) => {
// Reject when aborted so getResult() can settle during cleanup
signal.addEventListener('abort', () => {
const err = new Error('AbortError');
err.name = 'AbortError';
reject(err);
});
});
},
);
const session = new LocalSubagentSession(
testDefinition,
mockContext,
mockMessageBus,
);
void session.send({ message: [{ type: 'text', text: 'first' }] });
// Wait for execution to start
await vi.waitFor(() => {
expect(abortSignal).toBeDefined();
});
// Second send() while first stream is active must throw
await expect(
session.send({ message: [{ type: 'text', text: 'second' }] }),
).rejects.toThrow('cannot be called while a stream is active');
// Clean up: abort to unblock the hanging executor
await session.abort();
await session.getResult().catch(() => {});
});
});
});
@@ -0,0 +1,427 @@
/**
* @license
* Copyright 2026 Google LLC
* SPDX-License-Identifier: Apache-2.0
*/
/**
* @fileoverview LocalSubagentProtocol — wraps LocalAgentExecutor behind the
* AgentProtocol interface, translating SubagentActivityEvent callbacks into
* AgentEvents and exposing the executor result via getResult().
*
* Pattern mirrors LegacyAgentProtocol, but the loop body runs
* LocalAgentExecutor instead of GeminiClient.sendMessageStream().
*/
import { randomUUID } from 'node:crypto';
import type { AgentLoopContext } from '../config/agent-loop-context.js';
import { AgentSession } from '../agent/agent-session.js';
import type {
AgentProtocol,
AgentSend,
AgentEvent,
StreamEndReason,
Unsubscribe,
ContentPart,
} from '../agent/types.js';
import { LocalAgentExecutor } from './local-executor.js';
import {
AgentTerminateMode,
type LocalAgentDefinition,
type AgentInputs,
type OutputObject,
type SubagentActivityEvent,
} from './types.js';
import type { MessageBus } from '../confirmation-bus/message-bus.js';
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function isAbortLikeError(err: unknown): boolean {
return err instanceof Error && err.name === 'AbortError';
}
function mapTerminateMode(mode: AgentTerminateMode): StreamEndReason {
switch (mode) {
case AgentTerminateMode.GOAL:
return 'completed';
case AgentTerminateMode.TIMEOUT:
return 'max_time';
case AgentTerminateMode.MAX_TURNS:
return 'max_turns';
case AgentTerminateMode.ABORTED:
return 'aborted';
case AgentTerminateMode.ERROR:
case AgentTerminateMode.ERROR_NO_COMPLETE_TASK_CALL:
default:
return 'failed';
}
}
// ---------------------------------------------------------------------------
// LocalSubagentProtocol
// ---------------------------------------------------------------------------
class LocalSubagentProtocol implements AgentProtocol {
private _events: AgentEvent[] = [];
private _subscribers = new Set<(event: AgentEvent) => void>();
private _streamId: string = randomUUID();
private _eventCounter = 0;
private _agentStartEmitted = false;
private _agentEndEmitted = false;
private _activeStreamId: string | undefined;
private _abortController = new AbortController();
// Result promise wiring
private _resultResolve!: (output: OutputObject) => void;
private _resultReject!: (err: unknown) => void;
private readonly _resultPromise: Promise<OutputObject>;
// Buffered config from send({update})
private _bufferedConfig: Record<string, unknown> = {};
constructor(
private readonly definition: LocalAgentDefinition,
private readonly context: AgentLoopContext,
_messageBus: MessageBus,
private readonly _rawActivityCallback?: (
activity: SubagentActivityEvent,
) => void,
) {
this._resultPromise = new Promise<OutputObject>((resolve, reject) => {
this._resultResolve = resolve;
this._resultReject = reject;
});
}
// ---------------------------------------------------------------------------
// AgentProtocol interface
// ---------------------------------------------------------------------------
get events(): readonly AgentEvent[] {
return this._events;
}
subscribe(callback: (event: AgentEvent) => void): Unsubscribe {
this._subscribers.add(callback);
return () => {
this._subscribers.delete(callback);
};
}
async send(payload: AgentSend): Promise<{ streamId: string | null }> {
if ('update' in payload && payload.update) {
// Buffer config for use when message send arrives
if (payload.update.config) {
this._bufferedConfig = {
...this._bufferedConfig,
...payload.update.config,
};
}
return { streamId: null };
}
if ('message' in payload && payload.message) {
if (this._activeStreamId) {
throw new Error(
'LocalSubagentProtocol.send() cannot be called while a stream is active.',
);
}
// Extract query text from the message ContentParts
const queryText = payload.message
.filter((p): p is ContentPart & { type: 'text' } => p.type === 'text')
.map((p) => p.text)
.join('');
// Only include 'query' in params when the message text is non-empty,
// so that callers that pass all fields via update.config are not affected.
const params: AgentInputs = {
...this._bufferedConfig,
...(queryText.length > 0 ? { query: queryText } : {}),
};
this._beginNewStream();
const streamId = this._streamId;
// Schedule execution in a macrotask so send() resolves before agent_start
setTimeout(() => {
void this._runExecutionInBackground(params);
}, 0);
return { streamId };
}
// action and elicitations are not supported
return { streamId: null };
}
async abort(): Promise<void> {
this._abortController.abort();
}
// ---------------------------------------------------------------------------
// Protocol-specific: result access
// ---------------------------------------------------------------------------
/**
* Resolves when the executor completes, with the raw OutputObject.
* Used by LocalSubagentInvocation to build the ToolResult.
*/
getResult(): Promise<OutputObject> {
return this._resultPromise;
}
// ---------------------------------------------------------------------------
// Core: execution
// ---------------------------------------------------------------------------
private _beginNewStream(): void {
this._streamId = randomUUID();
this._eventCounter = 0;
this._abortController = new AbortController();
this._agentStartEmitted = false;
this._agentEndEmitted = false;
this._activeStreamId = this._streamId;
}
private async _runExecutionInBackground(params: AgentInputs): Promise<void> {
this._ensureAgentStart();
try {
await this._runExecution(params);
} catch (err: unknown) {
if (this._abortController.signal.aborted || isAbortLikeError(err)) {
this._ensureAgentEnd('aborted');
this._resultReject(err);
} else {
this._emitErrorAndAgentEnd(err);
this._resultReject(err);
}
this._clearActiveStream();
}
}
private async _runExecution(params: AgentInputs): Promise<void> {
const signal = this._abortController.signal;
const onActivity = (activity: SubagentActivityEvent): void => {
// Forward raw activity to invocation-level callback (for rich SubagentProgress display)
this._rawActivityCallback?.(activity);
this._emit(this._translateActivity(activity));
};
const executor = await LocalAgentExecutor.create(
this.definition,
this.context,
onActivity,
);
const output = await executor.run(params, signal);
if (
output.terminate_reason === AgentTerminateMode.ABORTED ||
signal.aborted
) {
this._finishStream('aborted');
} else {
this._finishStream(mapTerminateMode(output.terminate_reason));
}
this._resultResolve(output);
}
// ---------------------------------------------------------------------------
// Activity → AgentEvent translation
// ---------------------------------------------------------------------------
private _translateActivity(activity: SubagentActivityEvent): AgentEvent[] {
switch (activity.type) {
case 'THOUGHT_CHUNK': {
const rawText = activity.data['text'];
const text = String(rawText ?? '');
return [
this._makeEvent('message', {
role: 'agent',
content: [{ type: 'thought', thought: text }],
}),
];
}
case 'TOOL_CALL_START': {
const rawCallId = activity.data['callId'];
const callId = String(rawCallId ?? randomUUID());
const rawName = activity.data['name'];
const name = String(rawName ?? 'unknown');
const rawArgs = activity.data['args'];
const args: Record<string, unknown> =
rawArgs !== null &&
typeof rawArgs === 'object' &&
!Array.isArray(rawArgs)
? // eslint-disable-next-line @typescript-eslint/no-unsafe-type-assertion
(rawArgs as Record<string, unknown>)
: {};
return [
this._makeEvent('tool_request', {
requestId: callId,
name,
args,
}),
];
}
case 'TOOL_CALL_END': {
const rawId = activity.data['id'];
const requestId = String(rawId ?? randomUUID());
const rawName = activity.data['name'];
const name = String(rawName ?? 'unknown');
const rawOutput = activity.data['output'];
const output = String(rawOutput ?? '');
return [
this._makeEvent('tool_response', {
requestId,
name,
content: [{ type: 'text', text: output }],
}),
];
}
case 'ERROR': {
const rawError = activity.data['error'];
const errorMsg = String(rawError ?? 'Unknown error');
return [
this._makeEvent('error', {
status: 'INTERNAL',
message: errorMsg,
fatal: false,
}),
];
}
default:
return [];
}
}
// ---------------------------------------------------------------------------
// Internal helpers (mirrors LegacyAgentProtocol)
// ---------------------------------------------------------------------------
private _emit(events: AgentEvent[]): void {
if (events.length === 0) return;
const subscribers = [...this._subscribers];
for (const event of events) {
this._events.push(event);
if (event.type === 'agent_end') {
this._agentEndEmitted = true;
}
for (const sub of subscribers) {
sub(event);
}
}
}
private _clearActiveStream(): void {
this._activeStreamId = undefined;
}
private _ensureAgentStart(): void {
if (!this._agentStartEmitted) {
this._agentStartEmitted = true;
this._emit([this._makeEvent('agent_start', {})]);
}
}
private _ensureAgentEnd(reason: StreamEndReason = 'completed'): void {
if (!this._agentEndEmitted && this._agentStartEmitted) {
this._emit([this._makeEvent('agent_end', { reason })]);
}
}
private _finishStream(
reason: StreamEndReason,
data?: Record<string, unknown>,
): void {
if (data && !this._agentEndEmitted) {
this._emit([this._makeEvent('agent_end', { reason, data })]);
this._agentEndEmitted = true;
} else {
this._ensureAgentEnd(reason);
}
this._clearActiveStream();
}
private _emitErrorAndAgentEnd(err: unknown): void {
const message = err instanceof Error ? err.message : String(err);
this._ensureAgentStart();
const meta: Record<string, unknown> = {};
if (err instanceof Error) {
meta['errorName'] = err.constructor.name;
if ('exitCode' in err && typeof err.exitCode === 'number') {
meta['exitCode'] = err.exitCode;
}
if ('code' in err) {
meta['code'] = err.code;
}
}
this._emit([
this._makeEvent('error', {
status: 'INTERNAL',
message,
fatal: true,
...(Object.keys(meta).length > 0 ? { _meta: meta } : {}),
}),
]);
this._ensureAgentEnd('failed');
}
private _nextEventFields() {
return {
id: `${this._streamId}-${this._eventCounter++}`,
timestamp: new Date().toISOString(),
streamId: this._streamId,
};
}
private _makeEvent<T extends AgentEvent['type']>(
type: T,
payload: Omit<AgentEvent<T>, 'id' | 'timestamp' | 'streamId' | 'type'>,
): AgentEvent {
// eslint-disable-next-line @typescript-eslint/no-unsafe-type-assertion
return {
...this._nextEventFields(),
type,
...payload,
} as AgentEvent;
}
}
// ---------------------------------------------------------------------------
// Public export
// ---------------------------------------------------------------------------
export class LocalSubagentSession extends AgentSession {
private readonly _localProtocol: LocalSubagentProtocol;
constructor(
definition: LocalAgentDefinition,
context: AgentLoopContext,
messageBus: MessageBus,
rawActivityCallback?: (activity: SubagentActivityEvent) => void,
) {
const protocol = new LocalSubagentProtocol(
definition,
context,
messageBus,
rawActivityCallback,
);
super(protocol);
this._localProtocol = protocol;
}
/**
* Returns the raw executor OutputObject once execution completes.
* Used by LocalSubagentInvocation to build the ToolResult.
*/
getResult(): Promise<OutputObject> {
return this._localProtocol.getResult();
}
}
+51 -148
View File
@@ -9,6 +9,7 @@ import {
type ToolConfirmationOutcome,
type ToolResult,
type ToolCallConfirmationDetails,
type ToolLiveOutput,
} from '../tools/tools.js';
import {
DEFAULT_QUERY_STRING,
@@ -16,44 +17,23 @@ import {
type RemoteAgentDefinition,
type AgentInputs,
type SubagentProgress,
getAgentCardLoadOptions,
getRemoteAgentTargetUrl,
} from './types.js';
import { type AgentLoopContext } from '../config/agent-loop-context.js';
import type { MessageBus } from '../confirmation-bus/message-bus.js';
import type {
A2AClientManager,
SendMessageResult,
} from './a2a-client-manager.js';
import { extractIdsFromResponse, A2AResultReassembler } from './a2aUtils.js';
import type { AuthenticationHandler } from '@a2a-js/sdk/client';
import { debugLogger } from '../utils/debugLogger.js';
import type { AnsiOutput } from '../utils/terminalSerializer.js';
import { A2AAuthProviderFactory } from './auth-provider/factory.js';
import { A2AAgentError } from './a2a-errors.js';
import { RemoteSubagentSession } from './remote-subagent-protocol.js';
import type { AgentEvent } from '../agent/types.js';
/**
* A tool invocation that proxies to a remote A2A agent.
*
* This implementation bypasses the local `LocalAgentExecutor` loop and directly
* invokes the configured A2A tool.
* This implementation delegates execution to {@link RemoteSubagentSession},
* which wraps the A2A client streaming behind the AgentProtocol interface.
*/
export class RemoteAgentInvocation extends BaseToolInvocation<
RemoteAgentInputs,
ToolResult
> {
// Persist state across ephemeral invocation instances.
private static readonly sessionState = new Map<
string,
{ contextId?: string; taskId?: string }
>();
// State for the ongoing conversation with the remote agent
private contextId: string | undefined;
private taskId: string | undefined;
private readonly clientManager: A2AClientManager;
private authHandler: AuthenticationHandler | undefined;
constructor(
private readonly definition: RemoteAgentDefinition,
private readonly context: AgentLoopContext,
@@ -61,6 +41,7 @@ export class RemoteAgentInvocation extends BaseToolInvocation<
messageBus: MessageBus,
_toolName?: string,
_toolDisplayName?: string,
private readonly _onAgentEvent?: (event: AgentEvent) => void,
) {
const query = params['query'] ?? DEFAULT_QUERY_STRING;
if (typeof query !== 'string') {
@@ -75,43 +56,19 @@ export class RemoteAgentInvocation extends BaseToolInvocation<
_toolName ?? definition.name,
_toolDisplayName ?? definition.displayName,
);
const clientManager = this.context.config.getA2AClientManager();
if (!clientManager) {
// Validate that A2AClientManager is available at construction time
if (!this.context.config.getA2AClientManager()) {
throw new Error(
`Failed to initialize RemoteAgentInvocation for '${definition.name}': A2AClientManager is not available.`,
);
}
this.clientManager = clientManager;
}
getDescription(): string {
return `Calling remote agent ${this.definition.displayName ?? this.definition.name}`;
}
private async getAuthHandler(): Promise<AuthenticationHandler | undefined> {
if (this.authHandler) {
return this.authHandler;
}
if (this.definition.auth) {
const targetUrl = getRemoteAgentTargetUrl(this.definition);
const provider = await A2AAuthProviderFactory.create({
authConfig: this.definition.auth,
agentName: this.definition.name,
targetUrl,
agentCardUrl: this.definition.agentCardUrl,
});
if (!provider) {
throw new Error(
`Failed to create auth provider for agent '${this.definition.name}'`,
);
}
this.authHandler = provider;
}
return this.authHandler;
}
protected override async getConfirmationDetails(
_abortSignal: AbortSignal,
): Promise<ToolCallConfirmationDetails | false> {
@@ -128,13 +85,33 @@ export class RemoteAgentInvocation extends BaseToolInvocation<
async execute(
_signal: AbortSignal,
updateOutput?: (output: string | AnsiOutput | SubagentProgress) => void,
updateOutput?: (output: ToolLiveOutput) => void,
): Promise<ToolResult> {
// 1. Ensure the agent is loaded (cached by manager)
// We assume the user has provided an access token via some mechanism (TODO),
// or we rely on ADC.
const reassembler = new A2AResultReassembler();
const agentName = this.definition.displayName ?? this.definition.name;
const session = new RemoteSubagentSession(
this.definition,
this.context,
this.messageBus,
);
// Wire external abort signal to session abort
const abortListener = () => void session.abort();
_signal.addEventListener('abort', abortListener, { once: true });
// Subscribe for parent session observability (future use)
let unsubscribeParent: (() => void) | undefined;
if (this._onAgentEvent) {
unsubscribeParent = session.subscribe(this._onAgentEvent);
}
// Subscribe to message events for live SubagentProgress updates
const unsubscribeProgress = session.subscribe((event: AgentEvent) => {
if (event.type === 'message' && updateOutput) {
const currentProgress = session.getLatestProgress();
if (currentProgress) updateOutput(currentProgress);
}
});
try {
if (updateOutput) {
updateOutput({
@@ -152,97 +129,25 @@ export class RemoteAgentInvocation extends BaseToolInvocation<
});
}
const priorState = RemoteAgentInvocation.sessionState.get(
this.definition.name,
);
if (priorState) {
this.contextId = priorState.contextId;
this.taskId = priorState.taskId;
}
await session.send({
message: [{ type: 'text', text: this.params.query }],
});
const authHandler = await this.getAuthHandler();
if (!this.clientManager.getClient(this.definition.name)) {
await this.clientManager.loadAgent(
this.definition.name,
getAgentCardLoadOptions(this.definition),
authHandler,
);
}
const message = this.params.query;
const stream = this.clientManager.sendMessageStream(
this.definition.name,
message,
{
contextId: this.contextId,
taskId: this.taskId,
signal: _signal,
},
);
let finalResponse: SendMessageResult | undefined;
for await (const chunk of stream) {
if (_signal.aborted) {
throw new Error('Operation aborted');
}
finalResponse = chunk;
reassembler.update(chunk);
if (updateOutput) {
updateOutput({
isSubagentProgress: true,
agentName,
state: 'running',
recentActivity: reassembler.toActivityItems(),
result: reassembler.toString(),
});
}
const {
contextId: newContextId,
taskId: newTaskId,
clearTaskId,
} = extractIdsFromResponse(chunk);
if (newContextId) {
this.contextId = newContextId;
}
this.taskId = clearTaskId ? undefined : (newTaskId ?? this.taskId);
}
if (!finalResponse) {
throw new Error('No response from remote agent.');
}
const finalOutput = reassembler.toString();
debugLogger.debug(
`[RemoteAgent] Final response from ${this.definition.name}:\n${JSON.stringify(finalResponse, null, 2)}`,
);
const finalProgress: SubagentProgress = {
isSubagentProgress: true,
agentName,
state: 'completed',
result: finalOutput,
recentActivity: reassembler.toActivityItems(),
};
const result = await session.getResult();
// Emit final completed progress
if (updateOutput) {
updateOutput(finalProgress);
const finalProgress = session.getLatestProgress();
if (finalProgress) updateOutput(finalProgress);
}
return {
llmContent: [{ text: finalOutput }],
returnDisplay: finalProgress,
};
return result;
} catch (error: unknown) {
const partialOutput = reassembler.toString();
// Surface structured, user-friendly error messages.
const partialProgress = session.getLatestProgress();
const partialOutput =
typeof partialProgress?.result === 'string'
? partialProgress.result
: '';
const errorMessage = this.formatExecutionError(error);
const fullDisplay = partialOutput
? `${partialOutput}\n\n${errorMessage}`
@@ -253,7 +158,7 @@ export class RemoteAgentInvocation extends BaseToolInvocation<
agentName,
state: 'error',
result: fullDisplay,
recentActivity: reassembler.toActivityItems(),
recentActivity: partialProgress?.recentActivity ?? [],
};
if (updateOutput) {
@@ -265,11 +170,9 @@ export class RemoteAgentInvocation extends BaseToolInvocation<
returnDisplay: errorProgress,
};
} finally {
// Persist state even on partial failures or aborts to maintain conversational continuity.
RemoteAgentInvocation.sessionState.set(this.definition.name, {
contextId: this.contextId,
taskId: this.taskId,
});
_signal.removeEventListener('abort', abortListener);
unsubscribeProgress();
unsubscribeParent?.();
}
}
@@ -0,0 +1,776 @@
/**
* @license
* Copyright 2026 Google LLC
* SPDX-License-Identifier: Apache-2.0
*/
import {
describe,
it,
expect,
vi,
beforeEach,
afterEach,
type Mock,
} from 'vitest';
import { RemoteSubagentSession } from './remote-subagent-protocol.js';
import { A2AAuthProviderFactory } from './auth-provider/factory.js';
import type { RemoteAgentDefinition, SubagentProgress } from './types.js';
import { createMockMessageBus } from '../test-utils/mock-message-bus.js';
import type { AgentLoopContext } from '../config/agent-loop-context.js';
import type { AgentEvent } from '../agent/types.js';
import type { Config } from '../config/config.js';
import type { A2AAuthProvider } from './auth-provider/types.js';
// Mock A2AClientManager at module level
vi.mock('./a2a-client-manager.js', () => ({
A2AClientManager: vi.fn().mockImplementation(() => ({
getClient: vi.fn(),
loadAgent: vi.fn(),
sendMessageStream: vi.fn(),
})),
}));
// Mock A2AAuthProviderFactory
vi.mock('./auth-provider/factory.js', () => ({
A2AAuthProviderFactory: {
create: vi.fn(),
},
}));
const mockDefinition: RemoteAgentDefinition = {
name: 'test-remote-agent',
kind: 'remote',
agentCardUrl: 'http://test-agent/card',
displayName: 'Test Remote Agent',
description: 'A test remote agent',
inputConfig: {
inputSchema: { type: 'object' },
},
};
function makeChunk(text: string) {
return {
kind: 'message' as const,
messageId: `msg-${Math.random()}`,
role: 'agent' as const,
parts: [{ kind: 'text' as const, text }],
};
}
describe('RemoteSubagentSession (protocol)', () => {
let mockClientManager: {
getClient: Mock;
loadAgent: Mock;
sendMessageStream: Mock;
};
let mockContext: AgentLoopContext;
let mockMessageBus: ReturnType<typeof createMockMessageBus>;
beforeEach(() => {
vi.clearAllMocks();
// Static session state is not cleared between tests — each test uses a
// unique agent name to avoid cross-test contamination.
mockClientManager = {
getClient: vi.fn().mockReturnValue(undefined), // client not yet loaded
loadAgent: vi.fn().mockResolvedValue(undefined),
sendMessageStream: vi.fn(),
};
const mockConfig = {
getA2AClientManager: vi.fn().mockReturnValue(mockClientManager),
injectionService: {
getLatestInjectionIndex: vi.fn().mockReturnValue(0),
},
} as unknown as Config;
mockContext = { config: mockConfig } as unknown as AgentLoopContext;
mockMessageBus = createMockMessageBus();
// Default: sendMessageStream yields one chunk with "Hello"
mockClientManager.sendMessageStream.mockImplementation(async function* () {
yield makeChunk('Hello');
});
});
afterEach(() => {
vi.restoreAllMocks();
});
// Helper: run a session with the default or custom stream and collect events
async function runSession(
definition: RemoteAgentDefinition = mockDefinition,
query = 'test query',
) {
const session = new RemoteSubagentSession(
definition,
mockContext,
mockMessageBus,
);
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await session.send({ message: [{ type: 'text', text: query }] });
const result = await session.getResult();
return { session, events, result };
}
// ---------------------------------------------------------------------------
// Lifecycle events
// ---------------------------------------------------------------------------
describe('lifecycle events', () => {
it('emits agent_start then agent_end(completed) on success', async () => {
const { events } = await runSession();
const types = events.map((e) => e.type);
expect(types[0]).toBe('agent_start');
expect(types[types.length - 1]).toBe('agent_end');
const end = events[events.length - 1];
if (end.type === 'agent_end') {
expect(end.reason).toBe('completed');
}
});
it('emits agent_start exactly once', async () => {
const { events } = await runSession();
expect(events.filter((e) => e.type === 'agent_start')).toHaveLength(1);
});
it('emits agent_end exactly once on error path', async () => {
mockClientManager.sendMessageStream.mockReturnValue({
[Symbol.asyncIterator]() {
return {
async next(): Promise<IteratorResult<never>> {
throw new Error('stream error');
},
};
},
});
const session = new RemoteSubagentSession(
mockDefinition,
mockContext,
mockMessageBus,
);
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await session.send({ message: [{ type: 'text', text: 'q' }] });
await expect(session.getResult()).rejects.toThrow('stream error');
expect(events.filter((e) => e.type === 'agent_end')).toHaveLength(1);
});
it('all events share the same streamId', async () => {
const { events } = await runSession();
const streamIds = new Set(events.map((e) => e.streamId));
expect(streamIds.size).toBe(1);
});
it('message returns a non-null streamId; unsupported payload returns null', async () => {
const session = new RemoteSubagentSession(
mockDefinition,
mockContext,
mockMessageBus,
);
const updateResult = await session.send({
update: { config: { key: 'val' } },
});
expect(updateResult.streamId).toBeNull();
const messageResult = await session.send({
message: [{ type: 'text', text: 'q' }],
});
expect(messageResult.streamId).not.toBeNull();
// complete the session to avoid dangling execution
await session.getResult();
});
});
// ---------------------------------------------------------------------------
// Chunk → AgentEvent translation
// ---------------------------------------------------------------------------
describe('chunk → AgentEvent translation', () => {
it('each A2A chunk produces a message event with current accumulated text', async () => {
mockClientManager.sendMessageStream.mockImplementation(
async function* () {
yield makeChunk('Hello');
yield makeChunk(' world');
},
);
const { events } = await runSession();
const msgEvents = events.filter((e) => e.type === 'message');
expect(msgEvents.length).toBeGreaterThanOrEqual(1);
// Final message event should contain the accumulated text
const lastMsg = msgEvents[msgEvents.length - 1];
if (lastMsg?.type === 'message') {
const textContent = lastMsg.content.find((c) => c.type === 'text');
expect(textContent).toBeDefined();
if (textContent?.type === 'text') {
expect(textContent.text).toContain('Hello');
}
}
});
it('getLatestProgress() is updated per chunk with state running', async () => {
let capturedProgress: SubagentProgress | undefined;
mockClientManager.sendMessageStream.mockImplementation(
async function* () {
yield makeChunk('Partial');
},
);
const session = new RemoteSubagentSession(
mockDefinition,
mockContext,
mockMessageBus,
);
session.subscribe((e) => {
if (e.type === 'message') {
capturedProgress = session.getLatestProgress();
}
});
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult();
// During streaming, progress should be 'running'
expect(capturedProgress).toBeDefined();
// Note: by the time we check, progress may be 'completed'.
// During the message event, it was 'running'.
expect(capturedProgress?.isSubagentProgress).toBe(true);
expect(capturedProgress?.agentName).toBe('Test Remote Agent');
});
it('getLatestProgress() state is completed after getResult() resolves', async () => {
const { session } = await runSession();
const progress = session.getLatestProgress();
expect(progress?.state).toBe('completed');
expect(progress?.result).toBe('Hello');
});
});
// ---------------------------------------------------------------------------
// getResult() promise
// ---------------------------------------------------------------------------
describe('getResult()', () => {
it('resolves with ToolResult containing llmContent and SubagentProgress returnDisplay', async () => {
mockClientManager.sendMessageStream.mockImplementation(
async function* () {
yield makeChunk('Result text');
},
);
const { result } = await runSession();
expect(result.llmContent).toEqual([{ text: 'Result text' }]);
const display = result.returnDisplay as SubagentProgress;
expect(display.isSubagentProgress).toBe(true);
expect(display.state).toBe('completed');
expect(display.result).toBe('Result text');
expect(display.agentName).toBe('Test Remote Agent');
});
it('rejects when stream throws a non-A2A error', async () => {
mockClientManager.sendMessageStream.mockReturnValue({
[Symbol.asyncIterator]() {
return {
async next(): Promise<IteratorResult<never>> {
throw new Error('network failure');
},
};
},
});
const session = new RemoteSubagentSession(
mockDefinition,
mockContext,
mockMessageBus,
);
await session.send({ message: [{ type: 'text', text: 'q' }] });
await expect(session.getResult()).rejects.toThrow();
});
it('resolves even with empty stream (empty final output)', async () => {
mockClientManager.sendMessageStream.mockImplementation(
async function* () {
// yield nothing
},
);
const { result } = await runSession();
expect(result.llmContent).toEqual([{ text: '' }]);
});
});
// ---------------------------------------------------------------------------
// Session state persistence
// ---------------------------------------------------------------------------
describe('session state persistence', () => {
it('second call reuses contextId captured from first call', async () => {
const agentName = 'persistent-agent';
const persistDef: RemoteAgentDefinition = {
...mockDefinition,
name: agentName,
};
let callCount = 0;
mockClientManager.sendMessageStream.mockImplementation(async function* (
_name: string,
_query: string,
opts: { contextId?: string },
) {
callCount++;
if (callCount === 1) {
// First call: return a chunk that yields a contextId
yield {
kind: 'message' as const,
messageId: 'msg-1',
role: 'agent' as const,
contextId: 'ctx-from-server',
parts: [{ kind: 'text' as const, text: 'First response' }],
};
} else {
// Second call: caller should have passed the contextId
expect(opts.contextId).toBe('ctx-from-server');
yield makeChunk('Second response');
}
});
// First call
const session1 = new RemoteSubagentSession(
persistDef,
mockContext,
mockMessageBus,
);
await session1.send({ message: [{ type: 'text', text: 'first' }] });
await session1.getResult();
// Second call — different session but same agent name → should reuse contextId
const session2 = new RemoteSubagentSession(
persistDef,
mockContext,
mockMessageBus,
);
await session2.send({ message: [{ type: 'text', text: 'second' }] });
await session2.getResult();
expect(callCount).toBe(2);
});
it('different agent names have independent session state', async () => {
const def1: RemoteAgentDefinition = {
...mockDefinition,
name: 'agent-alpha',
};
const def2: RemoteAgentDefinition = {
...mockDefinition,
name: 'agent-beta',
};
const capturedContextIds: Array<string | undefined> = [];
mockClientManager.sendMessageStream.mockImplementation(async function* (
_name: string,
_query: string,
opts: { contextId?: string },
) {
capturedContextIds.push(opts.contextId);
yield {
kind: 'message' as const,
messageId: 'msg-1',
role: 'agent' as const,
contextId: `ctx-for-${_name}`,
parts: [{ kind: 'text' as const, text: 'ok' }],
};
});
const session1 = new RemoteSubagentSession(
def1,
mockContext,
mockMessageBus,
);
await session1.send({ message: [{ type: 'text', text: 'q' }] });
await session1.getResult();
const session2 = new RemoteSubagentSession(
def2,
mockContext,
mockMessageBus,
);
await session2.send({ message: [{ type: 'text', text: 'q' }] });
await session2.getResult();
// Both start with no contextId (different agents, different state entries)
expect(capturedContextIds[0]).toBeUndefined();
expect(capturedContextIds[1]).toBeUndefined();
});
it('taskId is cleared when a terminal-state task chunk is received', async () => {
// A task chunk with a terminal status sets clearTaskId=true, which
// should clear this.taskId so it is NOT passed on the next call.
const agentName = 'clearTaskId-agent';
const def: RemoteAgentDefinition = { ...mockDefinition, name: agentName };
let callCount = 0;
const capturedTaskIds: Array<string | undefined> = [];
mockClientManager.sendMessageStream.mockImplementation(async function* (
_n: string,
_q: string,
opts: { taskId?: string },
) {
callCount++;
capturedTaskIds.push(opts.taskId);
if (callCount === 1) {
// First call: yield a task chunk with taskId + terminal status → clearTaskId
yield {
kind: 'task' as const,
id: 'task-123',
contextId: 'ctx-1',
status: { state: 'completed' as const },
};
} else {
yield makeChunk('done');
}
});
const session1 = new RemoteSubagentSession(
def,
mockContext,
mockMessageBus,
);
await session1.send({ message: [{ type: 'text', text: 'first' }] });
await session1.getResult();
const session2 = new RemoteSubagentSession(
def,
mockContext,
mockMessageBus,
);
await session2.send({ message: [{ type: 'text', text: 'second' }] });
await session2.getResult();
expect(callCount).toBe(2);
// First call starts with no taskId
expect(capturedTaskIds[0]).toBeUndefined();
// Second call: taskId was cleared because terminal-state task chunk was received
expect(capturedTaskIds[1]).toBeUndefined();
});
});
// ---------------------------------------------------------------------------
// Auth setup
// ---------------------------------------------------------------------------
describe('auth setup', () => {
it('no auth → loadAgent called without auth handler', async () => {
await runSession();
expect(mockClientManager.loadAgent).toHaveBeenCalledWith(
'test-remote-agent',
{ type: 'url', url: 'http://test-agent/card' },
undefined,
);
});
it('definition.auth present → A2AAuthProviderFactory.create called', async () => {
const authDef: RemoteAgentDefinition = {
...mockDefinition,
name: 'auth-agent',
auth: {
type: 'http' as const,
scheme: 'Bearer' as const,
token: 'secret',
},
};
const mockProvider = {
type: 'http' as const,
headers: vi.fn().mockResolvedValue({ Authorization: 'Bearer secret' }),
shouldRetryWithHeaders: vi.fn(),
} as unknown as A2AAuthProvider;
(A2AAuthProviderFactory.create as Mock).mockResolvedValue(mockProvider);
await runSession(authDef, 'q');
expect(A2AAuthProviderFactory.create).toHaveBeenCalledWith(
expect.objectContaining({
agentName: 'auth-agent',
agentCardUrl: 'http://test-agent/card',
}),
);
expect(mockClientManager.loadAgent).toHaveBeenCalledWith(
'auth-agent',
expect.any(Object),
mockProvider,
);
});
it('auth factory returns undefined → throws error that rejects getResult()', async () => {
const authDef: RemoteAgentDefinition = {
...mockDefinition,
name: 'failing-auth-agent',
auth: {
type: 'http' as const,
scheme: 'Bearer' as const,
token: 'secret',
},
};
(A2AAuthProviderFactory.create as Mock).mockResolvedValue(undefined);
const session = new RemoteSubagentSession(
authDef,
mockContext,
mockMessageBus,
);
await session.send({ message: [{ type: 'text', text: 'q' }] });
await expect(session.getResult()).rejects.toThrow(
"Failed to create auth provider for agent 'failing-auth-agent'",
);
});
it('agent already loaded → loadAgent not called again', async () => {
// Return a client object (truthy) so getClient returns defined
mockClientManager.getClient.mockReturnValue({});
await runSession();
expect(mockClientManager.loadAgent).not.toHaveBeenCalled();
});
});
// ---------------------------------------------------------------------------
// Error handling
// ---------------------------------------------------------------------------
describe('error handling', () => {
it('stream error → error event + agent_end(failed)', async () => {
mockClientManager.sendMessageStream.mockReturnValue({
[Symbol.asyncIterator]() {
return {
async next(): Promise<IteratorResult<never>> {
throw new Error('network error');
},
};
},
});
const session = new RemoteSubagentSession(
mockDefinition,
mockContext,
mockMessageBus,
);
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await session.send({ message: [{ type: 'text', text: 'q' }] });
await expect(session.getResult()).rejects.toThrow();
const errEvent = events.find((e) => e.type === 'error');
expect(errEvent).toBeDefined();
const endEvent = events.find((e) => e.type === 'agent_end');
expect(endEvent).toBeDefined();
if (endEvent?.type === 'agent_end') {
expect(endEvent.reason).toBe('failed');
}
});
it('missing A2AClientManager → rejects getResult()', async () => {
const mockConfig = {
getA2AClientManager: vi.fn().mockReturnValue(undefined),
injectionService: {
getLatestInjectionIndex: vi.fn().mockReturnValue(0),
},
} as unknown as Config;
const noClientContext = {
config: mockConfig,
} as unknown as AgentLoopContext;
const session = new RemoteSubagentSession(
mockDefinition,
noClientContext,
mockMessageBus,
);
await session.send({ message: [{ type: 'text', text: 'q' }] });
await expect(session.getResult()).rejects.toThrow(
'A2AClientManager not available',
);
});
});
// ---------------------------------------------------------------------------
// Subscription
// ---------------------------------------------------------------------------
describe('subscription', () => {
it('unsubscribe stops event delivery', async () => {
const session = new RemoteSubagentSession(
mockDefinition,
mockContext,
mockMessageBus,
);
const received: AgentEvent[] = [];
const unsub = session.subscribe((e) => received.push(e));
unsub();
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult();
expect(received).toHaveLength(0);
});
it('multiple subscribers all receive events', async () => {
const session = new RemoteSubagentSession(
mockDefinition,
mockContext,
mockMessageBus,
);
const events1: AgentEvent[] = [];
const events2: AgentEvent[] = [];
session.subscribe((e) => events1.push(e));
session.subscribe((e) => events2.push(e));
await session.send({ message: [{ type: 'text', text: 'q' }] });
await session.getResult();
expect(events1.length).toBeGreaterThan(0);
expect(events1).toEqual(events2);
});
});
// ---------------------------------------------------------------------------
// Abort
// ---------------------------------------------------------------------------
describe('abort()', () => {
it('abort() causes agent_end(reason:aborted)', async () => {
let resolveChunk: (() => void) | undefined;
// Stream that blocks until we abort
mockClientManager.sendMessageStream.mockImplementation(
async function* () {
// Hang until aborted
await new Promise<void>((resolve) => {
resolveChunk = resolve;
});
yield makeChunk('Too late');
},
);
const session = new RemoteSubagentSession(
mockDefinition,
mockContext,
mockMessageBus,
);
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
void session.send({ message: [{ type: 'text', text: 'q' }] });
// Wait for agent_start to be emitted before aborting
await vi.waitFor(() => {
expect(events.some((e) => e.type === 'agent_start')).toBe(true);
});
await session.abort();
// Resolve the hanging chunk generator so it can check the signal
resolveChunk?.();
await expect(session.getResult()).rejects.toThrow();
const endEvent = events.find((e) => e.type === 'agent_end');
expect(endEvent).toBeDefined();
if (endEvent?.type === 'agent_end') {
expect(endEvent.reason).toBe('aborted');
}
});
});
// ---------------------------------------------------------------------------
// sendMessageStream call args
// ---------------------------------------------------------------------------
describe('sendMessageStream call arguments', () => {
it('passes the query string from the message payload', async () => {
await runSession(mockDefinition, 'my specific query');
expect(mockClientManager.sendMessageStream).toHaveBeenCalledWith(
'test-remote-agent',
'my specific query',
expect.objectContaining({ signal: expect.any(Object) }),
);
});
it('uses DEFAULT_QUERY_STRING when message text is empty', async () => {
const session = new RemoteSubagentSession(
mockDefinition,
mockContext,
mockMessageBus,
);
await session.send({ message: [{ type: 'text', text: '' }] });
await session.getResult();
// DEFAULT_QUERY_STRING = 'Get Started!'
expect(mockClientManager.sendMessageStream).toHaveBeenCalledWith(
'test-remote-agent',
'Get Started!',
expect.objectContaining({ signal: expect.any(Object) }),
);
});
});
// ---------------------------------------------------------------------------
// Concurrent send() guard
// ---------------------------------------------------------------------------
describe('concurrent send() guard', () => {
it('calling send() while a stream is active throws', async () => {
let resolveChunk!: () => void;
mockClientManager.sendMessageStream.mockImplementation(
async function* () {
// Block until test releases the chunk
await new Promise<void>((resolve) => {
resolveChunk = resolve;
});
yield makeChunk('late');
},
);
const session = new RemoteSubagentSession(
mockDefinition,
mockContext,
mockMessageBus,
);
void session.send({ message: [{ type: 'text', text: 'first' }] });
// Wait for the stream to actually start (agent_start emitted)
const events: AgentEvent[] = [];
session.subscribe((e) => events.push(e));
await vi.waitFor(() => {
expect(events.some((e) => e.type === 'agent_start')).toBe(true);
});
// Second send() while first stream is active must throw
await expect(
session.send({ message: [{ type: 'text', text: 'second' }] }),
).rejects.toThrow('cannot be called while a stream is active');
// Clean up: release the blocked generator so getResult() can settle
resolveChunk();
await session.getResult().catch(() => {});
});
});
});
@@ -0,0 +1,461 @@
/**
* @license
* Copyright 2026 Google LLC
* SPDX-License-Identifier: Apache-2.0
*/
/**
* @fileoverview RemoteSubagentProtocol — wraps A2A remote agent streaming
* behind the AgentProtocol interface.
*
* Pattern mirrors LocalSubagentProtocol and LegacyAgentProtocol, but the loop
* body drives A2AClientManager instead of LocalAgentExecutor.
*/
import { randomUUID } from 'node:crypto';
import type { AgentLoopContext } from '../config/agent-loop-context.js';
import { AgentSession } from '../agent/agent-session.js';
import type {
AgentProtocol,
AgentSend,
AgentEvent,
StreamEndReason,
Unsubscribe,
ContentPart,
} from '../agent/types.js';
import type { ToolResult } from '../tools/tools.js';
import {
DEFAULT_QUERY_STRING,
type RemoteAgentDefinition,
type SubagentProgress,
getRemoteAgentTargetUrl,
getAgentCardLoadOptions,
} from './types.js';
import { A2AResultReassembler, extractIdsFromResponse } from './a2aUtils.js';
import type { AuthenticationHandler } from '@a2a-js/sdk/client';
import { A2AAuthProviderFactory } from './auth-provider/factory.js';
import { A2AAgentError } from './a2a-errors.js';
import { debugLogger } from '../utils/debugLogger.js';
import type { MessageBus } from '../confirmation-bus/message-bus.js';
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function isAbortLikeError(err: unknown): boolean {
return err instanceof Error && err.name === 'AbortError';
}
// ---------------------------------------------------------------------------
// RemoteSubagentProtocol
// ---------------------------------------------------------------------------
class RemoteSubagentProtocol implements AgentProtocol {
private _events: AgentEvent[] = [];
private _subscribers = new Set<(event: AgentEvent) => void>();
private _streamId: string = randomUUID();
private _eventCounter = 0;
private _agentStartEmitted = false;
private _agentEndEmitted = false;
private _activeStreamId: string | undefined;
private _abortController = new AbortController();
// Session state persisted across sends (mirrors RemoteAgentInvocation)
private contextId: string | undefined;
private taskId: string | undefined;
private authHandler: AuthenticationHandler | undefined;
// Agent display name (for SubagentProgress construction)
private readonly _agentName: string;
// Latest SubagentProgress — updated per chunk, used for error recovery
private _latestProgress: SubagentProgress | undefined;
// Result promise wiring
private _resultResolve!: (result: ToolResult) => void;
private _resultReject!: (err: unknown) => void;
private readonly _resultPromise: Promise<ToolResult>;
constructor(
private readonly definition: RemoteAgentDefinition,
private readonly context: AgentLoopContext,
_messageBus: MessageBus,
) {
this._agentName = definition.displayName ?? definition.name;
this._resultPromise = new Promise<ToolResult>((resolve, reject) => {
this._resultResolve = resolve;
this._resultReject = reject;
});
// Restore persisted session state (mirrors static map in RemoteAgentInvocation)
const priorState = RemoteSubagentProtocol._sessionState.get(
definition.name,
);
if (priorState) {
this.contextId = priorState.contextId;
this.taskId = priorState.taskId;
}
}
// Per-agent session state, mirrors RemoteAgentInvocation.sessionState
private static readonly _sessionState = new Map<
string,
{ contextId?: string; taskId?: string }
>();
// ---------------------------------------------------------------------------
// AgentProtocol interface
// ---------------------------------------------------------------------------
get events(): readonly AgentEvent[] {
return this._events;
}
subscribe(callback: (event: AgentEvent) => void): Unsubscribe {
this._subscribers.add(callback);
return () => {
this._subscribers.delete(callback);
};
}
async send(payload: AgentSend): Promise<{ streamId: string | null }> {
if ('message' in payload && payload.message) {
if (this._activeStreamId) {
throw new Error(
'RemoteSubagentProtocol.send() cannot be called while a stream is active.',
);
}
const query =
payload.message
.filter((p): p is ContentPart & { type: 'text' } => p.type === 'text')
.map((p) => p.text)
.join('') || DEFAULT_QUERY_STRING;
this._beginNewStream();
const streamId = this._streamId;
setTimeout(() => {
void this._runStreamInBackground(query);
}, 0);
return { streamId };
}
// update/action/elicitations not used for remote agents
return { streamId: null };
}
async abort(): Promise<void> {
this._abortController.abort();
}
// ---------------------------------------------------------------------------
// Protocol-specific: result access
// ---------------------------------------------------------------------------
getResult(): Promise<ToolResult> {
return this._resultPromise;
}
getLatestProgress(): SubagentProgress | undefined {
return this._latestProgress;
}
// ---------------------------------------------------------------------------
// Core: A2A streaming
// ---------------------------------------------------------------------------
private _beginNewStream(): void {
this._streamId = randomUUID();
this._eventCounter = 0;
this._abortController = new AbortController();
this._agentStartEmitted = false;
this._agentEndEmitted = false;
this._activeStreamId = this._streamId;
}
private async _runStreamInBackground(query: string): Promise<void> {
this._ensureAgentStart();
try {
await this._runStream(query);
} catch (err: unknown) {
// Save state before rejecting so callers see updated contextId/taskId
// immediately after the returned promise settles.
this._saveSessionState();
if (this._abortController.signal.aborted || isAbortLikeError(err)) {
this._ensureAgentEnd('aborted');
this._resultReject(err);
} else {
this._emitErrorAndAgentEnd(err);
this._resultReject(err);
}
this._clearActiveStream();
}
}
private _saveSessionState(): void {
RemoteSubagentProtocol._sessionState.set(this.definition.name, {
contextId: this.contextId,
taskId: this.taskId,
});
}
private async _runStream(query: string): Promise<void> {
const clientManager = this.context.config.getA2AClientManager();
if (!clientManager) {
throw new Error(
`RemoteSubagentProtocol: A2AClientManager not available for '${this.definition.name}'.`,
);
}
const authHandler = await this._getAuthHandler();
if (!clientManager.getClient(this.definition.name)) {
await clientManager.loadAgent(
this.definition.name,
getAgentCardLoadOptions(this.definition),
authHandler,
);
}
const reassembler = new A2AResultReassembler();
let prevText = '';
const stream = clientManager.sendMessageStream(
this.definition.name,
query,
{
contextId: this.contextId,
taskId: this.taskId,
signal: this._abortController.signal,
},
);
for await (const chunk of stream) {
if (this._abortController.signal.aborted) {
this._saveSessionState();
this._finishStream('aborted');
this._resultReject(new Error('Operation aborted'));
this._clearActiveStream();
return;
}
reassembler.update(chunk);
const {
contextId: newContextId,
taskId: newTaskId,
clearTaskId,
} = extractIdsFromResponse(chunk);
if (newContextId) this.contextId = newContextId;
this.taskId = clearTaskId ? undefined : (newTaskId ?? this.taskId);
// Update latest progress snapshot (for invocation's error recovery)
this._latestProgress = {
isSubagentProgress: true,
agentName: this._agentName,
state: 'running',
recentActivity: reassembler.toActivityItems(),
result: reassembler.toString(),
};
// Emit delta as a message event
const currentText = reassembler.toString();
const delta = currentText.slice(prevText.length);
if (delta) {
this._emit([
this._makeEvent('message', {
role: 'agent',
content: [{ type: 'text', text: currentText }],
}),
]);
prevText = currentText;
}
}
const finalOutput = reassembler.toString();
debugLogger.debug(
`[RemoteSubagentProtocol] ${this.definition.name} finished, output length: ${finalOutput.length}`,
);
const finalProgress: SubagentProgress = {
isSubagentProgress: true,
agentName: this._agentName,
state: 'completed',
result: finalOutput,
recentActivity: reassembler.toActivityItems(),
};
this._latestProgress = finalProgress;
this._finishStream('completed');
// Save state before resolving so callers see updated contextId/taskId
// immediately after the returned promise settles.
this._saveSessionState();
this._resultResolve({
llmContent: [{ text: finalOutput }],
returnDisplay: finalProgress,
});
}
private async _getAuthHandler(): Promise<AuthenticationHandler | undefined> {
if (this.authHandler) return this.authHandler;
if (!this.definition.auth) return undefined;
const targetUrl = getRemoteAgentTargetUrl(this.definition);
const provider = await A2AAuthProviderFactory.create({
authConfig: this.definition.auth,
agentName: this.definition.name,
targetUrl,
agentCardUrl: this.definition.agentCardUrl,
});
if (!provider) {
throw new Error(
`Failed to create auth provider for agent '${this.definition.name}'`,
);
}
this.authHandler = provider;
return this.authHandler;
}
// ---------------------------------------------------------------------------
// Internal helpers
// ---------------------------------------------------------------------------
private _emit(events: AgentEvent[]): void {
if (events.length === 0) return;
const subscribers = [...this._subscribers];
for (const event of events) {
this._events.push(event);
if (event.type === 'agent_end') {
this._agentEndEmitted = true;
}
for (const sub of subscribers) {
sub(event);
}
}
}
private _clearActiveStream(): void {
this._activeStreamId = undefined;
}
private _ensureAgentStart(): void {
if (!this._agentStartEmitted) {
this._agentStartEmitted = true;
this._emit([this._makeEvent('agent_start', {})]);
}
}
private _ensureAgentEnd(reason: StreamEndReason = 'completed'): void {
if (!this._agentEndEmitted && this._agentStartEmitted) {
this._emit([this._makeEvent('agent_end', { reason })]);
}
}
private _finishStream(
reason: StreamEndReason,
data?: Record<string, unknown>,
): void {
if (data && !this._agentEndEmitted) {
this._emit([this._makeEvent('agent_end', { reason, data })]);
this._agentEndEmitted = true;
} else {
this._ensureAgentEnd(reason);
}
this._clearActiveStream();
}
private _emitErrorAndAgentEnd(err: unknown): void {
const message = this._formatError(err);
this._ensureAgentStart();
const meta: Record<string, unknown> = {};
if (err instanceof Error) {
meta['errorName'] = err.constructor.name;
}
this._emit([
this._makeEvent('error', {
status: 'INTERNAL',
message,
fatal: true,
...(Object.keys(meta).length > 0 ? { _meta: meta } : {}),
}),
]);
this._ensureAgentEnd('failed');
}
private _formatError(error: unknown): string {
if (error instanceof A2AAgentError) {
return error.userMessage;
}
return `Error calling remote agent: ${error instanceof Error ? error.message : String(error)}`;
}
private _nextEventFields() {
return {
id: `${this._streamId}-${this._eventCounter++}`,
timestamp: new Date().toISOString(),
streamId: this._streamId,
};
}
private _makeEvent<T extends AgentEvent['type']>(
type: T,
payload: Omit<AgentEvent<T>, 'id' | 'timestamp' | 'streamId' | 'type'>,
): AgentEvent {
// eslint-disable-next-line @typescript-eslint/no-unsafe-type-assertion
return {
...this._nextEventFields(),
type,
...payload,
} as AgentEvent;
}
}
// ---------------------------------------------------------------------------
// Public export
// ---------------------------------------------------------------------------
export class RemoteSubagentSession extends AgentSession {
private readonly _remoteProtocol: RemoteSubagentProtocol;
constructor(
definition: RemoteAgentDefinition,
context: AgentLoopContext,
messageBus: MessageBus,
) {
const protocol = new RemoteSubagentProtocol(
definition,
context,
messageBus,
);
super(protocol);
this._remoteProtocol = protocol;
}
/**
* Returns the ToolResult once the remote agent stream completes.
* Used by RemoteAgentInvocation to return the result.
*/
getResult(): Promise<ToolResult> {
return this._remoteProtocol.getResult();
}
/**
* Returns the most recent SubagentProgress snapshot, updated per streaming
* chunk. Useful for constructing error progress when getResult() rejects.
*/
getLatestProgress(): SubagentProgress | undefined {
return this._remoteProtocol.getLatestProgress();
}
/**
* Convenience: start execution with a query string.
* Equivalent to send({message: [{type:'text', text: query}]}).
*/
async startWithQuery(query: string): Promise<{ streamId: string | null }> {
return this.send({ message: [{ type: 'text', text: query }] });
}
}
@@ -139,6 +139,7 @@ describe('SubagentToolWrapper', () => {
mockMessageBus,
mockDefinition.name,
mockDefinition.displayName,
undefined, // onAgentEvent (not set when wrapper has no onAgentEvent)
);
});
@@ -164,6 +165,7 @@ describe('SubagentToolWrapper', () => {
specificMessageBus,
mockDefinition.name,
mockDefinition.displayName,
undefined, // onAgentEvent (not set when wrapper has no onAgentEvent)
);
});
@@ -18,6 +18,7 @@ import { RemoteAgentInvocation } from './remote-invocation.js';
import { BrowserAgentInvocation } from './browser/browserAgentInvocation.js';
import { BROWSER_AGENT_NAME } from './browser/browserAgentDefinition.js';
import type { MessageBus } from '../confirmation-bus/message-bus.js';
import type { AgentEvent } from '../agent/types.js';
/**
* A tool wrapper that dynamically exposes a subagent as a standard,
@@ -41,6 +42,7 @@ export class SubagentToolWrapper extends BaseDeclarativeTool<
private readonly definition: AgentDefinition,
private readonly context: AgentLoopContext,
messageBus: MessageBus,
private readonly onAgentEvent?: (event: AgentEvent) => void,
) {
super(
definition.name,
@@ -80,6 +82,7 @@ export class SubagentToolWrapper extends BaseDeclarativeTool<
effectiveMessageBus,
_toolName,
_toolDisplayName,
this.onAgentEvent,
);
}
@@ -101,6 +104,7 @@ export class SubagentToolWrapper extends BaseDeclarativeTool<
effectiveMessageBus,
_toolName,
_toolDisplayName,
this.onAgentEvent,
);
}
}
@@ -121,6 +121,7 @@ describe('SubAgentInvocation', () => {
testDefinition,
mockConfig,
mockMessageBus,
undefined, // onAgentEvent (not set on SubAgentInvocation)
);
});
@@ -165,6 +166,7 @@ describe('SubAgentInvocation', () => {
testRemoteDefinition,
mockConfig,
mockMessageBus,
undefined, // onAgentEvent (not set on SubAgentInvocation)
);
});
@@ -19,6 +19,7 @@ import { type AgentLoopContext } from '../config/agent-loop-context.js';
import type { MessageBus } from '../confirmation-bus/message-bus.js';
import type { AgentDefinition, AgentInputs } from './types.js';
import { SubagentToolWrapper } from './subagent-tool-wrapper.js';
import type { AgentEvent } from '../agent/types.js';
import { SchemaValidator } from '../utils/schemaValidator.js';
import { formatUserHintsForModel } from '../utils/fastAckHelper.js';
import { runInDevTraceSpan } from '../telemetry/trace.js';
@@ -33,6 +34,7 @@ export class SubagentTool extends BaseDeclarativeTool<AgentInputs, ToolResult> {
private readonly definition: AgentDefinition,
private readonly context: AgentLoopContext,
messageBus: MessageBus,
private readonly onAgentEvent?: (event: AgentEvent) => void,
) {
const inputSchema = definition.inputConfig.inputSchema;
@@ -116,6 +118,7 @@ export class SubagentTool extends BaseDeclarativeTool<AgentInputs, ToolResult> {
messageBus,
_toolName,
_toolDisplayName,
this.onAgentEvent,
);
}
}
@@ -130,6 +133,7 @@ class SubAgentInvocation extends BaseToolInvocation<AgentInputs, ToolResult> {
messageBus: MessageBus,
_toolName?: string,
_toolDisplayName?: string,
private readonly onAgentEvent?: (event: AgentEvent) => void,
) {
super(
params,
@@ -229,6 +233,7 @@ class SubAgentInvocation extends BaseToolInvocation<AgentInputs, ToolResult> {
definition,
this.context,
this.messageBus,
this.onAgentEvent,
);
return wrapper.build(agentArgs);
+2
View File
@@ -187,6 +187,8 @@ export * from './agents/agent-scheduler.js';
// Export agent session interface
export * from './agent/agent-session.js';
export * from './agent/legacy-agent-session.js';
export { LocalSubagentSession } from './agents/local-subagent-protocol.js';
export { RemoteSubagentSession } from './agents/remote-subagent-protocol.js';
export * from './agent/event-translator.js';
export * from './agent/content-utils.js';
// Agent event types — namespaced to avoid collisions with existing exports
@@ -6,7 +6,6 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { LinuxSandboxManager } from './LinuxSandboxManager.js';
import * as sandboxManager from '../../services/sandboxManager.js';
import type { SandboxRequest } from '../../services/sandboxManager.js';
import fs from 'node:fs';
@@ -18,14 +17,16 @@ vi.mock('node:fs', async () => {
// @ts-expect-error - Property 'default' does not exist on type 'typeof import("node:fs")'
...actual.default,
existsSync: vi.fn(() => true),
realpathSync: vi.fn((p: string | Buffer) => p.toString()),
realpathSync: vi.fn((p) => p.toString()),
statSync: vi.fn(() => ({ isDirectory: () => true }) as fs.Stats),
mkdirSync: vi.fn(),
openSync: vi.fn(),
closeSync: vi.fn(),
writeFileSync: vi.fn(),
},
existsSync: vi.fn(() => true),
realpathSync: vi.fn((p: string | Buffer) => p.toString()),
realpathSync: vi.fn((p) => p.toString()),
statSync: vi.fn(() => ({ isDirectory: () => true }) as fs.Stats),
mkdirSync: vi.fn(),
openSync: vi.fn(),
closeSync: vi.fn(),
@@ -48,8 +49,12 @@ describe('LinuxSandboxManager', () => {
vi.restoreAllMocks();
});
const getBwrapArgs = async (req: SandboxRequest) => {
const result = await manager.prepareCommand(req);
const getBwrapArgs = async (
req: SandboxRequest,
customManager?: LinuxSandboxManager,
) => {
const mgr = customManager || manager;
const result = await mgr.prepareCommand(req);
expect(result.program).toBe('sh');
expect(result.args[0]).toBe('-c');
expect(result.args[1]).toBe(
@@ -60,41 +65,6 @@ describe('LinuxSandboxManager', () => {
return result.args.slice(4);
};
/**
* Helper to verify only the dynamic, policy-based binds (e.g. allowedPaths, forbiddenPaths).
* It asserts that the base workspace and governance files are present exactly once,
* then strips them away, leaving only the dynamic binds for a focused, non-brittle assertion.
*/
const expectDynamicBinds = (
bwrapArgs: string[],
expectedDynamicBinds: string[],
) => {
const bindsIndex = bwrapArgs.indexOf('--seccomp');
const allBinds = bwrapArgs.slice(bwrapArgs.indexOf('--bind'), bindsIndex);
const baseBinds = [
'--bind',
workspace,
workspace,
'--ro-bind',
`${workspace}/.gitignore`,
`${workspace}/.gitignore`,
'--ro-bind',
`${workspace}/.geminiignore`,
`${workspace}/.geminiignore`,
'--ro-bind',
`${workspace}/.git`,
`${workspace}/.git`,
];
// Verify the base binds are present exactly at the beginning
expect(allBinds.slice(0, baseBinds.length)).toEqual(baseBinds);
// Extract the remaining dynamic binds
const dynamicBinds = allBinds.slice(baseBinds.length);
expect(dynamicBinds).toEqual(expectedDynamicBinds);
};
describe('prepareCommand', () => {
it('should correctly format the base command and args', async () => {
const bwrapArgs = await getBwrapArgs({
@@ -117,7 +87,7 @@ describe('LinuxSandboxManager', () => {
'/proc',
'--tmpfs',
'/tmp',
'--bind',
'--ro-bind-try',
workspace,
workspace,
'--ro-bind',
@@ -137,6 +107,73 @@ describe('LinuxSandboxManager', () => {
]);
});
it('binds workspace read-write when readonly is false', async () => {
const customManager = new LinuxSandboxManager({
workspace,
modeConfig: { readonly: false },
});
const bwrapArgs = await getBwrapArgs(
{
command: 'ls',
args: [],
cwd: workspace,
env: {},
},
customManager,
);
expect(bwrapArgs).toContain('--bind-try');
expect(bwrapArgs).toContain(workspace);
});
it('maps network permissions to --share-net', async () => {
const bwrapArgs = await getBwrapArgs({
command: 'curl',
args: [],
cwd: workspace,
env: {},
policy: { additionalPermissions: { network: true } },
});
expect(bwrapArgs).toContain('--share-net');
});
it('maps explicit write permissions to --bind-try', async () => {
const bwrapArgs = await getBwrapArgs({
command: 'touch',
args: [],
cwd: workspace,
env: {},
policy: {
additionalPermissions: {
fileSystem: { write: ['/home/user/workspace/out/dir'] },
},
},
});
const index = bwrapArgs.indexOf('--bind-try');
expect(index).not.toBe(-1);
expect(bwrapArgs[index + 1]).toBe('/home/user/workspace/out/dir');
});
it('rejects overrides in plan mode', async () => {
const customManager = new LinuxSandboxManager({
workspace,
modeConfig: { allowOverrides: false },
});
await expect(
customManager.prepareCommand({
command: 'ls',
args: [],
cwd: workspace,
env: {},
policy: { additionalPermissions: { network: true } },
}),
).rejects.toThrow(
/Cannot override readonly\/network\/filesystem restrictions in Plan mode/,
);
});
it('should correctly pass through the cwd to the resulting command', async () => {
const req: SandboxRequest = {
command: 'ls',
@@ -184,12 +221,7 @@ describe('LinuxSandboxManager', () => {
},
});
expect(bwrapArgs).toContain('--unshare-user');
expect(bwrapArgs).toContain('--unshare-ipc');
expect(bwrapArgs).toContain('--unshare-pid');
expect(bwrapArgs).toContain('--unshare-uts');
expect(bwrapArgs).toContain('--unshare-cgroup');
expect(bwrapArgs).not.toContain('--unshare-all');
expect(bwrapArgs).toContain('--share-net');
});
describe('governance files', () => {
@@ -252,15 +284,32 @@ describe('LinuxSandboxManager', () => {
},
});
// Verify the specific bindings were added correctly
expectDynamicBinds(bwrapArgs, [
expect(bwrapArgs).toContain('--bind-try');
expect(bwrapArgs[bwrapArgs.indexOf('/tmp/cache') - 1]).toBe(
'--bind-try',
'/tmp/cache',
'/tmp/cache',
);
expect(bwrapArgs[bwrapArgs.indexOf('/opt/tools') - 1]).toBe(
'--bind-try',
'/opt/tools',
'/opt/tools',
]);
);
});
it('should not grant read-write access to allowedPaths inside the workspace when readonly mode is active', async () => {
const manager = new LinuxSandboxManager({
workspace,
modeConfig: { readonly: true },
});
const result = await manager.prepareCommand({
command: 'ls',
args: [],
cwd: workspace,
env: {},
policy: {
allowedPaths: [workspace + '/subdirectory'],
},
});
const bwrapArgs = result.args;
const bindIndex = bwrapArgs.indexOf(workspace + '/subdirectory');
expect(bwrapArgs[bindIndex - 1]).toBe('--ro-bind-try');
});
it('should not bind the workspace twice even if it has a trailing slash in allowedPaths', async () => {
@@ -274,23 +323,20 @@ describe('LinuxSandboxManager', () => {
},
});
// Should only contain the primary workspace bind and governance files, not the second workspace bind with a trailing slash
expectDynamicBinds(bwrapArgs, []);
const binds = bwrapArgs.filter((a) => a === workspace);
expect(binds.length).toBe(2);
});
});
describe('forbiddenPaths', () => {
it('should parameterize forbidden paths and explicitly deny them', async () => {
vi.spyOn(fs.promises, 'stat').mockImplementation(async (p) => {
// Mock /tmp/cache as a directory, and /opt/secret.txt as a file
vi.mocked(fs.statSync).mockImplementation((p) => {
if (p.toString().includes('cache')) {
return { isDirectory: () => true } as fs.Stats;
}
return { isDirectory: () => false } as fs.Stats;
});
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(async (p) =>
p.toString(),
);
vi.mocked(fs.realpathSync).mockImplementation((p) => p.toString());
const bwrapArgs = await getBwrapArgs({
command: 'ls',
@@ -302,27 +348,22 @@ describe('LinuxSandboxManager', () => {
},
});
expectDynamicBinds(bwrapArgs, [
'--tmpfs',
'/tmp/cache',
'--remount-ro',
'/tmp/cache',
'--ro-bind-try',
'/dev/null',
'/opt/secret.txt',
]);
const cacheIndex = bwrapArgs.indexOf('/tmp/cache');
expect(bwrapArgs[cacheIndex - 1]).toBe('--tmpfs');
const secretIndex = bwrapArgs.indexOf('/opt/secret.txt');
expect(bwrapArgs[secretIndex - 2]).toBe('--ro-bind');
expect(bwrapArgs[secretIndex - 1]).toBe('/dev/null');
});
it('resolves forbidden symlink paths to their real paths', async () => {
vi.spyOn(fs.promises, 'stat').mockImplementation(
async () => ({ isDirectory: () => false }) as fs.Stats,
);
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(
async (p) => {
if (p === '/tmp/forbidden-symlink') return '/opt/real-target.txt';
return p.toString();
},
vi.mocked(fs.statSync).mockImplementation(
() => ({ isDirectory: () => false }) as fs.Stats,
);
vi.mocked(fs.realpathSync).mockImplementation((p) => {
if (p === '/tmp/forbidden-symlink') return '/opt/real-target.txt';
return p.toString();
});
const bwrapArgs = await getBwrapArgs({
command: 'ls',
@@ -334,24 +375,18 @@ describe('LinuxSandboxManager', () => {
},
});
// Should explicitly mask both the resolved path and the original symlink path
expectDynamicBinds(bwrapArgs, [
'--ro-bind-try',
'/dev/null',
'/opt/real-target.txt',
'--ro-bind-try',
'/dev/null',
'/tmp/forbidden-symlink',
]);
const secretIndex = bwrapArgs.indexOf('/opt/real-target.txt');
expect(bwrapArgs[secretIndex - 2]).toBe('--ro-bind');
expect(bwrapArgs[secretIndex - 1]).toBe('/dev/null');
});
it('explicitly denies non-existent forbidden paths to prevent creation', async () => {
const error = new Error('File not found') as NodeJS.ErrnoException;
error.code = 'ENOENT';
vi.spyOn(fs.promises, 'stat').mockRejectedValue(error);
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(async (p) =>
p.toString(),
);
vi.mocked(fs.statSync).mockImplementation(() => {
throw error;
});
vi.mocked(fs.realpathSync).mockImplementation((p) => p.toString());
const bwrapArgs = await getBwrapArgs({
command: 'ls',
@@ -363,23 +398,19 @@ describe('LinuxSandboxManager', () => {
},
});
expectDynamicBinds(bwrapArgs, [
'--symlink',
'/.forbidden',
'/tmp/not-here.txt',
]);
const idx = bwrapArgs.indexOf('/tmp/not-here.txt');
expect(bwrapArgs[idx - 2]).toBe('--symlink');
expect(bwrapArgs[idx - 1]).toBe('/dev/null');
});
it('masks directory symlinks with tmpfs for both paths', async () => {
vi.spyOn(fs.promises, 'stat').mockImplementation(
async () => ({ isDirectory: () => true }) as fs.Stats,
);
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(
async (p) => {
if (p === '/tmp/dir-link') return '/opt/real-dir';
return p.toString();
},
vi.mocked(fs.statSync).mockImplementation(
() => ({ isDirectory: () => true }) as fs.Stats,
);
vi.mocked(fs.realpathSync).mockImplementation((p) => {
if (p === '/tmp/dir-link') return '/opt/real-dir';
return p.toString();
});
const bwrapArgs = await getBwrapArgs({
command: 'ls',
@@ -391,25 +422,15 @@ describe('LinuxSandboxManager', () => {
},
});
expectDynamicBinds(bwrapArgs, [
'--tmpfs',
'/opt/real-dir',
'--remount-ro',
'/opt/real-dir',
'--tmpfs',
'/tmp/dir-link',
'--remount-ro',
'/tmp/dir-link',
]);
const idx = bwrapArgs.indexOf('/opt/real-dir');
expect(bwrapArgs[idx - 1]).toBe('--tmpfs');
});
it('should override allowed paths if a path is also in forbidden paths', async () => {
vi.spyOn(fs.promises, 'stat').mockImplementation(
async () => ({ isDirectory: () => true }) as fs.Stats,
);
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(async (p) =>
p.toString(),
vi.mocked(fs.statSync).mockImplementation(
() => ({ isDirectory: () => true }) as fs.Stats,
);
vi.mocked(fs.realpathSync).mockImplementation((p) => p.toString());
const bwrapArgs = await getBwrapArgs({
command: 'ls',
@@ -422,15 +443,12 @@ describe('LinuxSandboxManager', () => {
},
});
expectDynamicBinds(bwrapArgs, [
'--bind-try',
'/tmp/conflict',
'/tmp/conflict',
'--tmpfs',
'/tmp/conflict',
'--remount-ro',
'/tmp/conflict',
]);
const bindTryIdx = bwrapArgs.indexOf('--bind-try');
const tmpfsIdx = bwrapArgs.lastIndexOf('--tmpfs');
expect(bwrapArgs[bindTryIdx + 1]).toBe('/tmp/conflict');
expect(bwrapArgs[tmpfsIdx + 1]).toBe('/tmp/conflict');
expect(tmpfsIdx).toBeGreaterThan(bindTryIdx);
});
});
});
@@ -5,6 +5,7 @@
*/
import fs from 'node:fs';
import { debugLogger } from '../../utils/debugLogger.js';
import { join, dirname, normalize } from 'node:path';
import os from 'node:os';
import {
@@ -12,15 +13,25 @@ import {
type GlobalSandboxOptions,
type SandboxRequest,
type SandboxedCommand,
type SandboxPermissions,
GOVERNANCE_FILES,
sanitizePaths,
tryRealpath,
} from '../../services/sandboxManager.js';
import {
sanitizeEnvironment,
getSecureSanitizationConfig,
} from '../../services/environmentSanitization.js';
import { isNodeError } from '../../utils/errors.js';
import { type SandboxPolicyManager } from '../../policy/sandboxPolicyManager.js';
import {
isStrictlyApproved,
verifySandboxOverrides,
getCommandName,
} from '../utils/commandUtils.js';
import {
tryRealpath,
resolveGitWorktreePaths,
isErrnoException,
} from '../utils/fsUtils.js';
let cachedBpfPath: string | undefined;
@@ -102,13 +113,24 @@ function touch(filePath: string, isDirectory: boolean) {
import {
isKnownSafeCommand,
isDangerousCommand,
} from '../macos/commandSafety.js';
} from '../utils/commandSafety.js';
/**
* A SandboxManager implementation for Linux that uses Bubblewrap (bwrap).
*/
export interface LinuxSandboxOptions extends GlobalSandboxOptions {
modeConfig?: {
readonly?: boolean;
network?: boolean;
approvedTools?: string[];
allowOverrides?: boolean;
};
policyManager?: SandboxPolicyManager;
}
export class LinuxSandboxManager implements SandboxManager {
constructor(private readonly options: GlobalSandboxOptions) {}
constructor(private readonly options: LinuxSandboxOptions) {}
isKnownSafeCommand(args: string[]): boolean {
return isKnownSafeCommand(args);
@@ -119,6 +141,41 @@ export class LinuxSandboxManager implements SandboxManager {
}
async prepareCommand(req: SandboxRequest): Promise<SandboxedCommand> {
const isReadonlyMode = this.options.modeConfig?.readonly ?? true;
const allowOverrides = this.options.modeConfig?.allowOverrides ?? true;
verifySandboxOverrides(allowOverrides, req.policy);
const commandName = await getCommandName(req);
const isApproved = allowOverrides
? await isStrictlyApproved(req, this.options.modeConfig?.approvedTools)
: false;
const workspaceWrite = !isReadonlyMode || isApproved;
const networkAccess =
this.options.modeConfig?.network ?? req.policy?.networkAccess ?? false;
const persistentPermissions = allowOverrides
? this.options.policyManager?.getCommandPermissions(commandName)
: undefined;
const mergedAdditional: SandboxPermissions = {
fileSystem: {
read: [
...(persistentPermissions?.fileSystem?.read ?? []),
...(req.policy?.additionalPermissions?.fileSystem?.read ?? []),
],
write: [
...(persistentPermissions?.fileSystem?.write ?? []),
...(req.policy?.additionalPermissions?.fileSystem?.write ?? []),
],
},
network:
networkAccess ||
persistentPermissions?.network ||
req.policy?.additionalPermissions?.network ||
false,
};
const sanitizationConfig = getSecureSanitizationConfig(
req.policy?.sanitizationConfig,
);
@@ -126,13 +183,142 @@ export class LinuxSandboxManager implements SandboxManager {
const sanitizedEnv = sanitizeEnvironment(req.env, sanitizationConfig);
const bwrapArgs: string[] = [
...this.getNetworkArgs(req),
...this.getBaseArgs(),
...this.getGovernanceArgs(),
...this.getAllowedPathsArgs(req.policy?.allowedPaths),
...(await this.getForbiddenPathsArgs(req.policy?.forbiddenPaths)),
'--unshare-all',
'--new-session', // Isolate session
'--die-with-parent', // Prevent orphaned runaway processes
];
if (mergedAdditional.network) {
bwrapArgs.push('--share-net');
}
bwrapArgs.push(
'--ro-bind',
'/',
'/',
'--dev', // Creates a safe, minimal /dev (replaces --dev-bind)
'/dev',
'--proc', // Creates a fresh procfs for the unshared PID namespace
'/proc',
'--tmpfs', // Provides an isolated, writable /tmp directory
'/tmp',
);
const workspacePath = tryRealpath(this.options.workspace);
const bindFlag = workspaceWrite ? '--bind-try' : '--ro-bind-try';
if (workspaceWrite) {
bwrapArgs.push(
'--bind-try',
this.options.workspace,
this.options.workspace,
);
if (workspacePath !== this.options.workspace) {
bwrapArgs.push('--bind-try', workspacePath, workspacePath);
}
} else {
bwrapArgs.push(
'--ro-bind-try',
this.options.workspace,
this.options.workspace,
);
if (workspacePath !== this.options.workspace) {
bwrapArgs.push('--ro-bind-try', workspacePath, workspacePath);
}
}
const { worktreeGitDir, mainGitDir } =
resolveGitWorktreePaths(workspacePath);
if (worktreeGitDir) {
bwrapArgs.push(bindFlag, worktreeGitDir, worktreeGitDir);
}
if (mainGitDir) {
bwrapArgs.push(bindFlag, mainGitDir, mainGitDir);
}
const allowedPaths = sanitizePaths(req.policy?.allowedPaths) || [];
const normalizedWorkspace = normalize(workspacePath).replace(/\/$/, '');
for (const allowedPath of allowedPaths) {
const resolved = tryRealpath(allowedPath);
if (!fs.existsSync(resolved)) continue;
const normalizedAllowedPath = normalize(resolved).replace(/\/$/, '');
if (normalizedAllowedPath !== normalizedWorkspace) {
if (
!workspaceWrite &&
normalizedAllowedPath.startsWith(normalizedWorkspace + '/')
) {
bwrapArgs.push('--ro-bind-try', resolved, resolved);
} else {
bwrapArgs.push('--bind-try', resolved, resolved);
}
}
}
const additionalReads =
sanitizePaths(mergedAdditional.fileSystem?.read) || [];
for (const p of additionalReads) {
try {
const safeResolvedPath = tryRealpath(p);
bwrapArgs.push('--ro-bind-try', safeResolvedPath, safeResolvedPath);
} catch (e: unknown) {
debugLogger.warn(e instanceof Error ? e.message : String(e));
}
}
const additionalWrites =
sanitizePaths(mergedAdditional.fileSystem?.write) || [];
for (const p of additionalWrites) {
try {
const safeResolvedPath = tryRealpath(p);
bwrapArgs.push('--bind-try', safeResolvedPath, safeResolvedPath);
} catch (e: unknown) {
debugLogger.warn(e instanceof Error ? e.message : String(e));
}
}
for (const file of GOVERNANCE_FILES) {
const filePath = join(this.options.workspace, file.path);
touch(filePath, file.isDirectory);
const realPath = tryRealpath(filePath);
bwrapArgs.push('--ro-bind', filePath, filePath);
if (realPath !== filePath) {
bwrapArgs.push('--ro-bind', realPath, realPath);
}
}
const forbiddenPaths = sanitizePaths(req.policy?.forbiddenPaths) || [];
for (const p of forbiddenPaths) {
let resolved: string;
try {
resolved = tryRealpath(p); // Forbidden paths should still resolve to block the real path
if (!fs.existsSync(resolved)) continue;
} catch (e: unknown) {
debugLogger.warn(
`Failed to resolve forbidden path ${p}: ${e instanceof Error ? e.message : String(e)}`,
);
bwrapArgs.push('--ro-bind', '/dev/null', p);
continue;
}
try {
const stat = fs.statSync(resolved);
if (stat.isDirectory()) {
bwrapArgs.push('--tmpfs', resolved, '--remount-ro', resolved);
} else {
bwrapArgs.push('--ro-bind', '/dev/null', resolved);
}
} catch (e: unknown) {
if (isErrnoException(e) && e.code === 'ENOENT') {
bwrapArgs.push('--symlink', '/dev/null', resolved);
} else {
debugLogger.warn(
`Failed to stat forbidden path ${resolved}: ${e instanceof Error ? e.message : String(e)}`,
);
bwrapArgs.push('--ro-bind', '/dev/null', resolved);
}
}
}
const bpfPath = getSeccompBpfPath();
bwrapArgs.push('--seccomp', '9');
@@ -153,142 +339,4 @@ export class LinuxSandboxManager implements SandboxManager {
cwd: req.cwd,
};
}
/**
* Generates arguments for network isolation.
*/
private getNetworkArgs(req: SandboxRequest): string[] {
return req.policy?.networkAccess
? [
'--unshare-user',
'--unshare-ipc',
'--unshare-pid',
'--unshare-uts',
'--unshare-cgroup',
]
: ['--unshare-all'];
}
/**
* Generates the base bubblewrap arguments for isolation.
*/
private getBaseArgs(): string[] {
return [
'--new-session', // Isolate session
'--die-with-parent', // Prevent orphaned runaway processes
'--ro-bind',
'/',
'/',
'--dev', // Creates a safe, minimal /dev (replaces --dev-bind)
'/dev',
'--proc', // Creates a fresh procfs for the unshared PID namespace
'/proc',
'--tmpfs', // Provides an isolated, writable /tmp directory
'/tmp',
// Note: --dev /dev sets up /dev/pts automatically
'--bind',
this.options.workspace,
this.options.workspace,
];
}
/**
* Generates arguments for protected governance files.
*/
private getGovernanceArgs(): string[] {
const args: string[] = [];
// Protected governance files are bind-mounted as read-only, even if the workspace is RW.
// We ensure they exist on the host and resolve real paths to prevent symlink bypasses.
// In bwrap, later binds override earlier ones for the same path.
for (const file of GOVERNANCE_FILES) {
const filePath = join(this.options.workspace, file.path);
touch(filePath, file.isDirectory);
const realPath = fs.realpathSync(filePath);
args.push('--ro-bind', filePath, filePath);
if (realPath !== filePath) {
args.push('--ro-bind', realPath, realPath);
}
}
return args;
}
/**
* Generates arguments for allowed paths.
*/
private getAllowedPathsArgs(allowedPaths?: string[]): string[] {
const args: string[] = [];
const paths = sanitizePaths(allowedPaths) || [];
const normalizedWorkspace = this.normalizePath(this.options.workspace);
for (const p of paths) {
if (this.normalizePath(p) !== normalizedWorkspace) {
args.push('--bind-try', p, p);
}
}
return args;
}
/**
* Generates arguments for forbidden paths.
*/
private async getForbiddenPathsArgs(
forbiddenPaths?: string[],
): Promise<string[]> {
const args: string[] = [];
const paths = sanitizePaths(forbiddenPaths) || [];
for (const p of paths) {
try {
const originalPath = this.normalizePath(p);
const resolvedPath = await tryRealpath(originalPath);
// Mask the resolved path to prevent access to the underlying file.
const resolvedMask = await this.getMaskArgs(resolvedPath);
args.push(...resolvedMask);
// If the original path was a symlink, mask it as well to prevent access
// through the link itself.
if (resolvedPath !== originalPath) {
const originalMask = await this.getMaskArgs(originalPath);
args.push(...originalMask);
}
} catch (e) {
throw new Error(
`Failed to deny access to forbidden path: ${p}. ${
e instanceof Error ? e.message : String(e)
}`,
);
}
}
return args;
}
/**
* Generates bubblewrap arguments to mask a forbidden path.
*/
private async getMaskArgs(path: string): Promise<string[]> {
try {
const stats = await fs.promises.stat(path);
if (stats.isDirectory()) {
// Directories are masked by mounting an empty, read-only tmpfs.
return ['--tmpfs', path, '--remount-ro', path];
}
// Existing files are masked by binding them to /dev/null.
return ['--ro-bind-try', '/dev/null', path];
} catch (e) {
if (isNodeError(e) && e.code === 'ENOENT') {
// Non-existent paths are masked by a broken symlink. This prevents
// creation within the sandbox while avoiding host remnants.
return ['--symlink', '/.forbidden', path];
}
throw e;
}
}
private normalizePath(p: string): string {
return normalize(p).replace(/\/$/, '');
}
}
@@ -38,7 +38,7 @@ describe('MacOsSandboxManager', () => {
manager = new MacOsSandboxManager({ workspace: mockWorkspace });
// Mock the seatbelt args builder to isolate manager tests
vi.spyOn(seatbeltArgsBuilder, 'buildSeatbeltArgs').mockResolvedValue([
vi.spyOn(seatbeltArgsBuilder, 'buildSeatbeltArgs').mockReturnValue([
'-p',
'(mock profile)',
'-D',
@@ -24,8 +24,9 @@ import {
isKnownSafeCommand,
isDangerousCommand,
isStrictlyApproved,
} from './commandSafety.js';
} from '../utils/commandSafety.js';
import { type SandboxPolicyManager } from '../../policy/sandboxPolicyManager.js';
import { verifySandboxOverrides } from '../utils/commandUtils.js';
export interface MacOsSandboxOptions extends GlobalSandboxOptions {
/** The current sandbox mode behavior from config. */
@@ -70,17 +71,7 @@ export class MacOsSandboxManager implements SandboxManager {
const allowOverrides = this.options.modeConfig?.allowOverrides ?? true;
// Reject override attempts in plan mode
if (!allowOverrides && req.policy?.additionalPermissions) {
const perms = req.policy.additionalPermissions;
if (
perms.network ||
(perms.fileSystem?.write && perms.fileSystem.write.length > 0)
) {
throw new Error(
'Sandbox request rejected: Cannot override readonly/network restrictions in Plan mode.',
);
}
}
verifySandboxOverrides(allowOverrides, req.policy);
// If not in readonly mode OR it's a strictly approved pipeline, allow workspace writes
const isApproved = allowOverrides
@@ -120,7 +111,7 @@ export class MacOsSandboxManager implements SandboxManager {
false,
};
const sandboxArgs = await buildSeatbeltArgs({
const sandboxArgs = buildSeatbeltArgs({
workspace: this.options.workspace,
allowedPaths: [...(req.policy?.allowedPaths || [])],
forbiddenPaths: req.policy?.forbiddenPaths,
@@ -3,25 +3,31 @@
* Copyright 2026 Google LLC
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { describe, it, expect, vi, afterEach } from 'vitest';
import { buildSeatbeltArgs } from './seatbeltArgsBuilder.js';
import * as sandboxManager from '../../services/sandboxManager.js';
import * as fsUtils from '../utils/fsUtils.js';
import fs from 'node:fs';
import os from 'node:os';
vi.mock('../utils/fsUtils.js', async () => {
const actual = await vi.importActual('../utils/fsUtils.js');
return {
...actual,
tryRealpath: vi.fn((p) => p),
resolveGitWorktreePaths: vi.fn(() => ({})),
};
});
describe('seatbeltArgsBuilder', () => {
beforeEach(() => {
afterEach(() => {
vi.restoreAllMocks();
});
describe('buildSeatbeltArgs', () => {
it('should build a strict allowlist profile allowing the workspace via param', async () => {
// Mock tryRealpath to just return the path for testing
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(
async (p) => p,
);
it('should build a strict allowlist profile allowing the workspace via param', () => {
vi.mocked(fsUtils.tryRealpath).mockImplementation((p) => p);
const args = await buildSeatbeltArgs({
const args = buildSeatbeltArgs({
workspace: '/Users/test/workspace',
});
@@ -38,11 +44,9 @@ describe('seatbeltArgsBuilder', () => {
expect(args).toContain(`TMPDIR=${os.tmpdir()}`);
});
it('should allow network when networkAccess is true', async () => {
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(
async (p) => p,
);
const args = await buildSeatbeltArgs({
it('should allow network when networkAccess is true', () => {
vi.mocked(fsUtils.tryRealpath).mockImplementation((p) => p);
const args = buildSeatbeltArgs({
workspace: '/test',
networkAccess: true,
});
@@ -51,10 +55,8 @@ describe('seatbeltArgsBuilder', () => {
});
describe('governance files', () => {
it('should inject explicit deny rules for governance files', async () => {
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(async (p) =>
p.toString(),
);
it('should inject explicit deny rules for governance files', () => {
vi.mocked(fsUtils.tryRealpath).mockImplementation((p) => p.toString());
vi.spyOn(fs, 'existsSync').mockReturnValue(true);
vi.spyOn(fs, 'lstatSync').mockImplementation(
(p) =>
@@ -64,35 +66,29 @@ describe('seatbeltArgsBuilder', () => {
}) as unknown as fs.Stats,
);
const args = await buildSeatbeltArgs({
workspace: '/Users/test/workspace',
const args = buildSeatbeltArgs({
workspace: '/test/workspace',
});
const profile = args[1];
// .gitignore should be a literal deny
expect(args).toContain('-D');
expect(args).toContain(
'GOVERNANCE_FILE_0=/Users/test/workspace/.gitignore',
);
expect(args).toContain('GOVERNANCE_FILE_0=/test/workspace/.gitignore');
expect(profile).toContain(
'(deny file-write* (literal (param "GOVERNANCE_FILE_0")))',
);
// .git should be a subpath deny
expect(args).toContain('GOVERNANCE_FILE_2=/Users/test/workspace/.git');
expect(args).toContain('GOVERNANCE_FILE_2=/test/workspace/.git');
expect(profile).toContain(
'(deny file-write* (subpath (param "GOVERNANCE_FILE_2")))',
);
});
it('should protect both the symlink and the real path if they differ', async () => {
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(
async (p) => {
if (p === '/test/workspace/.gitignore')
return '/test/real/.gitignore';
return p.toString();
},
);
it('should protect both the symlink and the real path if they differ', () => {
vi.mocked(fsUtils.tryRealpath).mockImplementation((p) => {
if (p === '/test/workspace/.gitignore')
return '/test/real/.gitignore';
return p.toString();
});
vi.spyOn(fs, 'existsSync').mockReturnValue(true);
vi.spyOn(fs, 'lstatSync').mockImplementation(
() =>
@@ -102,7 +98,7 @@ describe('seatbeltArgsBuilder', () => {
}) as unknown as fs.Stats,
);
const args = await buildSeatbeltArgs({ workspace: '/test/workspace' });
const args = buildSeatbeltArgs({ workspace: '/test/workspace' });
const profile = args[1];
expect(args).toContain('GOVERNANCE_FILE_0=/test/workspace/.gitignore');
@@ -117,15 +113,13 @@ describe('seatbeltArgsBuilder', () => {
});
describe('allowedPaths', () => {
it('should parameterize allowed paths and normalize them', async () => {
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(
async (p) => {
if (p === '/test/symlink') return '/test/real_path';
return p;
},
);
it('should parameterize allowed paths and normalize them', () => {
vi.mocked(fsUtils.tryRealpath).mockImplementation((p) => {
if (p === '/test/symlink') return '/test/real_path';
return p;
});
const args = await buildSeatbeltArgs({
const args = buildSeatbeltArgs({
workspace: '/test',
allowedPaths: ['/custom/path1', '/test/symlink'],
});
@@ -141,12 +135,10 @@ describe('seatbeltArgsBuilder', () => {
});
describe('forbiddenPaths', () => {
it('should parameterize forbidden paths and explicitly deny them', async () => {
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(
async (p) => p,
);
it('should parameterize forbidden paths and explicitly deny them', () => {
vi.mocked(fsUtils.tryRealpath).mockImplementation((p) => p);
const args = await buildSeatbeltArgs({
const args = buildSeatbeltArgs({
workspace: '/test',
forbiddenPaths: ['/secret/path'],
});
@@ -161,22 +153,21 @@ describe('seatbeltArgsBuilder', () => {
);
});
it('resolves forbidden symlink paths to their real paths', async () => {
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(
async (p) => {
if (p === '/test/symlink') return '/test/real_path';
return p;
},
);
it('resolves forbidden symlink paths to their real paths', () => {
vi.mocked(fsUtils.tryRealpath).mockImplementation((p) => {
if (p === '/test/symlink' || p === '/test/missing-dir') {
return '/test/real_path';
}
return p;
});
const args = await buildSeatbeltArgs({
const args = buildSeatbeltArgs({
workspace: '/test',
forbiddenPaths: ['/test/symlink'],
});
const profile = args[1];
// The builder should resolve the symlink and explicitly deny the real target path
expect(args).toContain('-D');
expect(args).toContain('FORBIDDEN_PATH_0=/test/real_path');
expect(profile).toContain(
@@ -184,12 +175,10 @@ describe('seatbeltArgsBuilder', () => {
);
});
it('explicitly denies non-existent forbidden paths to prevent creation', async () => {
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(
async (p) => p,
);
it('explicitly denies non-existent forbidden paths to prevent creation', () => {
vi.mocked(fsUtils.tryRealpath).mockImplementation((p) => p);
const args = await buildSeatbeltArgs({
const args = buildSeatbeltArgs({
workspace: '/test',
forbiddenPaths: ['/test/missing-dir/missing-file.txt'],
});
@@ -205,12 +194,10 @@ describe('seatbeltArgsBuilder', () => {
);
});
it('should override allowed paths if a path is also in forbidden paths', async () => {
vi.spyOn(sandboxManager, 'tryRealpath').mockImplementation(
async (p) => p,
);
it('should override allowed paths if a path is also in forbidden paths', () => {
vi.mocked(fsUtils.tryRealpath).mockImplementation((p) => p);
const args = await buildSeatbeltArgs({
const args = buildSeatbeltArgs({
workspace: '/test',
allowedPaths: ['/custom/path1'],
forbiddenPaths: ['/custom/path1'],
@@ -226,8 +213,6 @@ describe('seatbeltArgsBuilder', () => {
expect(profile).toContain(allowString);
expect(profile).toContain(denyString);
// Verify ordering: The explicit deny must appear AFTER the explicit allow in the profile string
// Seatbelt rules are evaluated in order where the latest rule matching a path wins
const allowIndex = profile.indexOf(allowString);
const denyIndex = profile.indexOf(denyString);
expect(denyIndex).toBeGreaterThan(allowIndex);
@@ -15,8 +15,8 @@ import {
type SandboxPermissions,
sanitizePaths,
GOVERNANCE_FILES,
tryRealpath,
} from '../../services/sandboxManager.js';
import { tryRealpath, resolveGitWorktreePaths } from '../utils/fsUtils.js';
/**
* Options for building macOS Seatbelt arguments.
@@ -44,13 +44,11 @@ export interface SeatbeltArgsOptions {
* Returns arguments up to the end of sandbox-exec configuration (e.g. ['-p', '<profile>', '-D', ...])
* Does not include the final '--' separator or the command to run.
*/
export async function buildSeatbeltArgs(
options: SeatbeltArgsOptions,
): Promise<string[]> {
export function buildSeatbeltArgs(options: SeatbeltArgsOptions): string[] {
let profile = BASE_SEATBELT_PROFILE + '\n';
const args: string[] = [];
const workspacePath = await tryRealpath(options.workspace);
const workspacePath = tryRealpath(options.workspace);
args.push('-D', `WORKSPACE=${workspacePath}`);
args.push('-D', `WORKSPACE_RAW=${options.workspace}`);
profile += `(allow file-read* (subpath (param "WORKSPACE_RAW")))\n`;
@@ -67,7 +65,7 @@ export async function buildSeatbeltArgs(
// (Seatbelt evaluates rules in order, later rules win for same path).
for (let i = 0; i < GOVERNANCE_FILES.length; i++) {
const governanceFile = path.join(workspacePath, GOVERNANCE_FILES[i].path);
const realGovernanceFile = await tryRealpath(governanceFile);
const realGovernanceFile = tryRealpath(governanceFile);
// Determine if it should be treated as a directory (subpath) or a file (literal).
// .git is generally a directory, while ignore files are literals.
@@ -92,42 +90,20 @@ export async function buildSeatbeltArgs(
}
// Auto-detect and support git worktrees by granting read and write access to the underlying git directory
try {
const gitPath = path.join(workspacePath, '.git');
const gitStat = fs.lstatSync(gitPath);
if (gitStat.isFile()) {
const gitContent = fs.readFileSync(gitPath, 'utf8');
const match = gitContent.match(/^gitdir:\s*(.+)$/m);
if (match && match[1]) {
let worktreeGitDir = match[1].trim();
if (!path.isAbsolute(worktreeGitDir)) {
worktreeGitDir = path.resolve(workspacePath, worktreeGitDir);
}
const resolvedWorktreeGitDir = await tryRealpath(worktreeGitDir);
// Grant write access to the worktree's specific .git directory
args.push('-D', `WORKTREE_GIT_DIR=${resolvedWorktreeGitDir}`);
profile += `(allow file-read* file-write* (subpath (param "WORKTREE_GIT_DIR")))\n`;
// Grant write access to the main repository's .git directory (objects, refs, etc. are shared)
// resolvedWorktreeGitDir is usually like: /path/to/main-repo/.git/worktrees/worktree-name
const mainGitDir = await tryRealpath(
path.dirname(path.dirname(resolvedWorktreeGitDir)),
);
if (mainGitDir && mainGitDir.endsWith('.git')) {
args.push('-D', `MAIN_GIT_DIR=${mainGitDir}`);
profile += `(allow file-read* file-write* (subpath (param "MAIN_GIT_DIR")))\n`;
}
}
}
} catch (_e) {
// Ignore if .git doesn't exist, isn't readable, etc.
const { worktreeGitDir, mainGitDir } = resolveGitWorktreePaths(workspacePath);
if (worktreeGitDir) {
args.push('-D', `WORKTREE_GIT_DIR=${worktreeGitDir}`);
profile += `(allow file-read* file-write* (subpath (param "WORKTREE_GIT_DIR")))\n`;
}
if (mainGitDir) {
args.push('-D', `MAIN_GIT_DIR=${mainGitDir}`);
profile += `(allow file-read* file-write* (subpath (param "MAIN_GIT_DIR")))\n`;
}
const tmpPath = await tryRealpath(os.tmpdir());
const tmpPath = tryRealpath(os.tmpdir());
args.push('-D', `TMPDIR=${tmpPath}`);
const nodeRootPath = await tryRealpath(
const nodeRootPath = tryRealpath(
path.dirname(path.dirname(process.execPath)),
);
args.push('-D', `NODE_ROOT=${nodeRootPath}`);
@@ -142,7 +118,7 @@ export async function buildSeatbeltArgs(
for (const p of paths) {
if (!p.trim()) continue;
try {
let resolved = await tryRealpath(p);
let resolved = tryRealpath(p);
// If this is a 'bin' directory (like /usr/local/bin or homebrew/bin),
// also grant read access to its parent directory so that symlinked
@@ -165,8 +141,10 @@ export async function buildSeatbeltArgs(
// Handle allowedPaths
const allowedPaths = sanitizePaths(options.allowedPaths) || [];
const resolvedAllowedPaths: string[] = [];
for (let i = 0; i < allowedPaths.length; i++) {
const allowedPath = await tryRealpath(allowedPaths[i]);
const allowedPath = tryRealpath(allowedPaths[i]);
resolvedAllowedPaths.push(allowedPath);
args.push('-D', `ALLOWED_PATH_${i}=${allowedPath}`);
profile += `(allow file-read* file-write* (subpath (param "ALLOWED_PATH_${i}")))\n`;
}
@@ -176,7 +154,7 @@ export async function buildSeatbeltArgs(
const { read, write } = options.additionalPermissions.fileSystem;
if (read) {
for (let i = 0; i < read.length; i++) {
const resolved = await tryRealpath(read[i]);
const resolved = tryRealpath(read[i]);
const paramName = `ADDITIONAL_READ_${i}`;
args.push('-D', `${paramName}=${resolved}`);
let isFile = false;
@@ -194,7 +172,7 @@ export async function buildSeatbeltArgs(
}
if (write) {
for (let i = 0; i < write.length; i++) {
const resolved = await tryRealpath(write[i]);
const resolved = tryRealpath(write[i]);
const paramName = `ADDITIONAL_WRITE_${i}`;
args.push('-D', `${paramName}=${resolved}`);
let isFile = false;
@@ -215,7 +193,7 @@ export async function buildSeatbeltArgs(
// Handle forbiddenPaths
const forbiddenPaths = sanitizePaths(options.forbiddenPaths) || [];
for (let i = 0; i < forbiddenPaths.length; i++) {
const forbiddenPath = await tryRealpath(forbiddenPaths[i]);
const forbiddenPath = tryRealpath(forbiddenPaths[i]);
args.push('-D', `FORBIDDEN_PATH_${i}=${forbiddenPath}`);
profile += `(deny file-read* file-write* (subpath (param "FORBIDDEN_PATH_${i}")))\n`;
}
@@ -0,0 +1,82 @@
/**
* @license
* Copyright 2026 Google LLC
* SPDX-License-Identifier: Apache-2.0
*/
import { type SandboxRequest } from '../../services/sandboxManager.js';
import {
getCommandRoots,
initializeShellParsers,
splitCommands,
stripShellWrapper,
} from '../../utils/shell-utils.js';
import { isKnownSafeCommand } from './commandSafety.js';
import { parse as shellParse } from 'shell-quote';
import path from 'node:path';
export async function isStrictlyApproved(
req: SandboxRequest,
approvedTools?: string[],
): Promise<boolean> {
if (!approvedTools || approvedTools.length === 0) {
return false;
}
await initializeShellParsers();
const fullCmd = [req.command, ...req.args].join(' ');
const stripped = stripShellWrapper(fullCmd);
const roots = getCommandRoots(stripped);
if (roots.length === 0) return false;
const allRootsApproved = roots.every((root) => approvedTools.includes(root));
if (allRootsApproved) {
return true;
}
const pipelineCommands = splitCommands(stripped);
if (pipelineCommands.length === 0) return false;
for (const cmdString of pipelineCommands) {
const parsedArgs = shellParse(cmdString).map(String);
if (!isKnownSafeCommand(parsedArgs)) {
return false;
}
}
return true;
}
export async function getCommandName(req: SandboxRequest): Promise<string> {
await initializeShellParsers();
const fullCmd = [req.command, ...req.args].join(' ');
const stripped = stripShellWrapper(fullCmd);
const roots = getCommandRoots(stripped).filter(
(r) => r !== 'shopt' && r !== 'set',
);
if (roots.length > 0) {
return roots[0];
}
return path.basename(req.command);
}
export function verifySandboxOverrides(
allowOverrides: boolean,
policy: SandboxRequest['policy'],
) {
if (!allowOverrides) {
if (
policy?.networkAccess ||
policy?.allowedPaths?.length ||
policy?.additionalPermissions?.network ||
policy?.additionalPermissions?.fileSystem?.read?.length ||
policy?.additionalPermissions?.fileSystem?.write?.length
) {
throw new Error(
'Sandbox request rejected: Cannot override readonly/network/filesystem restrictions in Plan mode.',
);
}
}
}
@@ -0,0 +1,92 @@
/**
* @license
* Copyright 2026 Google LLC
* SPDX-License-Identifier: Apache-2.0
*/
import fs from 'node:fs';
import path from 'node:path';
export function isErrnoException(e: unknown): e is NodeJS.ErrnoException {
return e instanceof Error && 'code' in e;
}
export function tryRealpath(p: string): string {
try {
return fs.realpathSync(p);
} catch (_e) {
if (isErrnoException(_e) && _e.code === 'ENOENT') {
const parentDir = path.dirname(p);
if (parentDir === p) {
return p;
}
return path.join(tryRealpath(parentDir), path.basename(p));
}
throw _e;
}
}
export function resolveGitWorktreePaths(workspacePath: string): {
worktreeGitDir?: string;
mainGitDir?: string;
} {
try {
const gitPath = path.join(workspacePath, '.git');
const gitStat = fs.lstatSync(gitPath);
if (gitStat.isFile()) {
const gitContent = fs.readFileSync(gitPath, 'utf8');
const match = gitContent.match(/^gitdir:\s+(.+)$/m);
if (match && match[1]) {
let worktreeGitDir = match[1].trim();
if (!path.isAbsolute(worktreeGitDir)) {
worktreeGitDir = path.resolve(workspacePath, worktreeGitDir);
}
const resolvedWorktreeGitDir = tryRealpath(worktreeGitDir);
// Security check: Verify the bidirectional link to prevent sandbox escape
let isValid = false;
try {
const backlinkPath = path.join(resolvedWorktreeGitDir, 'gitdir');
const backlink = fs.readFileSync(backlinkPath, 'utf8').trim();
// The backlink must resolve to the workspace's .git file
if (tryRealpath(backlink) === tryRealpath(gitPath)) {
isValid = true;
}
} catch (_e) {
// Fallback for submodules: check core.worktree in config
try {
const configPath = path.join(resolvedWorktreeGitDir, 'config');
const config = fs.readFileSync(configPath, 'utf8');
const match = config.match(/^\s*worktree\s*=\s*(.+)$/m);
if (match && match[1]) {
const worktreePath = path.resolve(
resolvedWorktreeGitDir,
match[1].trim(),
);
if (tryRealpath(worktreePath) === tryRealpath(workspacePath)) {
isValid = true;
}
}
} catch (_e2) {
// Ignore
}
}
if (!isValid) {
return {}; // Reject: valid worktrees/submodules must have a readable backlink
}
const mainGitDir = tryRealpath(
path.dirname(path.dirname(resolvedWorktreeGitDir)),
);
return {
worktreeGitDir: resolvedWorktreeGitDir,
mainGitDir: mainGitDir.endsWith('.git') ? mainGitDir : undefined,
};
}
}
} catch (_e) {
// Ignore if .git doesn't exist, isn't readable, etc.
}
return {};
}
@@ -111,7 +111,7 @@ describe('WindowsSandboxManager', () => {
};
await expect(planManager.prepareCommand(req)).rejects.toThrow(
'Sandbox request rejected: Cannot override readonly/network restrictions in Plan mode.',
'Sandbox request rejected: Cannot override readonly/network/filesystem restrictions in Plan mode.',
);
});
@@ -31,6 +31,7 @@ import {
isStrictlyApproved,
} from './commandSafety.js';
import { type SandboxPolicyManager } from '../../policy/sandboxPolicyManager.js';
import { verifySandboxOverrides } from '../utils/commandUtils.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
@@ -214,17 +215,7 @@ export class WindowsSandboxManager implements SandboxManager {
const allowOverrides = this.options.modeConfig?.allowOverrides ?? true;
// Reject override attempts in plan mode
if (!allowOverrides && req.policy?.additionalPermissions) {
const perms = req.policy.additionalPermissions;
if (
perms.network ||
(perms.fileSystem?.write && perms.fileSystem.write.length > 0)
) {
throw new Error(
'Sandbox request rejected: Cannot override readonly/network restrictions in Plan mode.',
);
}
}
verifySandboxOverrides(allowOverrides, req.policy);
// Fetch persistent approvals for this command
const commandName = await getCommandName(req.command, req.args);
+1 -1
View File
@@ -10,7 +10,7 @@ import path from 'node:path';
import {
isKnownSafeCommand as isMacSafeCommand,
isDangerousCommand as isMacDangerousCommand,
} from '../sandbox/macos/commandSafety.js';
} from '../sandbox/utils/commandSafety.js';
import {
isKnownSafeCommand as isWindowsSafeCommand,
isDangerousCommand as isWindowsDangerousCommand,
@@ -42,7 +42,11 @@ export function createSandboxManager(
policyManager,
});
} else if (os.platform() === 'linux') {
return new LinuxSandboxManager({ workspace });
return new LinuxSandboxManager({
workspace,
modeConfig,
policyManager,
});
} else if (os.platform() === 'darwin') {
return new MacOsSandboxManager({
workspace,
+117
View File
@@ -0,0 +1,117 @@
#!/bin/bash
# Gemini API Reliability Harvester
# -------------------------------
# This script gathers data about 500 API errors encountered during evaluation runs
# (eval.yml) from GitHub Actions. It is used to analyze developer friction caused
# by transient API failures.
#
# Usage:
# ./scripts/harvest_api_reliability.sh [SINCE] [LIMIT] [BRANCH]
#
# Examples:
# ./scripts/harvest_api_reliability.sh # Last 7 days, all branches
# ./scripts/harvest_api_reliability.sh 14d 500 # Last 14 days, limit 500
# ./scripts/harvest_api_reliability.sh 2026-03-01 100 my-branch # Specific date and branch
#
# Prerequisites:
# - GitHub CLI (gh) installed and authenticated (`gh auth login`)
# - jq installed
# Arguments & Defaults
if [[ -n "$1" && $1 =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]; then
SINCE="$1"
elif [[ -n "$1" && $1 =~ ^([0-9]+)d$ ]]; then
DAYS="${BASH_REMATCH[1]}"
if [[ "$OSTYPE" == "darwin"* ]]; then
SINCE=$(date -u -v-"${DAYS}"d +%Y-%m-%d)
else
SINCE=$(date -u -d "${DAYS} days ago" +%Y-%m-%d)
fi
else
# Default to 7 days ago in YYYY-MM-DD format (UTC)
if [[ "$OSTYPE" == "darwin"* ]]; then
SINCE=$(date -u -v-7d +%Y-%m-%d)
else
SINCE=$(date -u -d "7 days ago" +%Y-%m-%d)
fi
fi
LIMIT=${2:-300}
BRANCH=${3:-""}
WORKFLOWS=("Testing: E2E (Chained)" "Evals: Nightly")
DEST_DIR=$(mktemp -d -t gemini-reliability-XXXXXX)
MERGED_FILE="api-reliability-summary.jsonl"
# Ensure cleanup on exit
trap 'rm -rf "$DEST_DIR"' EXIT
if ! command -v gh &> /dev/null; then
echo "❌ Error: GitHub CLI (gh) is not installed."
exit 1
fi
if ! command -v jq &> /dev/null; then
echo "❌ Error: jq is not installed."
exit 1
fi
# Clean start
rm -f "$MERGED_FILE"
# gh run list --created expects a date (YYYY-MM-DD) or a range
CREATED_QUERY=">=$SINCE"
for WORKFLOW in "${WORKFLOWS[@]}"; do
echo "🔍 Fetching runs for '$WORKFLOW' created since $SINCE (max $LIMIT runs, branch: ${BRANCH:-all})..."
# Construct arguments for gh run list
GH_ARGS=("--workflow" "$WORKFLOW" "--created" "$CREATED_QUERY" "--limit" "$LIMIT" "--json" "databaseId" "--jq" ".[].databaseId")
if [ -n "$BRANCH" ]; then
GH_ARGS+=("--branch" "$BRANCH")
fi
RUN_IDS=$(gh run list "${GH_ARGS[@]}")
exit_code=$?
if [ $exit_code -ne 0 ]; then
echo "❌ Failed to fetch runs for '$WORKFLOW' (exit code: $exit_code). Please check 'gh auth status' and permissions." >&2
continue
fi
if [ -z "$RUN_IDS" ]; then
echo "📭 No runs found for workflow '$WORKFLOW' since $SINCE."
continue
fi
for ID in $RUN_IDS; do
# Download artifacts named 'eval-logs-*'
# Silencing output because many older runs won't have artifacts
gh run download "$ID" -p "eval-logs-*" -D "$DEST_DIR/$ID" &>/dev/null || continue
# Append to master log
# Use find to locate api-reliability.jsonl in any subdirectory of $DEST_DIR/$ID
find "$DEST_DIR/$ID" -type f -name "api-reliability.jsonl" -exec cat {} + >> "$MERGED_FILE" 2>/dev/null
done
done
if [ ! -f "$MERGED_FILE" ]; then
echo "📭 No reliability data found in the retrieved logs."
exit 0
fi
echo -e "\n✅ Harvest Complete! Data merged into: $MERGED_FILE"
echo "------------------------------------------------"
echo "📊 Gemini API Reliability Summary (Since $SINCE)"
echo "------------------------------------------------"
cat "$MERGED_FILE" | jq -s '
group_by(.model) | map({
model: .[0].model,
"500s": (map(select(.errorCode == "500")) | length),
"503s": (map(select(.errorCode == "503")) | length),
retries: (map(select(.status == "RETRY")) | length),
skips: (map(select(.status == "SKIP")) | length)
})'
echo -e "\n💡 Total events captured: $(wc -l < "$MERGED_FILE")"