cocosheng-g
fe50e580fa
fix(ci): isolate workflow evals, revert unrelated changes, and fix aggregation
2026-02-03 23:04:50 -05:00
cocosheng-g
9da1542071
feat(ci): isolate workflow evals into independent nightly job
...
- Splits 'Evals: Nightly' into 'evals' (general capabilities) and 'workflow-evals' (specific workflow simulations).
- 'workflow-evals' runs only on 'gemini-2.5-pro' (the target model).
- 'evals' excludes workflow tests to prevent noise/skewed metrics on other models.
- Removes code-level 'targetModels' restrictions in favor of CI configuration.
- Updates aggregation script to handle skipped tests correctly (though exclusion avoids them).
2026-02-03 22:37:15 -05:00
cocosheng-g
8c82777ea0
fix(evals): use direct path to tsx binary in dedup mocks
...
Avoids reliance on 'npx' which might be flaky or prompt for installation in CI environments.
2026-02-03 21:39:16 -05:00
cocosheng-g
36ce66933e
chore: remove old eval files moved to workflows/
2026-02-03 21:03:24 -05:00
cocosheng-g
ff4e816a70
refactor(evals): isolate workflow evals and target specific models
2026-02-03 21:02:55 -05:00
cocosheng-g
d9a0e9b9c5
fix(triage): tune prompt to return json text and harden eval json extraction
2026-02-03 19:55:00 -05:00
cocosheng-g
5c2f477adf
feat(evals): add comprehensive workflow evaluations and tune prompts (Issue #219 )
...
- Established evals for all agent workflows (triage, dedup, refresh).
- Refactored all evals to use modern --output-format=json flag for robust validation.
- Tuned prompts for strict JSON compliance and corrected spam handling in scheduled triage.
- Expanded edge case coverage for false positives, security leaks, and mixed batches.
2026-02-03 19:31:56 -05:00
cocosheng-g
2daee0d066
feat(evals): add more edge case tests
2026-02-03 19:31:56 -05:00
cocosheng-g
aa4b1c0056
fix(evals): address robustness feedback
2026-02-03 19:31:56 -05:00
cocosheng-g
9f8f31cce9
fix(evals): address review feedback on triage tests
2026-02-03 19:31:56 -05:00
cocosheng-g
259a3e7891
fix(workflows): tune triage prompt and add robustness evals
2026-02-03 19:31:56 -05:00
Coco Sheng
3183e4137a
fix(test): improve test isolation and enable subagent evaluations ( #18138 )
2026-02-03 19:05:26 +00:00
Christian Gunderman
bc258eba4c
Cleanup post delegate_to_agent removal ( #17875 )
2026-01-29 18:24:35 +00:00
Sandy Tao
0b169e9867
fix(evals): use absolute path for activity log directory ( #17830 )
2026-01-28 15:20:21 -08:00
Sandy Tao
9e09db1ddb
feat(cli): enable activity logging for non-interactive mode and evals ( #17703 )
2026-01-28 17:02:41 +00:00
Christian Gunderman
b6cf189ab2
Fix issue where Gemini CLI can make changes when simply asked a question ( #17608 )
2026-01-27 19:47:33 +00:00
Christian Gunderman
5cf06503c8
Slash command for helping in debugging ( #17609 )
2026-01-27 02:47:04 +00:00
Sehoon Shon
5c649d8db1
feat(ui): display user tier in about command ( #17400 )
2026-01-23 21:03:53 +00:00
Christian Gunderman
2c6781d134
Refactor subagent delegation to be one tool per agent ( #17346 )
2026-01-23 02:18:31 +00:00
Jerop Kipruto
c21c297133
feat(plan): refactor TestRig and eval helper to support configurable approval modes ( #17171 )
2026-01-21 15:43:48 +00:00
joshualitt
f42b4c80ac
feat(core): Add initial eval for generalist agent. ( #16856 )
2026-01-20 20:32:48 +00:00
Christian Gunderman
12b0fe1cc2
Demote the subagent test to nightly ( #17105 )
2026-01-20 18:18:16 +00:00
Christian Gunderman
49769152d6
Demote git evals to nightly run. ( #17030 )
2026-01-19 19:00:41 +00:00
Christian Gunderman
d87a3acdef
Fix inverted logic. ( #17007 )
2026-01-19 07:16:05 +00:00
Christian Gunderman
203f5209ba
Stabilize the git evals ( #16989 )
2026-01-19 06:18:06 +00:00
Christian Gunderman
e03042657b
Don't commit unless user asks us to. ( #16902 )
2026-01-17 01:00:46 +00:00
Christian Gunderman
a15978593a
Steer outer agent to use expert subagents when present ( #16763 )
2026-01-16 16:51:10 +00:00
Christian Gunderman
66e7b479ae
Aggregate test results. ( #16581 )
2026-01-14 07:08:05 +00:00
Christian Gunderman
8030404b08
Behavioral evals framework. ( #16047 )
2026-01-14 04:49:17 +00:00
