GAIA基准测试失败诊断工具

v20260707

gaia-debugging

当AI智能体在GAIA基准测试中出现失败或错误答案时，本技能提供系统化的诊断流程。用户可以加载任务执行的完整追踪记录，根据症状（如推理错误、工具缺失等）科学分类失败模式，并指导用户应用针对性的修复措施，从而解决性能回归或错误问题。

AI 调试基准测试根因分析智能体改进

获取技能

421 次下载

概览

GAIA Debugging Skill

When a GAIA question fails, systematically diagnose the root cause and propose a targeted fix.

When to use

A specific task_id returns the wrong answer or times out
Pass-rate dropped between two runs and you need to find the regression
You want to understand why a particular question class is consistently failing

Failure mode taxonomy

Code	Mode	Symptom	Fix direction
TG	Tool Gap	Agent lacks a required tool (no image OCR, no PDF reader)	Add tool to catalogue
RM	Reasoning Miss	Agent has the right data but draws wrong conclusion	Improve system prompt, add CoT instruction
EB	Extraction Bug	Answer is in the trace but `FINAL_ANSWER:` regex fails	Fix answer extraction pattern
LI	Loop Issue	Agent loops (re-asks same tool call) and hits turn limit	Increase max-turns or add loop-detection
DS	Dataset Shift	Ground truth differs from what web currently shows	Flag for HAL dataset audit
AT	API Timeout	Tool call times out; agent never gets the result	Increase per-turn timeout

Diagnostic workflow

Step 1 — Load the question trace

# Find the result for the task_id in the latest run
RESULTS=~/.cache/ruflo/gaia/results-latest.json
node -e "
  const r = JSON.parse(require('fs').readFileSync('$RESULTS'));
  const q = r.results.find(x => x.task_id === '$TASK_ID');
  console.log(JSON.stringify(q, null, 2));
"

Step 2 — Classify the failure

Look at the trace output:

No tools called at all → RM or configuration issue
Tool called but returned error → TG or AT
Tool returned data, wrong answer → RM or EB
Correct answer in trace but marked wrong → EB
max-turns hit → LI or question too hard for current model

Step 3 — Re-run with extended logging

node v3/@claude-flow/cli/bin/cli.js gaia-bench run \
  --level 1 --limit 1 \
  --task-id $TASK_ID \
  --models claude-sonnet-4-6 \
  --max-turns 20 \
  --output json

Step 4 — Apply targeted fix

Failure	Action
TG — missing web_browse	Verify `gaia-tools/index.ts` exports `web_browse`; check tool registration
TG — missing image OCR	Add `image_describe` tool call; verify `GOOGLE_AI_API_KEY`
RM — reasoning	Add a system prompt instruction: "Before answering, list all facts you have gathered"
EB — extraction	Test the `FINAL_ANSWER_RE` regex against the trace manually
LI — loop	Add a tool-call deduplication guard in `gaia-agent.ts`
AT — timeout	Set `DEFAULT_PER_TURN_TIMEOUT_MS` higher or use `--max-turns` flag

Step 5 — Verify fix and store pattern

# Re-run the single question
node … gaia-bench run --task-id $TASK_ID --models $MODEL --output json

# If now passing, store the pattern
npx @claude-flow/cli@latest memory store \
  --namespace gaia-debug-patterns \
  --key "fix-$FAILURE_CODE-$(date +%Y%m%d)" \
  --value "task_id=$TASK_ID, mode=$FAILURE_CODE, fix=$FIX_DESCRIPTION"

Quick reference: tool catalogue check

node -e "
  const { createDefaultToolCatalogue } = require('./v3/@claude-flow/cli/src/benchmarks/gaia-tools/index.js');
  const cat = createDefaultToolCatalogue({});
  console.log('Tools registered:', cat.definitions.map(t => t.name));
"

Expected: web_search, file_read, web_browse, image_describe, python_exec

Pattern storage

After resolving a debugging session, store the finding:

npx @claude-flow/cli@latest memory store \
  --namespace gaia-debug-patterns \
  --key "session-$(date +%Y%m%d-%H%M)" \
  --value '{"task_id":"$TASK_ID","failure_mode":"$CODE","fix":"$FIX","verified":true}'

Search for similar past failures:

npx @claude-flow/cli@latest memory search \
  --namespace gaia-debug-patterns \
  --query "extraction bug final answer regex"

信息

Category 人工智能

Name gaia-debugging

版本 v20260707

大小 4.29KB

Source ruvnet/ruflo

更新时间 2026-07-09