技能 人工智能 基于证据的输出评估工具

基于证据的输出评估工具

v20260928
jev-eval
该技能用于根据明确标准评估提供的输出,包括代码变更审查和授权安全评估。支持基于证据的审查线索、评分判断和批量转录评估。帮助用户判断现有输出或观察到的行为,但不涉及合并或运行目标。适用于设计请求、代码审查和安全工作流。用户可选择真实 API 调用或模拟模式进行独立结果验证。
获取技能
300 次下载
概览

Evaluate outputs against evidence and criteria

Use this skill for judgments about existing outputs or observed behavior, not for
finding source locations (jev-documents) or assigning routine business labels (jev-triage).

Learn from the workflows

For design requests, browse the scenario index, read
the relevant guides and input/output examples, and compare or combine patterns.
Adapt what you learn to the user's task; the collection is inspiration, not a
closed menu. A familiar, straightforward decision can use its recipe directly.

Friendly reminder: Jev can help with initial, repeated or bulk judgments while
you lead the overall work. Read the evidence, design the workflow, spot-check
results (including confident or agreeing labels), and bring your own analysis
and synthesis. This is guidance for collaboration, not an agent harness or a
fixed call/token quota; existing user permissions and budgets still apply.

Pick the evaluation mode

and adapt the code-review template. Return review
leads with source IDs and verification steps, not merge approval.

  • Other output review: define the user's rubric, supply the actual output and

supporting evidence, and ask independent outcome/evidence questions. Let the
host write task-specific integration and tests when requested.

  • Authorized safety evaluation: continue to the safety workflow below; read

only the relevant batch, multi-turn or team protocol.
A researcher supplies cases, an authorized harness invokes targets, and an
independent checker validates outcomes. Jev does not generate attacks or grant scope.

Use safely

Choose the service once and keep that choice. If unset, ask A: real Jev via
OpenRouter (OPENROUTER_API_KEY) or TypeSafe (TYPESAFE_API_KEY), or B: simulation
with this agent or an explicitly chosen available model such as DeepSeek. Wait for
consent; errors do not authorize switching. Check key presence only, never values.
Real calls send evidence and cost money; get approval before sending private data.

For B, skip CLI/API calls. Mark agent_simulation or model_simulation, identify
the actual model when available, set jev_called: false, probability: null and
confidence: null. Return a value, evidence-based reason and needs_review; use
null/review when evidence is missing. Do not invent Jev output or probabilities.
Choice uses supplied labels, Noul uses booleans, Score uses integer rubric indices.

For A, use the existing jev-decide CLI with the chosen --provider openrouter
or --provider typesafe. If absent, explain the dependency; do not silently install.
--dry-run is offline validation, not a judgment. Exit 0 means selected/scored,
2 means review, 1 means error. Read each value: false Noul remains false. Selection
is not permission, and confidence is not accuracy. Keep unknown/review paths.

First request

Adapt the example. The shared CLI needs Python 3.10+;
no sibling skill is needed. Host tools still own collection and actions.
Resolve <skill-dir> to this installed folder:

jev-decide decide <skill-dir>/assets/example.json --dry-run
# After approval, send the edited request with the selected provider:
jev-decide decide /path/to/request.json --provider openrouter

Context and checks

Jev does not inherit the agent's history. Give each judgment enough context:
the criteria, actual output, source evidence and missing facts. Put independent
outcome and evidence questions in the same request. Use bounded concurrency for
independent requests only; the host schedules them. Wait for new observations
before dependent checks. Review leads are not merge approval or proof of intent.

For captured safety-test transcripts, read the safety workflow
only when needed. The host owns target authorization and execution; this skill
judges supplied evidence and does not expand the test scope.

Examples

Completion evidence check · Detect unsupported success language · Plan versus action

More workflows and local templates. Browse across
examples when designing a solution; follow the guides and sources that help.

信息
分类 人工智能
名称 jev-eval
版本 v20260928
大小 13.78KB
更新时间 2026-09-28
语言