Evaluate outputs against evidence and criteria

Use this skill for judgments about existing outputs or observed behavior, not for finding source locations (jev-documents) or assigning routine business labels (jev-triage).

Learn from the workflows

For design requests, browse the scenario index, read the relevant guides and input/output examples, and compare or combine patterns. Adapt what you learn to the user's task; the collection is inspiration, not a closed menu. A familiar, straightforward decision can use its recipe directly.

Friendly reminder: Jev can help with initial, repeated or bulk judgments while you lead the overall work. Read the evidence, design the workflow, spot-check results (including confident or agreeing labels), and bring your own analysis and synthesis. This is guidance for collaboration, not an agent harness or a fixed call/token quota; existing user permissions and budgets still apply.

Pick the evaluation mode

• Code review: read diff and test-evidence review and adapt the code-review template. Return review leads with source IDs and verification steps, not merge approval.
• Other output review: define the user's rubric, supply the actual output and supporting evidence, and ask independent outcome/evidence questions. Let the host write task-specific integration and tests when requested.
• Authorized safety evaluation: continue to the safety workflow below; read only the relevant batch, multi-turn or team protocol. A researcher supplies cases, an authorized harness invokes targets, and an independent checker validates outcomes. Jev does not generate attacks or grant scope.

Use safely

Choose the service once and keep that choice. If unset, ask A: real Jev via OpenRouter (OPENROUTER_API_KEY) or TypeSafe (TYPESAFE_API_KEY), or B: simulation with this agent or an explicitly chosen available model such as DeepSeek. Wait for consent; errors do not authorize switching. Check key presence only, never values. Real calls send evidence and cost money; get approval before sending private data.

For B, skip CLI/API calls. Mark agent_simulation or model_simulation, identify the actual model when available, set jev_called: false, probability: null and confidence: null. Return a value, evidence-based reason and needs_review; use null/review when evidence is missing. Do not invent Jev output or probabilities. Choice uses supplied labels, Noul uses booleans, Score uses integer rubric indices.

For A, use the existing jev-decide CLI with the chosen --provider openrouter or --provider typesafe. If absent, explain the dependency; do not silently install. --dry-run is offline validation, not a judgment. Exit 0 means selected/scored, 2 means review, 1 means error. Read each value: false Noul remains false. Selection is not permission, and confidence is not accuracy. Keep unknown/review paths.

First request

Adapt the example. The shared CLI needs Python 3.10+; no sibling skill is needed. Host tools still own collection and actions. Resolve <skill-dir> to this installed folder:

jev-decide decide <skill-dir>/assets/example.json --dry-run
# After approval, send the edited request with the selected provider:
jev-decide decide /path/to/request.json --provider openrouter

Context and checks

Jev does not inherit the agent's history. Give each judgment enough context: the criteria, actual output, source evidence and missing facts. Put independent outcome and evidence questions in the same request. Use bounded concurrency for independent requests only; the host schedules them. Wait for new observations before dependent checks. Review leads are not merge approval or proof of intent.

For captured safety-test transcripts, read the safety workflow only when needed. The host owns target authorization and execution; this skill judges supplied evidence and does not expand the test scope.

Examples

Completion evidence check · Detect unsupported success language · Plan versus action

More workflows and local templates. Browse across examples when designing a solution; follow the guides and sources that help.