Model Evaluator

What it does

Model Evaluator provides a structured eval harness that runs your LLM outputs through configurable quality dimensions. Each dimension produces a score from 0 to 10 with a short reasoning paragraph, flagged spans within the output text that contributed to deductions, and suggested rewrites for failing sections. The harness is fully reproducible: the same input, configuration, and model always returns the same scores, making it safe to use in CI pipelines or automated regression testing for AI-powered features.

The skill supports both single-response evaluation and batch evaluation from CSV or JSONL files, which enables you to assess hundreds of outputs in a single run and export results as a JSON report, a CSV matrix, or a formatted Markdown comparison table ready to paste into a PR review or stakeholder report. Side-by-side multi-model comparison mode runs each test case through up to five models simultaneously and ranks them by aggregate score, giving you an objective basis for model selection decisions.

Capabilities

• Accuracy scoring against ground truth — compares model output to a reference answer and scores factual alignment on a 0-10 scale
• Hallucination span detection — identifies specific text spans that contain unsupported claims not present in the source material
• Toxicity and bias checks — runs outputs through safety classifiers and reports any flagged content with confidence scores
• Custom rubric evaluation — define your own dimensions with weighted criteria using a YAML rubric file
• Side-by-side multi-model comparison — evaluate the same prompt across Claude, GPT-4o, Gemini, and others in one command
• Batch eval from CSV or JSONL — process hundreds of test cases in a single run with progress reporting
• Exportable results — output to JSON, CSV, or Markdown table format for reporting and CI integration

How to install

skills add model-eval

Configuration

Add the following to your .claude/skills.json to set evaluation dimensions, output format, and the pass/fail threshold score:

{
  "model-eval": {
    "dimensions": ["accuracy", "hallucination", "coherence"],
    "output": "json",
    "threshold": 7.0
  }
}

Example

Provide a model output and a reference answer; the skill returns a structured eval report:

Evaluate this model output against the reference answer

Example output — structured eval JSON with per-dimension scores:

{
  "eval_id": "eval-20260530-001",
  "verdict": "PASS",
  "aggregate_score": 7.8,
  "dimensions": {
    "accuracy": {
      "score": 8.5,
      "reasoning": "Core facts align with reference. One minor date discrepancy.",
      "flagged_spans": ["launched in 2021"]
    },
    "hallucination": {
      "score": 9.0,
      "reasoning": "No unsupported claims detected.",
      "flagged_spans": []
    },
    "coherence": {
      "score": 6.0,
      "reasoning": "Paragraph 3 introduces a topic without prior context.",
      "flagged_spans": ["Furthermore, the integration allows..."]
    }
  },
  "threshold": 7.0,
  "pass": true
}