Evaluate completed answers

Basic command

lladar eval responses.jsonl --output evaluation.json

The default method compares the expected answer with the Agent's actual response and saves a judgment for each completed trial.

Why save individual judgments?

Each judgment explains one result. Python then counts those saved results to calculate totals and rates. That keeps the summary consistent when you recreate a report.

Use a custom evaluation Skill

Your SKILL.md can say what matters for your domain. A support policy Skill, for example, can check refund deadlines, required caveats, and unsupported promises.

lladar eval responses.jsonl --skill skills/policy-verdict --output evaluation.json

Inspect the result

evaluation.json contains the plan, per-question judgments, saved Skill information, and the calculated summary. An execution error shows that the run failed for that question; it does not show that the Agent gave a wrong answer.