Before you begin
- Python 3.11 or 3.12.
- Credentials for the LLaDAR model provider, in
.envor the process environment. - A clone of this repository for the included demo Agent.
The demo Agent itself needs only Python. Dataset generation, evaluation, and reporting use your configured provider.
1. Install LLaDAR and obtain the demo
git clone https://github.com/stoday/LLaDAR.git
cd LLaDAR
python -m pip install -e .2. Create questions from the support policy
lladar create test-dataset --knowledge examples/support-demo-agent/knowledge.md --output demo-dataset.jsonlThis creates questions and expected answers based on the policy.
3. Run the demo Agent
lladar run-agent demo-dataset.jsonl --project examples/support-demo-agent --no-interactive --output demo-responses.jsonlLLaDAR finds the demo Agent's public entry point, checks that it can use it, and records the responses.
4. Evaluate answers and create a report
lladar eval demo-responses.jsonl --output demo-evaluation.json
lladar report demo-evaluation.json --output demo-report.mdWhat success looks like
| File | What it shows |
|---|---|
demo-dataset.jsonl | The questions and expected answers. |
demo-responses.jsonl | The Agent's answer for each question. |
demo-responses.jsonl.run.json | How LLaDAR reached the Agent and whether the replay worked. |
demo-evaluation.json and demo-report.md | Per-question judgments and a readable summary. |
Important: a successful replay proves that the Agent entry point was exercised. Use the saved evaluation and report to assess answer quality.