Evaluations
Full Datasets/Cases/Suites/Runs/Reviews CRUD, real suite execution, cancel/resume, statistical + LLM-judge pairwise comparison, and standalone scoring — all via API key.
The full Evaluation Studio surface via API key — the same resource the dashboard uses, sharing one underlying service, so a suite or dataset created here shows up in the dashboard immediately and vice versa. A Suite binds a Dataset to one specific intent; running a suite calls that intent for real and scores what it produces. If you already generated a response yourself, score it directly with POST /v1/evals/score or POST /v1/evals/runs instead — no intent execution involved.
Every run is async — creation endpoints return a run immediately in pending status; poll GET /v1/evals/runs/{id} for progress and results.
Datasets
GET /v1/evals/datasets
POST /v1/evals/datasets # optionally with inline cases
GET /v1/evals/datasets/{id} # includes every case
PATCH /v1/evals/datasets/{id}
DELETE /v1/evals/datasets/{id}
POST /v1/evals/datasets/{id}/import # JSON body {cases:[...]} — no multipart
POST /v1/evals/datasets/{id}/cases # add one case
PATCH /v1/evals/cases/{caseId}
DELETE /v1/evals/cases/{caseId}
GET /v1/evals/datasets/templates # shared starter content
POST /v1/evals/datasets/templates/{id}/clone # clone one into your tenantPOST https://api.liyaengine.ai/v1/evals/datasets
Authorization: Bearer liya_xxxxxxxxxxxx
Content-Type: application/json
{
"name": "Permit renewal support cases",
"cases": [
{ "input": { "question": "How long does renewal take?" }, "expected_output": "5 business days" }
]
}POST .../{id}/import is the API-key counterpart to the dashboard's file-upload import — send an already-structured JSON array directly ({ "cases": [...] }), no CSV support here (convert client-side if needed).
Suites
GET /v1/evals/suites?domain_key=&intent_key=
POST /v1/evals/suites
PATCH /v1/evals/suites/{id}
DELETE /v1/evals/suites/{id}POST https://api.liyaengine.ai/v1/evals/suites
Authorization: Bearer liya_xxxxxxxxxxxx
Content-Type: application/json
{
"name": "Order Status Suite",
"domain_key": "support",
"intent_key": "order-status",
"dataset_id": "ds_01HZ..."
}domain_key/intent_key are fixed at creation — changing what a suite tests means creating a new suite (and a new baseline), not editing this one and losing its run history. A suite carries at most one custom scorer (custom_scorer_expression or custom_scorer_webhook_url + custom_scorer_webhook_secret, not both) — see Evaluation Studio for the expression grammar and webhook contract.
custom_scorer_webhook_secret is write-only. Every read (GET, list) only ever returns custom_scorer_webhook_secret_set: true|false — the plaintext is never round-tripped after you submit it.
Running a suite
POST /v1/evals/suites/{id}/run # { model? } — 202 Accepted
GET /v1/evals/suites/{id}/runs # alias of GET /v1/evals/runs?suite_id=
POST /v1/evals/suites/{id}/baseline # { run_id: string | null } — pin/clear
POST /v1/evals/suites/{id}/compare-models # { model_a, model_b } — starts two runsCalls the suite's intent for real — engine.execute() per case, then the same deterministic-gate-then-judge sequence Evaluation Studio uses. Rate limited and gated against your tenant's monthly Evals budget (429 EVALS_BUDGET_EXCEEDED with a details object once exhausted).
Runs
GET /v1/evals/runs?suite_id=
GET /v1/evals/runs/{id} # poll status + results
POST /v1/evals/runs/{id}/cancel # cooperative
POST /v1/evals/runs/{id}/resume # only a failed run
GET /v1/evals/runs/{id}/compare?against= # statistical
POST /v1/evals/runs/{id}/compare-pairwise # LLM judge, head-to-headresume is idempotent — already-scored cases are skipped, so a resumed run continues from where it stopped rather than re-billing every case from scratch. It works whether the run called a real intent or scored a caller-supplied output (see Evaluation API for the latter) — for the latter, the original submission is retained internally so a crashed run has something to resume from.
{
"success": true,
"data": {
"run": {
"id": "run_01HZ...",
"status": "completed",
"mean_score": 4.5,
"dimension_scores": { "llm_coherence": 4.5, "llm_actionability": 4.2 },
"cases_total": 5,
"cases_passed": 5
}
}
}Reviews
POST /v1/evals/results/{resultId}/reviews # { verdict: "agree"|"override"|"flag", corrected_score?, note? }
DELETE /v1/evals/reviews/{id}Record agreement or an override against any judge-scored result, independent of who triggered the run. corrected_score (1-5) is required when verdict is override.
Using an SDK
import { LiyaEngine } from '@liyaengine/sdk';
const client = new LiyaEngine({ apiKey: process.env.LIYA_API_KEY! });
const dataset = await client.evaluations.datasets.create({
name: 'Support Replies',
cases: [{ input: { message: 'Where is my order?' } }],
});
const suite = await client.evaluations.suites.create({
name: 'Order Status Suite',
domain_key: 'support',
intent_key: 'order-status',
dataset_id: dataset.id,
});
const run = await client.evaluations.suites.run(suite.id);
const result = await client.evaluations.runs.get(run.id);