Evaluation Studio

Evaluation Studio scores your custom intents against a real dataset of cases — an LLM judge on a 1-5 scale, gated by any deterministic assertions you attach, with RAG-native faithfulness scoring whenever a call actually retrieved knowledge context. Everything runs from the dashboard against your own domain/intent configuration; no separate eval environment to stand up.

Datasets & cases

A dataset is a named set of cases — each case is an input object (the same shape your intent expects), an optional expected_output for a deterministic assertion, and an optional reference message for judge context.

GET    /dashboard/evaluations/datasets                    # list datasets
POST   /dashboard/evaluations/datasets                     # create
GET    /dashboard/evaluations/datasets/:id                  # get + cases
PATCH  /dashboard/evaluations/datasets/:id                   # update
DELETE /dashboard/evaluations/datasets/:id                    # delete
GET    /dashboard/evaluations/datasets/:id/export?format=json|csv
POST   /dashboard/evaluations/datasets/:id/import            # multipart JSON/CSV upload
POST   /dashboard/evaluations/datasets/:id/cases              # add a case
PATCH  /dashboard/evaluations/cases/:caseId                    # update a case
DELETE /dashboard/evaluations/cases/:caseId                     # delete a case
GET    /dashboard/evaluations/datasets/templates                # shared starter templates
POST   /dashboard/evaluations/datasets/templates/:id/clone        # clone a template into your tenant

Starting from a blank dataset works, but the templates gallery (three ready-made datasets covering support triage, RAG grounding, and structured-output accuracy) is the fastest way to see a real run before writing your own cases.

Suites

A suite binds one dataset to one domain + intent — that's what actually gets executed and scored.

POST   /dashboard/evaluations/suites          # { name, domainKey, intentKey, datasetId, customScorerExpression?, customScorerWebhookUrl? }
PATCH  /dashboard/evaluations/suites/:id        # update name/dataset/scorer (domain+intent are fixed at creation)
DELETE /dashboard/evaluations/suites/:id         # delete
GET    /dashboard/evaluations/suites?domainKey=&intentKey=

Custom scorers

A suite can carry one deterministic custom scorer, evaluated before the judge (same gate expected_output uses — a failing check skips the judge call entirely):

• Expression — a small boolean grammar over input/output/expected (comparisons, boolean logic, arithmetic, a whitelisted function set — no arbitrary code execution). Example: output.total == input.unit_price * input.quantity.
• Webhook — point at your own endpoint; it receives { input, output, expected_output } and returns { passed, score?, reason? }. Requests are HMAC-signed (X-Liya-Signature) the same way tenant action webhooks are.

Running an evaluation

POST /dashboard/evaluations/runs               # { suiteId, model? } — 202 Accepted, starts async
GET  /dashboard/evaluations/runs?suiteId=       # list runs
GET  /dashboard/evaluations/runs/:id             # poll status + per-case results
POST /dashboard/evaluations/runs/:id/cancel       # cooperative cancellation
POST /dashboard/evaluations/runs/:id/resume        # resume a failed run — already-scored cases are skipped, not re-billed

Each case runs through your intent, then:

1. Deterministic gate — expected_output (substring match for a string, partial deep-equality for an object) and any custom scorer. A failing check marks the case failed and skips the judge call.
2. LLM judge — 1-5 Likert scoring (temperature 0, JSON mode) on coherence and actionability for a custom intent, plus faithfulness, context_relevance, and answer_relevance automatically whenever the call actually retrieved knowledge context — independent of intent, so it applies to any RAG-grounded custom domain.
3. Judge spend is capped at $2.00 per run (soft-skips remaining checks past the cap rather than failing the run) — surfaced on the run as judge_total_cost_usd / judge_cap_reached.

Model comparison

Run the same suite through two different models and compare results side by side:

POST /dashboard/evaluations/suites/:id/compare-models     # { modelA, modelB } — starts two runs
GET  /dashboard/evaluations/runs/:id/compare?against=:id   # statistical comparison (t-test, Cohen's d)

Model selection respects your tenant's own BYOK credentials first (default provider, then your configured fallback chain) before falling back to the platform's own providers — so comparing against a model only your own Anthropic or Bedrock key supports works the same as comparing built-in models.

Baseline pinning & trend

Pin one completed run per suite as its baseline; every later run of that suite automatically gets a baseline_comparison on GET .../runs/:id once both are complete.

POST /dashboard/evaluations/suites/:id/baseline    # { runId: string | null }

Pairwise judging

Rather than two independent 1-5 scores, ask the judge to pick a winner directly for each shared case between two completed runs of the same intent:

POST /dashboard/evaluations/runs/:id/compare-pairwise    # { against: runId } — capped at $1.00 total

Human review

Record agreement or an override against any judge-scored result, independent of who triggered the run — supports multi-reviewer consensus (no limit on reviews per result).

POST   /dashboard/evaluations/results/:resultId/reviews   # { verdict: "agree" | "override" | "flag", correctedScore?, note? }
DELETE /dashboard/evaluations/reviews/:id

Related