Evaluation API

The Evaluation API is Evaluation Studio's scoring engine, reachable standalone. If your model or pipeline already runs outside LiyaEngine, you don't need to migrate it in to get real scoring — POST your own (input, output) pairs and get back the same deterministic-check-then-LLM-judge sequence the dashboard uses.

API reference

POST /v1/evals/score              # score one pair synchronously
POST /v1/evals/datasets            # create a dataset, optionally with inline cases
POST /v1/evals/runs                 # submit a batch for async scoring
GET  /v1/evals/runs/:id              # poll status + results

These four are the standalone scoring surface — no dataset, suite, or intent required beyond what you already have. All require an API key (see Authentication):

Authorization: Bearer liya_...

Scoring a single pair

No dataset, suite, domain, or intent needed — just the input your system used and the output it produced.

POST https://api.liyaengine.ai/v1/evals/score
Authorization: Bearer liya_...
Content-Type: application/json

{
  "input": { "question": "What documents do I need to renew a business permit?" },
  "output": "You need a valid ID, your previous permit, and proof of tax payment.",
  "expected_output": "proof of tax payment"
}

expected_output is optional — a string checks case-insensitive substring containment, an object checks partial deep-equality (every key in expected_output must be present and equal in output). When set, it's checked first and gates the judge call: a failing deterministic check skips the LLM call entirely, so you're never billed for a judge score on a case you already know failed. custom_scorer_expression (the same boolean-expression grammar Evaluation Studio suites use) is accepted too, checked alongside expected_output.

Response:

{
  "success": true,
  "data": {
    "passed": true,
    "score": 4,
    "checks": [
      {
        "check_type": "llm_coherence",
        "passed": true,
        "judge_score": 4,
        "judge_rationale": "The response is clear and lists specific documents required."
      },
      {
        "check_type": "llm_actionability",
        "passed": true,
        "judge_score": 4,
        "judge_rationale": "The response gives the user a concrete list to act on."
      }
    ]
  }
}

score is the mean of the judge's dimension scores — null if a deterministic check failed before the judge ran, since there's nothing to average.

Batch scoring

Create a dataset once, then submit outputs against it any time you want to re-score the same cases (e.g. after a prompt or model change):

POST https://api.liyaengine.ai/v1/evals/datasets
Authorization: Bearer liya_...
Content-Type: application/json

{
  "name": "Permit renewal support cases",
  "cases": [
    { "input": { "question": "How long does renewal take?" }, "expected_output": "5 business days" },
    { "input": { "question": "What is the renewal fee?" } }
  ]
}

POST https://api.liyaengine.ai/v1/evals/runs
Authorization: Bearer liya_...
Content-Type: application/json

{
  "dataset_id": "<dataset id>",
  "cases": [
    { "case_id": "<case id>", "output": "Renewal takes about 10 business days on average." },
    { "case_id": "<case id>", "output": "The renewal fee is 5,000 naira." }
  ]
}

Returns 202 { "data": { "run_id": "..." } } immediately — scoring runs in the background. Poll GET /v1/evals/runs/:id for status and per-case results, same shape as a single /score call plus run-level aggregates (mean_score, dimension_scores, cases_total, cases_passed).

No dataset yet? Submit cases fully inline instead — each one is persisted as a standalone case for the run's own record, without needing a reusable dataset first:

POST https://api.liyaengine.ai/v1/evals/runs
Authorization: Bearer liya_...
Content-Type: application/json

{
  "cases": [
    { "input": { "question": "Can I renew online?" }, "output": "Yes, through the city portal." }
  ]
}

A batch accepts up to 100 cases per request. custom_scorer_expression or custom_scorer_webhook_url (+ custom_scorer_webhook_secret) can be set once per batch, applied to every case in it — not both at the same time.

Limits

• /v1/evals/score is rate limited to 30 requests/minute per tenant, capped at $0.25 judge spend per call.
• /v1/evals/runs is rate limited to 10 requests/minute per tenant, capped at $2.00 judge spend per run (matching Evaluation Studio's own per-run cap) and 100 cases per request.
• Judge dimensions scored are the same defaults any custom intent gets in Evaluation Studio (coherence, actionability) — there's no per-call dimension override yet.

Related