# ECAI + DamageBDD: LLM Evaluation for Real Business Requirements Turn business requirements into repeatable LLM evaluations with reviewed datasets, explicit release gates, dashboard controls and inspectable evidence. Canonical HTML: ## A convincing answer is not an acceptance test An assistant can explain a policy confidently and still apply the wrong rule. It can produce valid JSON with the wrong values. It can propose a tool that looks useful but is outside its permissions. The question is not simply, "Which model sounds better?" It is: > Does this model satisfy the requirements of this workflow, and what evidence > supports that decision? **Introducing LLM evaluation for ECAI + DamageBDD, available as a developer preview.** The evaluation workspace brings reviewed datasets, repeatable model runs, explicit grading rules, human review and release gates into the dashboard. DamageBDD turns the acceptance decision into an executable scenario. ECAI provides the evaluation subsystem and versioned reference context. Together, they give teams a way to test model behaviour against their own requirements instead of treating a persuasive demonstration as a deployment decision. **Test the behaviour. Inspect the evidence. Gate the release.** ## Start with a requirement the business can explain Consider a fictional shop with a 30-day return window. A customer asks whether an item purchased 45 days ago is eligible for return. A useful evaluation contract is more specific than "the answer should be good." It asks for a JSON boolean named `eligible`, requires that boolean to be `false`, and checks that the response includes the approved policy's reference ID. Those are separate, inspectable assertions. A fluent explanation cannot compensate for the wrong eligibility decision. A valid citation ID cannot compensate for a failed value check. The team also decides whether the eligibility rule is critical. That is a business decision encoded in the suite, not a severity level invented by the model after it sees the result. This is the central idea: **make acceptance explicit before asking the model to perform the task.** ## One evaluation lifecycle, two complementary components The workflow moves from dataset import and review to a frozen suite, then to model execution, grading, comparison and a release decision. DamageBDD supplies the authenticated account boundary, node settings and Gherkin integration. ECAI supplies the evaluation workspace, reference adapter, persistent run ledger, bounded workers, graders and reports. A run records its selected suite, split, targets, repetitions and evaluation settings. The model receives the task messages and any selected reference context, not the evaluator's expected values, rubric anchors or reviewer votes. Repeatable evaluation does not mean that model responses become deterministic. It means that the requirements, inputs and recorded decisions can be inspected and compared. The reference data still needs to be correct, and the cases still need to represent the work that matters. ## Build a reviewed suite, not just a collection of prompts A suite carries annotation guidelines, reference records and cases with named dimensions. Cases belong to development, test or holdout splits and have group identifiers so related examples can be tracked together. The import path checks for exact prompt or group reuse across splits. Trading cases also have chronological separation checks. These are useful safeguards, not proof that a model has never encountered the material or that semantically similar examples cannot leak between partitions. Review is an explicit, authenticated action. Changing an expected value or a rubric means importing a new suite version, not rewriting an old result to make a release pass. Here is a complete, deliberately small source-format example. It uses a fictional policy and should be treated as an evaluator smoke test, not evidence of production accuracy. Import it through **Datasets & coverage**, review and approve it, and select `test` with ECAI reference mode for a reference-assisted run. ```json { "name": "Fictional shop: return eligibility", "version": "1", "description": "Synthetic acceptance example, not a production benchmark.", "annotation_guidelines": "Check the fictional policy, expected boolean and source ID. Review the critical flag before approval.", "references": [ { "id": "returns-v1", "title": "Fictional shop return policy", "text": "Returns are permitted within 30 days after purchase. After 30 days a return is not eligible.", "origin": "synthetic-fixture", "revision": "1" } ], "cases": [ { "id": "late-return", "group": "late-return", "domain": "dialogue", "dimension": "factual_grounding", "split": "test", "messages": [ { "role": "user", "content": "A purchase was made 45 days ago. Using the supplied fictional shop policy, reply only JSON with eligible and citations." } ], "query": "shop return policy days", "evidence_ids": [ "returns-v1" ], "checks": [ { "id": "eligibility", "type": "json_equal", "description": "The 45-day return is outside the approved window.", "critical": true, "path": [ "eligible" ], "expected": false }, { "id": "source", "type": "citation_ids", "description": "The policy reference ID must be present.", "path": [ "citations" ], "required": [ "returns-v1" ] } ] } ] } ``` The boolean comparison is type-strict: `false` is not the string `"false"`. The source check establishes that the required reference ID is present. It does not establish that every sentence in a longer answer follows from that source. Add the relevant checks or a reviewed semantic rubric for that claim. ## Use the right grader for the requirement The preview combines objective checks with structured human judgment. | Requirement | Evaluation mechanism | | --- | --- | | Exact output or required/prohibited text | Deterministic text assertions | | Required JSON keys, typed values and numeric ranges | Structured output checks | | Required reference IDs | Citation-ID validation against the suite | | Permitted tool proposals | An allowlist check over returned structured text | | Helpfulness or an explanation needing interpretation | Human rubric with pass, fail and indeterminate anchors | Human rubrics specify what each verdict means. When required reviews are missing or disagree, an otherwise passing machine result remains indeterminate. A human vote cannot waive a failed deterministic check. Tool-proposal evaluation is intentionally narrower than agent execution. Checking that a response proposes only `lookup_order` or `create_ticket` does not execute those tools or prove the resulting workflow is safe. Likewise, checking generated source text is not the same as compiling and testing it. Code execution and live-agent validation require additional execution paths. ## Run the workflow from the dashboard The ECAI console adds an **LLM evaluation** workspace with five panels. | Panel | What the operator can do | | --- | --- | | Runs & results | Submit reviewed suites, inspect target scorecards and failures, compare runs, cancel work and export evidence | | Datasets & coverage | Import JSON, inspect source definitions and coverage, and approve annotations | | Review queue | Inspect authorized review tasks and record rubric judgments with notes | | Account settings | Choose account preferences and inspect effective limits | | Node policy | Authorized administrators manage the shared node evaluation controls | The DamageBDD dashboard also exposes evaluation preferences in account settings and an LLM evaluation group in node administration. These views use the same underlying settings rather than creating competing policy stores. ### The node sets the boundaries Administrators choose permitted configured model targets, whether remote inference is allowed, concurrency, request allowances, token and byte limits, timeouts, review requirements and retention. Evaluation starts disabled, with remote inference disabled and no allowed target IDs. Installing the feature does not itself authorize model calls. Provider addresses and credentials remain in operator-managed configuration; accounts cannot submit arbitrary destinations through the evaluation form. ### The account chooses within those boundaries A managed account selects its targets, repetitions, output-token preference, daily request allowance, pass threshold and reference mode within node policy. Remote inference requires both operator permission and account consent. These controls limit evaluation work, but they are not a monetary spending guarantee. Request reservations are not provider-confirmed charges, and the preview does not import authoritative billing data. It should not display an invented dollar saving as a measured result. The evaluation ledger stores plaintext records with restricted filesystem permissions; it is not the encrypted private-knowledge vault. Begin with sanitized material, and review storage, backups, retention and remote-data permissions before introducing sensitive cases. ## A release gate is more useful than a flattering average The primary pass ratio uses **all planned trials** as its denominator. Errors, uncertain requests, incomplete work and unresolved reviews remain visible. They are not quietly removed to improve the score. A release gate requires completed pass/fail outcomes, no critical failures, and the configured pass threshold for **every selected target**. A strong model cannot conceal a weak model behind a pooled average. A high overall score cannot cancel a critical failure. DamageBDD makes that decision executable: ```gherkin Feature: Reviewed model acceptance Scenario: Require an evidence-backed evaluation gate When I run the ECAI evaluation suite "REPLACE_WITH_SUITE_HASH" on split "test" with key "REPLACE_WITH_UNIQUE_RUN_KEY" And I wait up to "120" seconds for the ECAI evaluation Then the ECAI evaluation release gate must pass ``` Replace the placeholders with the reviewed suite ID and a deliberate run key. The authenticated runner supplies the managed-account bearer through its server-side context; never put credentials in the feature file. Reuse the same key and submission body when retrying an uncertain submission. Use a new key for an intentionally new evaluation. The supplied steps submit, wait and assert the gate; they do not automatically deploy a release. ## Compare the model with and without reference context The reference adapter uses ECAI's shared term-processing pipeline to select context from imported, frozen reference texts. Selection uses lexical term overlap with stable ID tie-breaking. It is not an automatic connection to every existing ECAI index, nor a mechanism for recovering arbitrary text from a cryptographic point. That creates a practical experiment: run the same suite, split and model with reference mode `none`, then with `ecai`. Keep the remaining generation and review settings comparable, and inspect which cases change. Repeated trials expose variation. Per-dimension results help separate a formatting problem from a policy failure or a missing reference. Paired comparisons keep the case identities and differing configurations visible. Repeated attempts are not independent new business examples, and a higher score alone does not establish statistical significance or a general capability improvement. Failure details and rule-based recommendations can inform the next change to a prompt, dataset, reference collection or model choice. Recommendations are diagnostic starting points, not proven root causes or automatic repairs. Keep a holdout for deliberate promotion decisions rather than repeatedly tuning against it. ## Financial evaluation stays in paper replay The preview includes a bounded, single-decision paper-trading simulator. Cases define an as-of decision time, cash and position assumptions, permitted actions, quantity and notional limits, latency, fees, slippage and a recorded bid/ask sequence. The model sees the decision-time context, not the future quote sequence used to assess the proposal. The evaluator checks the risk contract and records the simulated outcome. Profit does not automatically turn an invalid action into a passing answer. There are **no live orders, exchange credentials or wallet mutations** in this path. It does not model a complete exchange, market impact, partial fills or a multi-period trading strategy. Its purpose is to test declared decision constraints under stated assumptions, not to certify profitability. Dialogue evaluations likewise replay fixed message histories. Adaptive conversations and autonomous tool loops are separate evaluation work. ## Preserve the evidence behind the score The persistent ledger records planned work and request reservations before provider dispatch. Pending work can resume under current policy. A lost in-flight result becomes uncertain instead of being automatically reissued and potentially charged again. This is conservative restart handling, not exactly-once execution across a network. A provider may have finished a request whose response never reached the evaluator. Exercise cancellation, restart and storage-failure cases on the actual deployment before relying on recovery behaviour. An exported receipt retains the suite, references, admitted configuration, trial messages, results, checks, reviews and report. Its JSON manifest has a SHA-256 digest for byte-integrity checking. **Current receipts are unsigned.** A digest delivered with the same file is not independent authentication: someone who changes the file can replace its digest too. Compare against a digest obtained through a separately trusted channel when that correspondence matters. The portable verifier checks the receipt bytes; it does not rerun the graders or attest hosted model weights. Blockchain anchoring, evaluation NFTs, independent signatures and universal model certification are not part of this preview. The useful deliverable is an inspectable record of a specific evaluation and its limits. ## Start with one workflow you already understand Use the shipped synthetic dialogue or paper-replay examples to check the wiring on a dedicated permitted local target. Then replace them with a small, reviewed suite for a real workflow whose expected behaviour the team can explain. Exercise a known pass, a known critical failure and a case requiring human review before making the gate part of a release process. On a source checkout containing the feature, the operator guide is `docs/ecai/llm_evaluation.org`. Example configuration lives at `apps/ecai/priv/config/eval.example.config`, and the executable acceptance scenario is under `apps/damage/priv/features/evaluation/`. Validate the matching application build and focused tests, then run the complete regression suite. Live authorization, real-provider behaviour, privacy and crash recovery need their own checks. This article describes the implementation; it does not report a production benchmark or a verified deployment. For the surrounding stack, start with the [installation guide](../install.md), [node administration](../node_admins.md), [DamageBDD acceptance evidence](damagebdd_evidence_workflows.md) and [ECAI engineering memory](ecai_engineering_memory.md). The goal is not to replace judgment with a single number. It is to make the requirements clear, the failures visible and the deployment decision easier to defend. **Test the behaviour. Inspect the evidence. Gate the release.**