Credenva Assessment Methodology

Model benchmarks evaluate models. Developer evaluation platforms evaluate applications during development. Neither answers the executive question of whether a specific AI worker behaves reliably under the conditions it will actually face.

• Deployment-first. We assess the system a customer interacts with, not the model in isolation.
• Evidence over opinion. Every score is backed by captured transcripts, tool calls and retrieved context.
• Vendor-neutral. The same scenarios, rubrics and reviewers apply to every AI worker in a given role.

Scenarios are version-controlled test cases authored with domain experts for a specific role — for example, an AI Customer Support Representative. Each scenario specifies the customer intent, the business context, the systems of record involved and the acceptable outcome space.

• Common flows: account, billing, order status, entitlement.
• Edge cases: ambiguous intent, conflicting records, partial data.
• Adversarial cases: policy pressure, jailbreak attempts, PII elicitation.

Every scenario ships with a documented expected outcome and a set of acceptable variants. Ground truth is authored by role-specific experts, reviewed by a second expert, and locked to a version hash before any AI worker is scored against it.

When an AI worker produces a novel-but-defensible resolution, reviewers may propose an amendment; amendments are applied only in subsequent versions and never retroactively.

Deployments receive a 0–100 score on each dimension: Task Accuracy, Resolution Effectiveness, Policy & Compliance, Groundedness, Escalation Judgment and Reliability. The Credenva Readiness Score™ is a weighted aggregate; weights are published per role.

An AI worker is marked Production Ready only when the aggregate meets the role threshold and no critical failures are recorded.

Certain outcomes are treated as disqualifying regardless of overall score. These include unauthorized disclosure of personal data, fabricated policy or entitlement claims, silent policy violations, and failure to escalate when a documented trigger is met.

A single critical failure prevents a passing verdict for the version under test. The full list is published in the role standard (see the Customer Support Standard).

Assessments must be reproducible. We record the AI worker version, model identifiers, retrieval index snapshots, tool schemas, prompt templates and environment configuration used during the run.

Re-running the same scenario set against the same AI worker version must yield equivalent verdicts within a documented variance band; exceeding that band invalidates the run.

Automated grading proposes verdicts; qualified human reviewers ratify them. Every critical failure, every escalation judgment and every scenario flagged as ambiguous is reviewed by at least two reviewers, with disagreements resolved by a senior adjudicator.

Reviewer identities, qualifications and inter-rater agreement statistics are published alongside each release.

Methodology changes are proposed as versioned amendments, opened for comment, and ratified by the methodology council before taking effect. Vendors under assessment do not sit on the council for their own role.

Conflicts of interest, funding sources and reviewer affiliations are disclosed in every report.