Challenge
Prompt injection, direct and hidden in documents
Agnotic approach
Cases that try to override instructions from the user turn and from retrieved files, with the expected safe behaviour written for each.
AI / LLM Evaluation
LLM Evaluation Services for GenAI Quality and Trust
The demo worked. Then the model version changed, a prompt was edited, and nobody could say whether the feature got better or worse. Agnotic Technologies builds the evaluation sets, red-team tests and release gates that answer that question before every release and keep answering it after launch.
Eval sets from real tasksRed teamingRelease gatesProduction sampling
Trusted by global innovators
What is LLM evaluation?
LLM evaluation is the testing that shows whether a generative AI feature does its job: an evaluation set built from real tasks, scored for groundedness, accuracy, refusals and format, plus red-team tests for attacks. Agnotic Technologies builds it as a release gate that runs on every prompt edit and model change.
Without it, quality is an opinion. With it, a team can compare two prompts, two models or two retrieval settings on the same questions and see the difference in numbers it defined itself.
This page covers testing before release and monitoring after it. The runtime controls that block attacks in production are on our AI firewall and LLM guardrails page.
Evaluation set design
How do we build an evaluation set from your real tasks?
A generic benchmark says little about your feature. The evaluation set comes from the work your users actually do.
Collect real tasks
Questions, documents and requests from logs, support tickets or subject experts, with personal data removed or replaced.
Write expected answers
Subject experts write what a good answer must contain, which source it must cite and what it must never say.
Add the hard cases
Questions with no answer in the sources, out-of-scope requests, ambiguous wording and the edge cases users have hit before.
Split and freeze
A development set the team tunes against and a held-out set used only at release, so scores are not inflated by tuning.
Version with the product
The set is stored in the repository and grows with every new failure found in review or production.
Metrics
Which metrics does an LLM evaluation track?
Each metric is defined for the feature and scored by rules, a reviewing model checked against people, or human reviewers.
| Code | Name | Typical use |
|---|---|---|
| Groundedness | Is every claim supported by the retrieved sources? | Catches fabricated detail, the failure OWASP groups under misinformation. |
| Citation accuracy | Does each citation point to the passage that supports it? | Lets users check an answer quickly and trust the ones they cannot check. |
| Task success | Did the answer contain what the expected answer requires? | The main quality score, compared release to release. |
| Refusal correctness | Did it decline when it should, and answer when it should? | Over-refusal frustrates users as much as a wrong answer. |
| Format validity | Does output match the schema the next system expects? | Stops broken JSON or missing fields reaching downstream code. |
| Safety and policy | Did it stay within the content and data rules for the product? | Checked on every red-team case and a sample of normal ones. |
Red teaming
What does LLM red teaming test?
Red-team suites are test cases, run like any other test. We organise them by the risk categories in the OWASP Top 10 for LLM Applications.
Challenge
Leaking data or the system prompt
Agnotic approach
Cases that ask for other users' data, hidden instructions or secrets, scored as fail if any of it appears in the answer.
Challenge
Jailbreaks against content rules
Agnotic approach
Role play, encoding tricks and multi-turn pressure, refreshed as new patterns are published.
Challenge
Unsafe tool use
Agnotic approach
For features that call tools, cases that try to trigger actions the user is not allowed to take.
Challenge
Confident wrong answers
Agnotic approach
Questions the sources cannot answer, where the only passing answer is to say so.
Original checklist
What does the release gate check?
The release gate is the checklist a change must pass before it ships. Each check has a method, a pass rule an owner signs off and a schedule.
| Check | Method | Pass rule owner | When it runs |
|---|---|---|---|
| Task success on the held-out set | Automated scoring against expected answers | Product owner sets the minimum score | Every prompt change and every model version |
| Groundedness and citations | Reviewing model, calibrated against human labels | Subject expert lead | Every prompt change and every model version |
| Red-team suite | Automated attack cases plus manual probing | Security lead | Every model version, and monthly as new attacks appear |
| Format and schema | Rule-based validation of every output | Engineering lead | Every build |
| Regression on past failures | Every production failure added as a test case | Engineering lead | Every build |
| Production sample review | People score a random sample of live answers | Product owner | Weekly in production |
| Drift check | Score trends and input changes compared with launch | Product owner | Weekly in production |
Thresholds are set by your team for your feature. We record each one, its owner and the evidence behind every release decision.
How is a GenAI feature monitored after launch?
Evaluation does not stop at release. Production shows the questions no test set predicted.
Sampled human review
A random share of live answers scored by people each week with the same rubric as the test set.
Drift signals
Shifts in question topics, retrieval hit rates and score trends that suggest the feature is meeting new ground.
Failure capture
User flags and reviewer fails saved with their context and added to the regression set.
Model change watch
Provider model updates trigger the full gate before traffic moves to the new version.
Risk framework
How does evaluation map to the NIST AI RMF?
We record evaluation work against the four AI RMF functions and use NIST AI 600-1, the Generative AI Profile, to choose which risks to test.
Govern
Named owners for each pass rule and a written record of every release decision.
Map
The feature's users, tasks and harms listed before the evaluation set is written.
Measure
The metrics, red-team suites and production sampling described on this page.
Manage
Gates that block a release, rollback plans and fixes tracked to closure.
Compliance-First Healthcare App Development Services Backed by Global Standards
01
HIPAA
Health Insurance Portability and Accountability Act
Protect PHI with privacy-first architecture, encrypted storage and transmission, strict access controls, and traceable audit logs.
02
GDPR
General Data Protection Regulation
Implement lawful consent flows, data minimization, retention controls, and secure processing for sensitive health data.
03
FHIR
Fast Healthcare Interoperability Resources
Enable standardized health data exchange across apps, care teams, and systems through standards-based FHIR APIs.
04
HL7
Health Level Seven International
Support enterprise-grade interoperability with HL7-based integrations for records, events, and clinical messaging workflows.
05
HITRUST
Health Information Trust Alliance
Align security programs to healthcare-specific control and risk management practices trusted by providers and ecosystem partners.
06
HITECH
Health Information Technology for Economic and Clinical Health Act
Design with breach notification readiness, digital record safeguards, and operational controls that support regulated care programs.
07
SaMD
FDA Software as a Medical Device
Plan software quality, traceability, and documentation pathways for products that may require SaMD review and submission.
08
MDR (EU)
Medical Device Regulation (European Union)
Prepare EU market-ready processes for risk classification, evidence tracking, and lifecycle governance under MDR expectations.
09
SAMHSA
Substance Abuse and Mental Health Services Administration
Apply confidentiality controls and consent-aware sharing models for behavioral and mental health data experiences.
Where it applies
Which GenAI features need an evaluation suite?
Any feature where a wrong answer reaches a user or another system.
Document question answering
RAG assistants over policies, manuals or records, scored for groundedness and citations.
Summaries and drafts
Notes, letters and reports a person reviews before sending, scored for omissions.
Extraction to structured data
Fields pulled from documents into a system, scored for accuracy and format.
Agents that call tools
Assistants that take actions, tested for unsafe and unintended tool use.
Classification and routing
Tickets, messages or claims sorted by a model, scored per category.
AI product startups
Teams that need evidence of quality for customers and investors.
How we work
How does an LLM evaluation engagement work?
Three stages, starting with one feature.
1.
Scoping call
We pick one feature, agree what a good answer is and who owns each pass rule.
One feature
2.
Fixed-scope first release
An evaluation set, the scoring pipeline, a first red-team suite and the gate wired into your CI.
Fixed scope
3.
Scale and support
More features, production sampling and monthly red-team refreshes as models and attacks change.
Ongoing
Facts and sources
Which frameworks guide LLM evaluation?
Short facts behind our test design, each linked to its primary source.
NIST released NIST AI 600-1, the Generative Artificial Intelligence Profile of the AI Risk Management Framework, on 26 July 2024 to help organizations identify risks unique to generative AI and propose actions to manage them.
Source: NIST AI Risk Management FrameworkNIST AI 600-1 is a cross-sectoral companion resource to AI RMF 1.0 for generative AI, intended for voluntary use in the design, development, use and evaluation of AI products, services and systems.
Source: NIST AI 600-1The OWASP Top 10 for LLM Applications 2025 names ten risk categories, from LLM01 Prompt Injection to LLM10 Unbounded Consumption, including System Prompt Leakage (LLM07) and Misinformation (LLM09).
Source: OWASP GenAI Security Project, LLM Top 10OWASP describes misinformation as an LLM producing false or misleading information that appears credible, and names hallucination, content that seems accurate but is fabricated, as one of its major causes.
Source: OWASP LLM09:2025 Misinformation
Frequently Asked Questions
Evaluation tests a feature before release and samples it after, to measure quality. Guardrails are runtime controls in front of the model that screen each request and response. Most products need both, and the red-team suite tests whether the guardrails hold.
Enough to cover every task type, the hard cases and past failures, with a held-out part kept for release decisions. We size it with your subject experts for the feature and grow it with every failure found in production.
Yes, for some metrics, if it is checked. We calibrate a reviewing model against human labels on a sample, report where they disagree and keep people scoring the metrics where the reviewing model is not reliable.
Yes. The gate runs as a pipeline step in the CI system you already use, stores results next to the build and blocks a merge or deploy when a pass rule fails.
It depends on how many features you test, how quickly experts can write expected answers and how many models you compare. We agree the first release for one feature on the scoping call.
Yes. One feature, one evaluation set, one red-team suite and the release gate is a complete first release that proves the approach before you extend it.
We use AI tools to draft test cases and code, and people review every one. Expected answers are written or approved by your subject experts, never generated and accepted unread.