An LLM evaluation pipeline that catches regressions before customers do
Every prompt tweak, model upgrade or retrieval change can quietly make some answers worse. We build an LLM evaluation pipeline that runs your test set in CI on every change using promptfoo or Langfuse, compares scores with the last release, blocks merges that fall below thresholds you set and posts a readable summary to the pull request and Slack. A typical build costs $1,000 to $2,500 and takes about four to five weeks.
The problem
Why manual LLM evaluation misses regressions
Most AI teams evaluate, but by hand: someone runs a notebook before a release, eyeballs a few outputs and shares numbers in a chat thread. When a customer reports that answers got worse, nobody can say which change caused it.
Model providers release new versions often, and switching can change tone, refusals, formatting and accuracy in ways a few spot checks will not catch. Without a fixed evaluation set and a record of every run, each upgrade is a gamble.
What we build
What our LLM evaluation pipeline includes
A versioned evaluation set
Real user questions, edge cases and known failures stored in the repository or Langfuse datasets, with expected behavior for each.
Scorers that fit the task
Exact checks for format and facts, similarity checks where wording varies, and model-graded rubrics where judgment is needed, each reviewed by your team.
Runs in CI on every change
A GitHub Actions or GitLab CI job that runs the set when prompts, model versions or retrieval code change.
Regression gates
Merges blocked when scores fall below the thresholds you set, with the failing cases listed.
Readable summaries
A pull request comment and Slack message showing what improved, what regressed and links to every failing output.
History of every run
Results stored so you can show which model, prompt and dataset produced which score on any date.
Who does what
Automation does the chasing, people make the calls.
-
01
Automated
Change proposed
A pull request changes a prompt, model version, retrieval setting or tool definition.
-
02
Automated
Evaluation runs
CI runs the evaluation set against the changed configuration and the current release.
-
03
Automated
Scores compared
Results are compared with thresholds and the last release, and a summary is posted.
-
04
Team review
Engineer reviews
An engineer reads failing cases and fixes the change or updates the test set with reasons.
-
05
Founder or model owner decides
Release decision
The model owner decides whether to ship, especially when scores trade off against each other.
How it behaves
How LLM regression testing behaves in CI
What customers see
- Fewer surprises when you upgrade a model or change a prompt
- Fixes for reported failures that stay fixed
- Release notes that describe behavior changes honestly
What engineers see
- A pull request comment with scores, regressions and failing cases
- A blocked merge when quality drops below thresholds
- A history of every run by model, prompt and dataset
What the automation does
- Runs the evaluation set on every relevant change
- Compares with the last release and applies gates
- Stores results and posts summaries to Slack
Edge cases and fallbacks
- Model provider rate limit during a run
- The run retries with backoff and reports incomplete rather than passing.
- Model-graded scorer disagrees with humans
- Flagged in calibration and the rubric or threshold is revised.
- Intended behavior change
- An engineer updates expected outputs with a written reason in the pull request.
- CI or evaluation store unavailable
- The gate fails closed and the team is alerted; nothing ships unevaluated by default.
- Flaky case with random output
- Run several times and scored on the pass rate, or marked for review.
Architecture
An LLM evaluation pipeline in CI at a glance.
- Input
- Logic and services
- Data
- AI step
- People
How it flows
- 1 A pull request changes a prompt, model version or retrieval setting.
- 2 CI loads the versioned evaluation set and runs it against both the change and the current release.
- 3 The regression gate compares scores with your thresholds and posts a summary to the pull request and Slack.
- 4 Results are stored in the run history, and the model owner decides whether to ship.
Project planner
How the project runs, phase by phase.
LLM evaluation pipeline & regression testing project plan
Typical time to launch: about 4 weeks, then support
-
01
Discovery
2 to 3 days
What the product must get right, current evaluation habits, models, CI and data sources.
-
02
Design
About 1 week
Evaluation set structure, scorers, thresholds, CI triggers and summary format.
-
03
Build
1 to 2 weeks
Dataset, scorers, CI job, gates, summaries and run storage.
-
04
Calibration
About 1 week
Run against recent releases, check model-graded scores against human judgment and tune thresholds.
-
05
Launch
1 to 2 days
Gate switched on for the main branch and the team walked through reading results.
-
06
Support
Ongoing
New cases from production failures, new models and scorer updates.
What it runs on
Real products only, connected through the access each vendor provides. See all integrations
Cost estimate
What it costs, line by line.
-
Discovery and design
Test set structure, scorers, thresholds and CI triggers.
$200 – $400
-
Build
Dataset, scorers, CI job, gates and summaries.
$500 – $1,300
-
Integrations
CI, model providers, Langfuse and Slack.
$150 – $400
-
Calibration
Scores checked against human judgment and past releases.
$100 – $250
-
Launch and training
Gate switch-on and team walkthrough.
$50 – $150
-
Total project estimate
$1,000 – $2,500
Optional monthly care after launch is quoted with your proposal. Fixed quote after a free call. Covers one product surface and one CI system. Writing a large labeled evaluation set from scratch is scoped separately. Model usage during runs is billed to your account.
Monthly running costs you pay directly
Estimates from the vendors' published pricing, checked November 2026. Billed to you by each vendor, never marked up.
- Model usage during runs: Billed by your providers per use; scales with test set size and how often it runs.
- Langfuse: Free when self-hosted (MIT-licensed core), or Langfuse Cloud at the vendor's own plan.
- CI minutes: Your GitHub or GitLab plan.
What pushes cost up
- Systems without an API, or with integration access that must be requested from the vendor
- Writing data back into your core system rather than only reading from it
- Several locations, brands or very different processes to support
- Custom apps, dashboards or a voice line on top of the workflows
What keeps cost down
- One location and one clear process to start with
- Cloud software with a documented API
- Using tools you already pay for instead of adding new ones
- Approved wording and sample data ready at the start
Privacy and consent
Rules we design for, with the sources.
EU: evaluation records for model providers
Article 53 of the EU AI Act requires general-purpose model providers to keep technical documentation including testing and evaluation results, applying from 2 August 2025; stored run history supports that record.
Source: artificialintelligenceact.euUS: NIST AI RMF
The NIST AI Risk Management Framework, released January 26, 2023 for voluntary use, and its Generative AI Profile from July 26, 2024 both treat measurement as a core activity.
Source: www.nist.govCustomer data in test sets
Evaluation cases drawn from production may contain personal data, so they are minimised or anonymised and stored with processors under GDPR Article 28 contracts.
Source: gdpr-info.euFree tools
Try these first, free.
How we deliver
Agile delivery, in 1 to 2 week sprints.
Every sprint
- 1 Sprint planning. At the start of each sprint we agree which backlog items we will finish, each with written acceptance criteria you can check.
- 2 Daily stand-up. A short internal check-in on progress and blockers. You get a written update in your Slack or Teams channel, not another meeting.
- 3 Sprint review and demo. At the end of each sprint we demo what was built on test data or in a shadow run, and you accept it or ask for changes.
- 4 Retrospective. The team looks at what went well and what to change in the next sprint, and we share anything that affects you.
How scope stays under control
- Product backlog
- Every feature, fix and question in one prioritised list you can see at any time.
- Acceptance criteria
- Plain-language conditions for "done" on each item, agreed before work starts.
- Change requests
- New ideas go into the backlog with an estimate. Nothing changes scope or price without your written approval.
- User acceptance testing (UAT)
- Before launch your team tests the real workflow against the acceptance criteria and signs off.
Tools we use
Project manager / Scrum master
Runs the sprints, keeps the backlog and timeline honest, and is your single point of contact.
Business analyst
Maps how the work runs today, writes the workflow specification and acceptance criteria with your team.
Automation engineer
Builds the n8n workflows, integrations, AI steps and error handling.
Backend developer
Writes custom code where a no-code step is not enough: APIs, data sync, secure storage.
Frontend / mobile developer
Builds forms, booking pages, dashboards, web apps or Android apps when the project needs them.
QA engineer
Tests every workflow against the acceptance criteria, including edge cases and failure paths.
DevOps engineer
Sets up hosting, access, backups, monitoring and alerts, and the deployment pipeline.
Who does what at each phase
- Free call
- Project manager, business analyst
- Understand the problem, check it is worth automating and outline a rough range.
- Discovery
- Business analyst, automation engineer
- Map the current process, systems and data, and agree success measures.
- Design
- Business analyst, automation engineer, backend developer
- Write the workflow specification, data flows, compliance notes and backlog with acceptance criteria.
- Sprints
- Automation engineer, developers, QA
- Build, test and demo in 1 to 2 week sprints; you accept or adjust after each demo.
- UAT and launch
- QA, DevOps, project manager
- Your team tests and signs off, we switch on, monitor closely and train the team.
- Support
- Automation engineer, DevOps
- Stabilisation window, then optional monthly care with monitoring and changes.
A point of contact
One person who can answer questions and approve decisions, with about an hour a week during the project.
Access
Admin or API access to the systems involved, created as separate accounts with the least access needed.
Sample data
Real but non-sensitive records (or de-identified ones) so we can design and test against reality.
Approved wording
Message templates, policies and consent wording (for SMS, your opt-in and opt-out text) reviewed by you.
Vendor agreements
Signed agreements with the software vendors that will handle personal data, such as a BAA or processor contract.
Time for testing
A few hours from the people who will use the system, for the demos and user acceptance testing.
Working system
Workflows, integrations and any apps running in your accounts, under your keys.
Documentation
Workflow specification, data-flow map, run book and exported workflow files.
Training
A walkthrough for the team and a short recorded video for new starters.
Handover
All access, credentials and code handed over; you can keep us or take it in-house.
Stabilisation window
Fixes for anything that does not meet the agreed acceptance criteria after launch.
Support options after launch
No plan
You run it yourselves after the stabilisation window and contact us when you need a change.
Monthly care
Monitoring, error alerts, small changes and updates each month, with an agreed response time.
Priority care
Faster response times for business-critical workflows and a monthly review call.
How a project moves through our team
- Input
- Logic and services
- People
- Output
- Data
How it flows
- 1 A free call with the project manager and business analyst to understand the problem and whether it is worth building.
- 2 Discovery and design produce a written specification, data-flow map and a backlog with acceptance criteria.
- 3 The team builds in 1 to 2 week sprints; you review a demo at the end of each and accept or adjust.
- 4 Your team runs user acceptance testing, we launch, then monitor and improve under a support plan.
FAQ
LLM evaluation pipeline & regression testing questions, answered.
A repeatable way to test a language model application: a fixed set of inputs, expected behavior for each and scorers that measure how close outputs come. Open-source tools such as promptfoo and Langfuse provide the framework; we build your test set, scorers and CI around them.
Whatever fits the task: exact match or schema checks for structured output, factual checks against a source, similarity for paraphrase, refusal checks for unsafe prompts and model-graded rubrics for tone or helpfulness. We calibrate model-graded scores against human ratings before trusting them.
A CI job runs the evaluation set whenever a prompt, model version or retrieval setting changes, compares scores with the last release and blocks the merge if they drop below your thresholds. promptfoo has an official GitHub Action for this.
They overlap. promptfoo is a lightweight, open-source CLI that fits neatly in CI. Langfuse is an open-source platform with tracing, datasets and experiments that you can self-host. Many teams use promptfoo in CI and Langfuse for production traces; we fit what you already run.
Yes. Cases can assert which tool was called, with which arguments and in what order, as well as the final answer.
Pass rate per category, regressions caught before release, time from model release to a confident upgrade decision and production failures that were not in the test set.
Free 20-minute call
Want this for your ai company?
Book a free 20-minute call. We will look at how the work runs today, tell you honestly whether it is worth building and give you a fixed quote if it is.
More for ai companies
All servicesCRM for AI startups: signups to sales pipeline
Route self-serve signups, demo requests and API key creations into one CRM, score accounts on real product usage, alert sales to product-qualified leads and keep founders out of spreadsheet triage.
From $1,000 · About 4 weeks Details
AI support agent for API and developer docs
An AI support agent that answers developers from your current docs, changelog and status page, explains errors and rate limits, and opens a well-formed ticket with request IDs when a person is needed.
From $2,500 · About 5 to 8 weeks Details
Usage-based billing alerts & anomaly detection
Watch usage and spend per customer against their normal pattern and plan, alert your team and the customer before a runaway loop or leaked key becomes a large invoice, and log every decision.
From $1,000 · About 4 weeks Details