Fixed quote after a free call

An LLM evaluation pipeline that catches regressions before customers do

Every prompt tweak, model upgrade or retrieval change can quietly make some answers worse. We build an LLM evaluation pipeline that runs your test set in CI on every change using promptfoo or Langfuse, compares scores with the last release, blocks merges that fall below thresholds you set and posts a readable summary to the pull request and Slack. A typical build costs $1,000 to $2,500 and takes about four to five weeks.

The problem

Why manual LLM evaluation misses regressions

Most AI teams evaluate, but by hand: someone runs a notebook before a release, eyeballs a few outputs and shares numbers in a chat thread. When a customer reports that answers got worse, nobody can say which change caused it.

Model providers release new versions often, and switching can change tone, refusals, formatting and accuracy in ways a few spot checks will not catch. Without a fixed evaluation set and a record of every run, each upgrade is a gamble.

What we build

What our LLM evaluation pipeline includes

A versioned evaluation set

Real user questions, edge cases and known failures stored in the repository or Langfuse datasets, with expected behavior for each.

Scorers that fit the task

Exact checks for format and facts, similarity checks where wording varies, and model-graded rubrics where judgment is needed, each reviewed by your team.

Runs in CI on every change

A GitHub Actions or GitLab CI job that runs the set when prompts, model versions or retrieval code change.

Regression gates

Merges blocked when scores fall below the thresholds you set, with the failing cases listed.

Readable summaries

A pull request comment and Slack message showing what improved, what regressed and links to every failing output.

History of every run

Results stored so you can show which model, prompt and dataset produced which score on any date.

Who does what

Automation does the chasing, people make the calls.

Every step is marked: what runs on its own, what a member of staff reviews and what stays with a person who decides.
  1. 01 Automated

    Change proposed

    A pull request changes a prompt, model version, retrieval setting or tool definition.

  2. 02 Automated

    Evaluation runs

    CI runs the evaluation set against the changed configuration and the current release.

  3. 03 Automated

    Scores compared

    Results are compared with thresholds and the last release, and a summary is posted.

  4. 04 Team review

    Engineer reviews

    An engineer reads failing cases and fixes the change or updates the test set with reasons.

  5. 05 Founder or model owner decides

    Release decision

    The model owner decides whether to ship, especially when scores trade off against each other.

How it behaves

How LLM regression testing behaves in CI

What each person sees and what runs in the background, including what happens when something goes wrong.

What customers see

  • Fewer surprises when you upgrade a model or change a prompt
  • Fixes for reported failures that stay fixed
  • Release notes that describe behavior changes honestly

What engineers see

  • A pull request comment with scores, regressions and failing cases
  • A blocked merge when quality drops below thresholds
  • A history of every run by model, prompt and dataset

What the automation does

  • Runs the evaluation set on every relevant change
  • Compares with the last release and applies gates
  • Stores results and posts summaries to Slack

Edge cases and fallbacks

Model provider rate limit during a run
The run retries with backoff and reports incomplete rather than passing.
Model-graded scorer disagrees with humans
Flagged in calibration and the rubric or threshold is revised.
Intended behavior change
An engineer updates expected outputs with a written reason in the pull request.
CI or evaluation store unavailable
The gate fails closed and the team is alerted; nothing ships unevaluated by default.
Flaky case with random output
Run several times and scored on the pass rate, or marked for review.

Architecture

An LLM evaluation pipeline in CI at a glance.

The setup we usually start from. We adapt it to the phone system and practice software you already use.
Pull request Prompt or model CI job GitHub Actions Eval dataset Versioned cases Eval runner promptfoo Regression gate Your thresholds Run history Langfuse Model owner Ship decision
  • Input
  • Logic and services
  • Data
  • AI step
  • People

How it flows

  1. 1 A pull request changes a prompt, model version or retrieval setting.
  2. 2 CI loads the versioned evaluation set and runs it against both the change and the current release.
  3. 3 The regression gate compares scores with your thresholds and posts a summary to the pull request and Slack.
  4. 4 Results are stored in the run history, and the model owner decides whether to ship.

Project planner

How the project runs, phase by phase.

Typical durations for a first version. Your proposal lists the exact phases, dates and what you sign off at each step.

LLM evaluation pipeline & regression testing project plan

Typical time to launch: about 4 weeks, then support

  1. 01

    Discovery

    2 to 3 days

    What the product must get right, current evaluation habits, models, CI and data sources.

  2. 02

    Design

    About 1 week

    Evaluation set structure, scorers, thresholds, CI triggers and summary format.

  3. 03

    Build

    1 to 2 weeks

    Dataset, scorers, CI job, gates, summaries and run storage.

  4. 04

    Calibration

    About 1 week

    Run against recent releases, check model-graded scores against human judgment and tune thresholds.

  5. 05

    Launch

    1 to 2 days

    Gate switched on for the main branch and the team walked through reading results.

  6. 06

    Support

    Ongoing

    New cases from production failures, new models and scorer updates.

What it runs on

Real products only, connected through the access each vendor provides. See all integrations

promptfoo (open source) and its GitHub Action Langfuse datasets and experiments (self-hosted or cloud) GitHub Actions or GitLab CI Your model providers' APIs Slack Python or TypeScript scorers

Cost estimate

What it costs, line by line.

Estimates to plan with. Your fixed quote follows a free call, once we have seen your systems and volumes.
  • Discovery and design

    Test set structure, scorers, thresholds and CI triggers.

    $200 – $400

  • Build

    Dataset, scorers, CI job, gates and summaries.

    $500 – $1,300

  • Integrations

    CI, model providers, Langfuse and Slack.

    $150 – $400

  • Calibration

    Scores checked against human judgment and past releases.

    $100 – $250

  • Launch and training

    Gate switch-on and team walkthrough.

    $50 – $150

  • Total project estimate

    $1,000 – $2,500

Optional monthly care after launch is quoted with your proposal. Fixed quote after a free call. Covers one product surface and one CI system. Writing a large labeled evaluation set from scratch is scoped separately. Model usage during runs is billed to your account.

Monthly running costs you pay directly

Estimates from the vendors' published pricing, checked November 2026. Billed to you by each vendor, never marked up.

  • Model usage during runs: Billed by your providers per use; scales with test set size and how often it runs.
  • Langfuse: Free when self-hosted (MIT-licensed core), or Langfuse Cloud at the vendor's own plan.
  • CI minutes: Your GitHub or GitLab plan.

What pushes cost up

  • Systems without an API, or with integration access that must be requested from the vendor
  • Writing data back into your core system rather than only reading from it
  • Several locations, brands or very different processes to support
  • Custom apps, dashboards or a voice line on top of the workflows

What keeps cost down

  • One location and one clear process to start with
  • Cloud software with a documented API
  • Using tools you already pay for instead of adding new ones
  • Approved wording and sample data ready at the start
Get a fixed quote

Privacy and consent

Rules we design for, with the sources.

General information to plan with, not legal advice. Your counsel or data protection adviser confirms what applies to your practice.

EU: evaluation records for model providers

Article 53 of the EU AI Act requires general-purpose model providers to keep technical documentation including testing and evaluation results, applying from 2 August 2025; stored run history supports that record.

Source: artificialintelligenceact.eu

US: NIST AI RMF

The NIST AI Risk Management Framework, released January 26, 2023 for voluntary use, and its Generative AI Profile from July 26, 2024 both treat measurement as a core activity.

Source: www.nist.gov

Customer data in test sets

Evaluation cases drawn from production may contain personal data, so they are minimised or anonymised and stored with processors under GDPR Article 28 contracts.

Source: gdpr-info.eu

How we deliver

Agile delivery, in 1 to 2 week sprints.

We deliver in short Agile sprints using Scrum, so you see working software every one or two weeks instead of waiting for a big reveal at the end.

Every sprint

  1. 1 Sprint planning. At the start of each sprint we agree which backlog items we will finish, each with written acceptance criteria you can check.
  2. 2 Daily stand-up. A short internal check-in on progress and blockers. You get a written update in your Slack or Teams channel, not another meeting.
  3. 3 Sprint review and demo. At the end of each sprint we demo what was built on test data or in a shadow run, and you accept it or ask for changes.
  4. 4 Retrospective. The team looks at what went well and what to change in the next sprint, and we share anything that affects you.

How scope stays under control

Product backlog
Every feature, fix and question in one prioritised list you can see at any time.
Acceptance criteria
Plain-language conditions for "done" on each item, agreed before work starts.
Change requests
New ideas go into the backlog with an estimate. Nothing changes scope or price without your written approval.
User acceptance testing (UAT)
Before launch your team tests the real workflow against the acceptance criteria and signs off.

Tools we use

Jira, ClickUp or Trello for the backlogSlack or Microsoft Teams for updatesGitHub for code and version historyLoom or a live call for demos

Project manager / Scrum master

Runs the sprints, keeps the backlog and timeline honest, and is your single point of contact.

Business analyst

Maps how the work runs today, writes the workflow specification and acceptance criteria with your team.

Automation engineer

Builds the n8n workflows, integrations, AI steps and error handling.

Backend developer

Writes custom code where a no-code step is not enough: APIs, data sync, secure storage.

Frontend / mobile developer

Builds forms, booking pages, dashboards, web apps or Android apps when the project needs them.

QA engineer

Tests every workflow against the acceptance criteria, including edge cases and failure paths.

DevOps engineer

Sets up hosting, access, backups, monitoring and alerts, and the deployment pipeline.

Who does what at each phase

Free call
Project manager, business analyst
Understand the problem, check it is worth automating and outline a rough range.
Discovery
Business analyst, automation engineer
Map the current process, systems and data, and agree success measures.
Design
Business analyst, automation engineer, backend developer
Write the workflow specification, data flows, compliance notes and backlog with acceptance criteria.
Sprints
Automation engineer, developers, QA
Build, test and demo in 1 to 2 week sprints; you accept or adjust after each demo.
UAT and launch
QA, DevOps, project manager
Your team tests and signs off, we switch on, monitor closely and train the team.
Support
Automation engineer, DevOps
Stabilisation window, then optional monthly care with monitoring and changes.

A point of contact

One person who can answer questions and approve decisions, with about an hour a week during the project.

Access

Admin or API access to the systems involved, created as separate accounts with the least access needed.

Sample data

Real but non-sensitive records (or de-identified ones) so we can design and test against reality.

Approved wording

Message templates, policies and consent wording (for SMS, your opt-in and opt-out text) reviewed by you.

Vendor agreements

Signed agreements with the software vendors that will handle personal data, such as a BAA or processor contract.

Time for testing

A few hours from the people who will use the system, for the demos and user acceptance testing.

Working system

Workflows, integrations and any apps running in your accounts, under your keys.

Documentation

Workflow specification, data-flow map, run book and exported workflow files.

Training

A walkthrough for the team and a short recorded video for new starters.

Handover

All access, credentials and code handed over; you can keep us or take it in-house.

Stabilisation window

Fixes for anything that does not meet the agreed acceptance criteria after launch.

Support options after launch

  • No plan

    You run it yourselves after the stabilisation window and contact us when you need a change.

  • Monthly care

    Monitoring, error alerts, small changes and updates each month, with an agreed response time.

  • Priority care

    Faster response times for business-critical workflows and a monthly review call.

How a project moves through our team

Free call PM and analyst Discovery Process and data Design Spec and backlog Sprints Build, test, demo Your review Accept or adjust UAT and launch Sign-off, go live Support Monitor and improve
  • Input
  • Logic and services
  • People
  • Output
  • Data

How it flows

  1. 1 A free call with the project manager and business analyst to understand the problem and whether it is worth building.
  2. 2 Discovery and design produce a written specification, data-flow map and a backlog with acceptance criteria.
  3. 3 The team builds in 1 to 2 week sprints; you review a demo at the end of each and accept or adjust.
  4. 4 Your team runs user acceptance testing, we launch, then monitor and improve under a support plan.

FAQ

LLM evaluation pipeline & regression testing questions, answered.

A repeatable way to test a language model application: a fixed set of inputs, expected behavior for each and scorers that measure how close outputs come. Open-source tools such as promptfoo and Langfuse provide the framework; we build your test set, scorers and CI around them.

Whatever fits the task: exact match or schema checks for structured output, factual checks against a source, similarity for paraphrase, refusal checks for unsafe prompts and model-graded rubrics for tone or helpfulness. We calibrate model-graded scores against human ratings before trusting them.

A CI job runs the evaluation set whenever a prompt, model version or retrieval setting changes, compares scores with the last release and blocks the merge if they drop below your thresholds. promptfoo has an official GitHub Action for this.

They overlap. promptfoo is a lightweight, open-source CLI that fits neatly in CI. Langfuse is an open-source platform with tracing, datasets and experiments that you can self-host. Many teams use promptfoo in CI and Langfuse for production traces; we fit what you already run.

Yes. Cases can assert which tool was called, with which arguments and in what order, as well as the final answer.

Pass rate per category, regressions caught before release, time from model release to a confident upgrade decision and production failures that were not in the test set.

Free 20-minute call

Want this for your ai company?

Book a free 20-minute call. We will look at how the work runs today, tell you honestly whether it is worth building and give you a fixed quote if it is.

Book a free 20-minute call