Case Study

Case Study: AI Startup Evals, Billing & Sales Automation

We built a connected set of automations for an AI startup selling an API and an enterprise plan: a signup-to-sales CRM pipeline, an LLM evaluation pipeline in CI, usage and billing anomaly alerts, security questionnaire drafting and release notes with a model change log. Engineers now see regressions before merging, the founder sees qualified accounts in Slack, and questionnaires no longer eat their evenings.

Business type
AI startup selling an API with self-serve and enterprise plans
Team involved
Founders, engineers, support lead, first account executive
Systems connected
App database, Stripe, CRM, GitHub, evaluation tools, Slack, n8n
Delivery
Scrum, 1-week sprints, about 8 weeks to full launch

The challenge

Where the practice started

Signups and API usage lived in the product database, and nobody saw which companies were growing until invoices went out.

Prompt and model changes were checked by running a notebook before release. Customers sometimes noticed regressions first.

Usage spikes from customer bugs showed up at month end as disputed invoices.

Enterprise questionnaires were answered by a founder from scratch each time, and release notes were written when someone remembered.

Goals

What the project had to change

  • Every signup and demo request lands in one CRM, and qualified accounts reach a person quickly.
  • Every prompt or model change is evaluated before it merges.
  • Unusual usage is caught within hours, not at month end.
  • Questionnaires are drafted from approved answers and reviewed, not written from scratch.
  • Every release has notes and a change log entry with evidence.

What we built

What we built, as one connected system.

01

Signup-to-sales pipeline

We synced signups, API key creation and daily usage into the CRM by company domain, added qualification signals the founders chose and sent Slack alerts with usage history.

Service details
02

LLM evaluation in CI

A versioned evaluation set, scorers calibrated against human ratings and a CI job that runs on every prompt, model or retrieval change and blocks regressions.

Service details
03

Usage and billing alerts

Hourly usage compared with each customer's baseline and plan, Slack alerts with charts, optional customer notices and a decision log for pauses and credits.

Service details
04

Questionnaire drafting

An approved answer library built from past questionnaires and policies, drafts with sources for each question and a gap list routed to owners.

Service details
05

Release notes and change log

Merged changes and model version events collected per release, linked to evaluation runs, drafted into notes and approved before publishing.

Service details

Architecture

How the startup's workflows connect

App and usage Signups, metering GitHub Pull requests Policies Answer library n8n and CI Five workflows Evaluations promptfoo, Langfuse CRM Accounts, deals Stripe Usage and billing Team Review, approve Founder Commitments Slack and notes Alerts, changelog
  • Input
  • Logic and services
  • AI step
  • Data
  • People
  • Output

How it flows

  1. 1 Usage, pull requests and policies feed n8n and the CI pipeline.
  2. 2 Five workflows run evaluations, update the CRM, watch Stripe usage and draft questionnaires and notes.
  3. 3 The team reviews and approves; the founder decides commitments, pricing and disclosures.
  4. 4 Alerts and approved notes go out through Slack and the changelog, and every step is logged.

How it behaves

How the system behaves day to day

What each person sees and what runs in the background, including what happens when something goes wrong.

What customers see

  • Faster, more relevant contact when their usage grows
  • Early warnings about unusual usage
  • Clear release notes when behavior changes

What the team sees

  • Qualified accounts and usage alerts in Slack
  • Evaluation summaries on every pull request
  • Questionnaire drafts and release notes to approve

What the automation does

  • Syncs, evaluates, monitors and drafts on schedule
  • Never pauses keys, sends answers or publishes notes without approval
  • Logs every step and alerts the team if a run fails

Edge cases and fallbacks

A usage spike is a real launch
The account owner marks it expected and the baseline adjusts.
An evaluation run hits provider rate limits
The run reports incomplete and the gate fails closed.
A questionnaire asks for a new commitment
Flagged to the founder; no draft is offered.
A model change has no eval run
The change log flags missing evidence before approval.
n8n or CI is unavailable
The team is alerted and can work manually until it recovers.

Integrations

What we connected

All integrations

HubSpot

CRM

Contacts, companies, deals and pipeline stages for signups and enterprise deals.

HubSpot CRM REST API.

Stripe

Billing

Usage meters, subscriptions, usage threshold alerts and invoices.

Stripe API and webhooks.

GitHub

Code and CI

Pull requests, labels, deployments and evaluation jobs.

GitHub REST API and GitHub Actions.

promptfoo

LLM evaluation

Evaluation runs and regression gates in CI.

Open-source CLI and its official GitHub Action.

Langfuse

LLM evaluation and tracing

Datasets, experiments, scores and run history.

Langfuse API, self-hosted or cloud.

Linear

Issue tracking

Issues linked to releases and engineering tickets from support.

Linear GraphQL API and webhooks.

Slack

Team alerts

Qualified accounts, usage alerts, evaluation summaries and approvals.

Slack apps and incoming webhooks.

n8n

Automation platform

Runs the workflows, self-hosted or on n8n Cloud.

Your own n8n instance; we hand over every workflow.

How we delivered it

Week by week, one workflow at a time.

Each workflow went live on its own after a shadow run and a sprint demo, so the team got used to one change before the next.

Delivery timeline

Typical time to launch: about 8 weeks, then support

  1. 01

    Discovery

    Week 1

    We mapped signups, usage, evaluation habits, billing incidents, questionnaires and releases, and agreed owners for every approval.

  2. 02

    Design and access

    Week 2

    Scoped API keys, qualification signals, evaluation set structure, usage baselines and the answer library outline.

  3. 03

    Sprint 1: evaluations

    Week 3

    Built the evaluation set, scorers and CI job, and calibrated scores on recent releases.

  4. 04

    Sprint 2: usage alerts

    Week 4

    The evaluation gate went live. Built usage baselines and alerts in a test channel.

  5. 05

    Sprint 3: CRM pipeline

    Week 5

    Usage alerts went live for the team. Built the signup and usage sync and qualified-account alerts.

  6. 06

    Sprint 4: questionnaires

    Week 6

    The pipeline went live. Built the answer library and drafting, tested on past questionnaires.

  7. 07

    Sprint 5: release notes

    Week 7

    Questionnaire drafting went live. Built change collection, evidence linking and note drafting.

  8. 08

    UAT and handover

    Week 8

    The team ran acceptance tests on every workflow, then we switched everything on and handed over run books.

  9. 09

    Support

    Ongoing

    Weekly check-ins during stabilisation, then monthly care.

Who does what

What the team does, and what automation does.

Automation does

  • Syncs signups and usage into the CRM and flags qualified accounts
  • Runs the evaluation set on every relevant change
  • Watches usage against baselines and sends alerts
  • Drafts questionnaire answers with sources
  • Drafts release notes and change log entries

The team does

  • Engineers read failing evaluation cases and fix changes
  • Support and finance investigate usage alerts with customers
  • Owners review questionnaire drafts and release notes
  • The account executive decides outreach to flagged accounts

Founder or model owner decides

  • Decides whether to ship when scores trade off
  • Approves commitments on data use, retention and training
  • Decides key pauses, credits and limit changes
  • Approves disclosures about behavior and policy changes

Compliance handled

Privacy, consent and safety, built in.

Evaluation and documentation records

Stored evaluation runs and the change log support Article 53 technical documentation duties for general-purpose model providers.

Source: artificialintelligenceact.eu

AI disclosure

Customer-facing AI features carry disclosures in line with Article 50 of the EU AI Act.

Source: artificialintelligenceact.eu

Processors under contract

CRM, automation host and model providers that touch user data are covered by GDPR Article 28 processor contracts.

Source: gdpr-info.eu

Commercial email

Onboarding and sales emails include a postal address and honor opt-outs within CAN-SPAM limits.

Source: www.ftc.gov

Risk management practice

Measurement and change governance follow the NIST AI RMF approach.

Source: www.nist.gov

Outcomes

What changed for the practice

  • Prompt and model changes are evaluated before they merge, and regressions show up on the pull request.
  • The founders see growing accounts in Slack with usage history instead of at invoice time.
  • Unusual usage is caught within hours and discussed with the customer before the invoice.
  • Questionnaires start from approved drafts, and answers stay consistent between deals.
  • Every release has notes and a change log entry linked to its evaluation evidence.

What we track

The measures the practice watches.

Regressions caught

Failing evaluation runs that blocked a merge.

Qualified accounts

Accounts flagged and time from flag to first contact.

Usage alert time

Time from spike start to alert, and disputed invoices from runaway usage.

Questionnaire turnaround

Days to return a questionnaire and share drafted from the library.

Release coverage

Releases with published notes and complete change log entries.

Cost estimate

What a project like this costs

Estimates to plan with. Your fixed quote follows a free call, once we have seen your systems and volumes.
  • Discovery and design

    Process mapping, owners, signals and the backlog for all five workflows.

    $1,000 – $2,000

  • Evaluation module

    Test set, scorers, CI job and gates.

    $700 – $1,700

  • Usage alert module

    Baselines, scoring, alerts and decision log.

    $800 – $2,000

  • CRM pipeline module

    Signup and usage sync, alerts and follow-ups.

    $800 – $2,000

  • Release notes module

    Change collection, evidence linking and drafting.

    $500 – $1,300

  • Questionnaire module

    Answer library, drafting and gap routing.

    $700 – $1,600

  • Integrations

    App database, Stripe, CRM, GitHub and Slack connections.

    $200 – $400

  • Testing, UAT and launch

    QA, acceptance testing, launch and training.

    $300 – $1,500

  • Total project estimate

    $5,000 – $12,500

Optional monthly care after launch is quoted with your proposal. Typical range for a project of this scope: five workflows at our fixed per-workflow range. Every company gets its own fixed quote after a free call.

Monthly running costs you pay directly

Estimates from the vendors' published pricing, checked November 2026. Billed to you by each vendor, never marked up.

  • n8n: Free self-hosted Community edition, or n8n Cloud from about €20 a month (Starter) or €50 (Pro), billed annually.
  • Langfuse: Free when self-hosted, or Langfuse Cloud at the vendor's plan.
  • Model usage: Pay-per-use fees for evaluation runs and drafting.
  • CRM and Stripe: Each vendor's own plan and fees.

What pushes cost up

  • Systems without an API, or with integration access that must be requested from the vendor
  • Writing data back into your core system rather than only reading from it
  • Several locations, brands or very different processes to support
  • Custom apps, dashboards or a voice line on top of the workflows

What keeps cost down

  • One location and one clear process to start with
  • Cloud software with a documented API
  • Using tools you already pay for instead of adding new ones
  • Approved wording and sample data ready at the start
Get a fixed quote

How we deliver

Agile delivery, in 1 to 2 week sprints.

We deliver in short Agile sprints using Scrum, so you see working software every one or two weeks instead of waiting for a big reveal at the end.

Every sprint

  1. 1 Sprint planning. At the start of each sprint we agree which backlog items we will finish, each with written acceptance criteria you can check.
  2. 2 Daily stand-up. A short internal check-in on progress and blockers. You get a written update in your Slack or Teams channel, not another meeting.
  3. 3 Sprint review and demo. At the end of each sprint we demo what was built on test data or in a shadow run, and you accept it or ask for changes.
  4. 4 Retrospective. The team looks at what went well and what to change in the next sprint, and we share anything that affects you.

How scope stays under control

Product backlog
Every feature, fix and question in one prioritised list you can see at any time.
Acceptance criteria
Plain-language conditions for "done" on each item, agreed before work starts.
Change requests
New ideas go into the backlog with an estimate. Nothing changes scope or price without your written approval.
User acceptance testing (UAT)
Before launch your team tests the real workflow against the acceptance criteria and signs off.

Tools we use

Jira, ClickUp or Trello for the backlogSlack or Microsoft Teams for updatesGitHub for code and version historyLoom or a live call for demos

Project manager / Scrum master

Runs the sprints, keeps the backlog and timeline honest, and is your single point of contact.

Business analyst

Maps how the work runs today, writes the workflow specification and acceptance criteria with your team.

Automation engineer

Builds the n8n workflows, integrations, AI steps and error handling.

Backend developer

Writes custom code where a no-code step is not enough: APIs, data sync, secure storage.

Frontend / mobile developer

Builds forms, booking pages, dashboards, web apps or Android apps when the project needs them.

QA engineer

Tests every workflow against the acceptance criteria, including edge cases and failure paths.

DevOps engineer

Sets up hosting, access, backups, monitoring and alerts, and the deployment pipeline.

Who does what at each phase

Free call
Project manager, business analyst
Understand the problem, check it is worth automating and outline a rough range.
Discovery
Business analyst, automation engineer
Map the current process, systems and data, and agree success measures.
Design
Business analyst, automation engineer, backend developer
Write the workflow specification, data flows, compliance notes and backlog with acceptance criteria.
Sprints
Automation engineer, developers, QA
Build, test and demo in 1 to 2 week sprints; you accept or adjust after each demo.
UAT and launch
QA, DevOps, project manager
Your team tests and signs off, we switch on, monitor closely and train the team.
Support
Automation engineer, DevOps
Stabilisation window, then optional monthly care with monitoring and changes.

A point of contact

One person who can answer questions and approve decisions, with about an hour a week during the project.

Access

Admin or API access to the systems involved, created as separate accounts with the least access needed.

Sample data

Real but non-sensitive records (or de-identified ones) so we can design and test against reality.

Approved wording

Message templates, policies and consent wording (for SMS, your opt-in and opt-out text) reviewed by you.

Vendor agreements

Signed agreements with the software vendors that will handle personal data, such as a BAA or processor contract.

Time for testing

A few hours from the people who will use the system, for the demos and user acceptance testing.

Working system

Workflows, integrations and any apps running in your accounts, under your keys.

Documentation

Workflow specification, data-flow map, run book and exported workflow files.

Training

A walkthrough for the team and a short recorded video for new starters.

Handover

All access, credentials and code handed over; you can keep us or take it in-house.

Stabilisation window

Fixes for anything that does not meet the agreed acceptance criteria after launch.

Support options after launch

  • No plan

    You run it yourselves after the stabilisation window and contact us when you need a change.

  • Monthly care

    Monitoring, error alerts, small changes and updates each month, with an agreed response time.

  • Priority care

    Faster response times for business-critical workflows and a monthly review call.

How a project moves through our team

Free call PM and analyst Discovery Process and data Design Spec and backlog Sprints Build, test, demo Your review Accept or adjust UAT and launch Sign-off, go live Support Monitor and improve
  • Input
  • Logic and services
  • People
  • Output
  • Data

How it flows

  1. 1 A free call with the project manager and business analyst to understand the problem and whether it is worth building.
  2. 2 Discovery and design produce a written specification, data-flow map and a backlog with acceptance criteria.
  3. 3 The team builds in 1 to 2 week sprints; you review a demo at the end of each and accept or adjust.
  4. 4 Your team runs user acceptance testing, we launch, then monitor and improve under a support plan.

Free 20-minute call

Want this built for your ai company?

Book a free 20-minute call. We will look at how the work runs today, tell you honestly whether it is worth building and give you a fixed quote if it is.

Book a free 20-minute call