Case Study
Case Study: AI Startup Evals, Billing & Sales Automation
We built a connected set of automations for an AI startup selling an API and an enterprise plan: a signup-to-sales CRM pipeline, an LLM evaluation pipeline in CI, usage and billing anomaly alerts, security questionnaire drafting and release notes with a model change log. Engineers now see regressions before merging, the founder sees qualified accounts in Slack, and questionnaires no longer eat their evenings.
- Business type
- AI startup selling an API with self-serve and enterprise plans
- Team involved
- Founders, engineers, support lead, first account executive
- Systems connected
- App database, Stripe, CRM, GitHub, evaluation tools, Slack, n8n
- Delivery
- Scrum, 1-week sprints, about 8 weeks to full launch
The challenge
Where the practice started
Signups and API usage lived in the product database, and nobody saw which companies were growing until invoices went out.
Prompt and model changes were checked by running a notebook before release. Customers sometimes noticed regressions first.
Usage spikes from customer bugs showed up at month end as disputed invoices.
Enterprise questionnaires were answered by a founder from scratch each time, and release notes were written when someone remembered.
Goals
What the project had to change
- Every signup and demo request lands in one CRM, and qualified accounts reach a person quickly.
- Every prompt or model change is evaluated before it merges.
- Unusual usage is caught within hours, not at month end.
- Questionnaires are drafted from approved answers and reviewed, not written from scratch.
- Every release has notes and a change log entry with evidence.
What we built
What we built, as one connected system.
Signup-to-sales pipeline
We synced signups, API key creation and daily usage into the CRM by company domain, added qualification signals the founders chose and sent Slack alerts with usage history.
Service detailsLLM evaluation in CI
A versioned evaluation set, scorers calibrated against human ratings and a CI job that runs on every prompt, model or retrieval change and blocks regressions.
Service detailsUsage and billing alerts
Hourly usage compared with each customer's baseline and plan, Slack alerts with charts, optional customer notices and a decision log for pauses and credits.
Service detailsQuestionnaire drafting
An approved answer library built from past questionnaires and policies, drafts with sources for each question and a gap list routed to owners.
Service detailsRelease notes and change log
Merged changes and model version events collected per release, linked to evaluation runs, drafted into notes and approved before publishing.
Service detailsArchitecture
How the startup's workflows connect
- Input
- Logic and services
- AI step
- Data
- People
- Output
How it flows
- 1 Usage, pull requests and policies feed n8n and the CI pipeline.
- 2 Five workflows run evaluations, update the CRM, watch Stripe usage and draft questionnaires and notes.
- 3 The team reviews and approves; the founder decides commitments, pricing and disclosures.
- 4 Alerts and approved notes go out through Slack and the changelog, and every step is logged.
How it behaves
How the system behaves day to day
What customers see
- Faster, more relevant contact when their usage grows
- Early warnings about unusual usage
- Clear release notes when behavior changes
What the team sees
- Qualified accounts and usage alerts in Slack
- Evaluation summaries on every pull request
- Questionnaire drafts and release notes to approve
What the automation does
- Syncs, evaluates, monitors and drafts on schedule
- Never pauses keys, sends answers or publishes notes without approval
- Logs every step and alerts the team if a run fails
Edge cases and fallbacks
- A usage spike is a real launch
- The account owner marks it expected and the baseline adjusts.
- An evaluation run hits provider rate limits
- The run reports incomplete and the gate fails closed.
- A questionnaire asks for a new commitment
- Flagged to the founder; no draft is offered.
- A model change has no eval run
- The change log flags missing evidence before approval.
- n8n or CI is unavailable
- The team is alerted and can work manually until it recovers.
Integrations
What we connected
HubSpot
CRMContacts, companies, deals and pipeline stages for signups and enterprise deals.
HubSpot CRM REST API.
Stripe
BillingUsage meters, subscriptions, usage threshold alerts and invoices.
Stripe API and webhooks.
GitHub
Code and CIPull requests, labels, deployments and evaluation jobs.
GitHub REST API and GitHub Actions.
promptfoo
LLM evaluationEvaluation runs and regression gates in CI.
Open-source CLI and its official GitHub Action.
Langfuse
LLM evaluation and tracingDatasets, experiments, scores and run history.
Langfuse API, self-hosted or cloud.
Linear
Issue trackingIssues linked to releases and engineering tickets from support.
Linear GraphQL API and webhooks.
Slack
Team alertsQualified accounts, usage alerts, evaluation summaries and approvals.
Slack apps and incoming webhooks.
n8n
Automation platformRuns the workflows, self-hosted or on n8n Cloud.
Your own n8n instance; we hand over every workflow.
How we delivered it
Week by week, one workflow at a time.
Delivery timeline
Typical time to launch: about 8 weeks, then support
-
01
Discovery
Week 1
We mapped signups, usage, evaluation habits, billing incidents, questionnaires and releases, and agreed owners for every approval.
-
02
Design and access
Week 2
Scoped API keys, qualification signals, evaluation set structure, usage baselines and the answer library outline.
-
03
Sprint 1: evaluations
Week 3
Built the evaluation set, scorers and CI job, and calibrated scores on recent releases.
-
04
Sprint 2: usage alerts
Week 4
The evaluation gate went live. Built usage baselines and alerts in a test channel.
-
05
Sprint 3: CRM pipeline
Week 5
Usage alerts went live for the team. Built the signup and usage sync and qualified-account alerts.
-
06
Sprint 4: questionnaires
Week 6
The pipeline went live. Built the answer library and drafting, tested on past questionnaires.
-
07
Sprint 5: release notes
Week 7
Questionnaire drafting went live. Built change collection, evidence linking and note drafting.
-
08
UAT and handover
Week 8
The team ran acceptance tests on every workflow, then we switched everything on and handed over run books.
-
09
Support
Ongoing
Weekly check-ins during stabilisation, then monthly care.
Who does what
What the team does, and what automation does.
Automation does
- Syncs signups and usage into the CRM and flags qualified accounts
- Runs the evaluation set on every relevant change
- Watches usage against baselines and sends alerts
- Drafts questionnaire answers with sources
- Drafts release notes and change log entries
The team does
- Engineers read failing evaluation cases and fix changes
- Support and finance investigate usage alerts with customers
- Owners review questionnaire drafts and release notes
- The account executive decides outreach to flagged accounts
Founder or model owner decides
- Decides whether to ship when scores trade off
- Approves commitments on data use, retention and training
- Decides key pauses, credits and limit changes
- Approves disclosures about behavior and policy changes
Compliance handled
Privacy, consent and safety, built in.
Evaluation and documentation records
Stored evaluation runs and the change log support Article 53 technical documentation duties for general-purpose model providers.
Source: artificialintelligenceact.euAI disclosure
Customer-facing AI features carry disclosures in line with Article 50 of the EU AI Act.
Source: artificialintelligenceact.euProcessors under contract
CRM, automation host and model providers that touch user data are covered by GDPR Article 28 processor contracts.
Source: gdpr-info.euCommercial email
Onboarding and sales emails include a postal address and honor opt-outs within CAN-SPAM limits.
Source: www.ftc.govRisk management practice
Measurement and change governance follow the NIST AI RMF approach.
Source: www.nist.govOutcomes
What changed for the practice
- Prompt and model changes are evaluated before they merge, and regressions show up on the pull request.
- The founders see growing accounts in Slack with usage history instead of at invoice time.
- Unusual usage is caught within hours and discussed with the customer before the invoice.
- Questionnaires start from approved drafts, and answers stay consistent between deals.
- Every release has notes and a change log entry linked to its evaluation evidence.
What we track
The measures the practice watches.
Regressions caught
Failing evaluation runs that blocked a merge.
Qualified accounts
Accounts flagged and time from flag to first contact.
Usage alert time
Time from spike start to alert, and disputed invoices from runaway usage.
Questionnaire turnaround
Days to return a questionnaire and share drafted from the library.
Release coverage
Releases with published notes and complete change log entries.
Cost estimate
What a project like this costs
-
Discovery and design
Process mapping, owners, signals and the backlog for all five workflows.
$1,000 – $2,000
-
Evaluation module
Test set, scorers, CI job and gates.
$700 – $1,700
-
Usage alert module
Baselines, scoring, alerts and decision log.
$800 – $2,000
-
CRM pipeline module
Signup and usage sync, alerts and follow-ups.
$800 – $2,000
-
Release notes module
Change collection, evidence linking and drafting.
$500 – $1,300
-
Questionnaire module
Answer library, drafting and gap routing.
$700 – $1,600
-
Integrations
App database, Stripe, CRM, GitHub and Slack connections.
$200 – $400
-
Testing, UAT and launch
QA, acceptance testing, launch and training.
$300 – $1,500
-
Total project estimate
$5,000 – $12,500
Optional monthly care after launch is quoted with your proposal. Typical range for a project of this scope: five workflows at our fixed per-workflow range. Every company gets its own fixed quote after a free call.
Monthly running costs you pay directly
Estimates from the vendors' published pricing, checked November 2026. Billed to you by each vendor, never marked up.
- n8n: Free self-hosted Community edition, or n8n Cloud from about €20 a month (Starter) or €50 (Pro), billed annually.
- Langfuse: Free when self-hosted, or Langfuse Cloud at the vendor's plan.
- Model usage: Pay-per-use fees for evaluation runs and drafting.
- CRM and Stripe: Each vendor's own plan and fees.
What pushes cost up
- Systems without an API, or with integration access that must be requested from the vendor
- Writing data back into your core system rather than only reading from it
- Several locations, brands or very different processes to support
- Custom apps, dashboards or a voice line on top of the workflows
What keeps cost down
- One location and one clear process to start with
- Cloud software with a documented API
- Using tools you already pay for instead of adding new ones
- Approved wording and sample data ready at the start
How we deliver
Agile delivery, in 1 to 2 week sprints.
Every sprint
- 1 Sprint planning. At the start of each sprint we agree which backlog items we will finish, each with written acceptance criteria you can check.
- 2 Daily stand-up. A short internal check-in on progress and blockers. You get a written update in your Slack or Teams channel, not another meeting.
- 3 Sprint review and demo. At the end of each sprint we demo what was built on test data or in a shadow run, and you accept it or ask for changes.
- 4 Retrospective. The team looks at what went well and what to change in the next sprint, and we share anything that affects you.
How scope stays under control
- Product backlog
- Every feature, fix and question in one prioritised list you can see at any time.
- Acceptance criteria
- Plain-language conditions for "done" on each item, agreed before work starts.
- Change requests
- New ideas go into the backlog with an estimate. Nothing changes scope or price without your written approval.
- User acceptance testing (UAT)
- Before launch your team tests the real workflow against the acceptance criteria and signs off.
Tools we use
Project manager / Scrum master
Runs the sprints, keeps the backlog and timeline honest, and is your single point of contact.
Business analyst
Maps how the work runs today, writes the workflow specification and acceptance criteria with your team.
Automation engineer
Builds the n8n workflows, integrations, AI steps and error handling.
Backend developer
Writes custom code where a no-code step is not enough: APIs, data sync, secure storage.
Frontend / mobile developer
Builds forms, booking pages, dashboards, web apps or Android apps when the project needs them.
QA engineer
Tests every workflow against the acceptance criteria, including edge cases and failure paths.
DevOps engineer
Sets up hosting, access, backups, monitoring and alerts, and the deployment pipeline.
Who does what at each phase
- Free call
- Project manager, business analyst
- Understand the problem, check it is worth automating and outline a rough range.
- Discovery
- Business analyst, automation engineer
- Map the current process, systems and data, and agree success measures.
- Design
- Business analyst, automation engineer, backend developer
- Write the workflow specification, data flows, compliance notes and backlog with acceptance criteria.
- Sprints
- Automation engineer, developers, QA
- Build, test and demo in 1 to 2 week sprints; you accept or adjust after each demo.
- UAT and launch
- QA, DevOps, project manager
- Your team tests and signs off, we switch on, monitor closely and train the team.
- Support
- Automation engineer, DevOps
- Stabilisation window, then optional monthly care with monitoring and changes.
A point of contact
One person who can answer questions and approve decisions, with about an hour a week during the project.
Access
Admin or API access to the systems involved, created as separate accounts with the least access needed.
Sample data
Real but non-sensitive records (or de-identified ones) so we can design and test against reality.
Approved wording
Message templates, policies and consent wording (for SMS, your opt-in and opt-out text) reviewed by you.
Vendor agreements
Signed agreements with the software vendors that will handle personal data, such as a BAA or processor contract.
Time for testing
A few hours from the people who will use the system, for the demos and user acceptance testing.
Working system
Workflows, integrations and any apps running in your accounts, under your keys.
Documentation
Workflow specification, data-flow map, run book and exported workflow files.
Training
A walkthrough for the team and a short recorded video for new starters.
Handover
All access, credentials and code handed over; you can keep us or take it in-house.
Stabilisation window
Fixes for anything that does not meet the agreed acceptance criteria after launch.
Support options after launch
No plan
You run it yourselves after the stabilisation window and contact us when you need a change.
Monthly care
Monitoring, error alerts, small changes and updates each month, with an agreed response time.
Priority care
Faster response times for business-critical workflows and a monthly review call.
How a project moves through our team
- Input
- Logic and services
- People
- Output
- Data
How it flows
- 1 A free call with the project manager and business analyst to understand the problem and whether it is worth building.
- 2 Discovery and design produce a written specification, data-flow map and a backlog with acceptance criteria.
- 3 The team builds in 1 to 2 week sprints; you review a demo at the end of each and accept or adjust.
- 4 Your team runs user acceptance testing, we launch, then monitor and improve under a support plan.
Tools we built
Free tools we built and use every day.
The Make It Flow platform
Website audits, monitoring and reports
- AI Search Readiness FreeCheck access for Google AI, Copilot, ChatGPT and Perplexity bots
- SEO Audit FreeTitles, meta, headings, indexability, sitemaps and schema
- Website Security Scan FreeSSL, security headers, cookies and exposed file checks
- Email Deliverability Check FreeMX, SPF, DKIM, DMARC, MTA-STS and BIMI with ready DNS fixes
- Website Monitoring Scheduled re-scans, score emails and 5-minute uptime alerts
- Technology Detection FreeSee the CMS, frameworks and analytics tools behind any website
Free website tools for practices
Services in this case study
CRM for AI startups: signups to sales pipeline
Route self-serve signups, demo requests and API key creations into one CRM, score accounts on real product usage, alert sales to product-qualified leads and keep founders out of spreadsheet triage.
From $1,000 · About 4 weeks Details
LLM evaluation pipeline & regression testing
Run your evaluation set on every prompt, model or retrieval change in CI, compare scores with the last release, block regressions and post a readable summary to the pull request and Slack.
From $1,000 · About 4 to 5 weeks Details
Usage-based billing alerts & anomaly detection
Watch usage and spend per customer against their normal pattern and plan, alert your team and the customer before a runaway loop or leaked key becomes a large invoice, and log every decision.
From $1,000 · About 4 weeks Details
Security & AI governance questionnaire automation
Draft answers to security and AI governance questionnaires from an approved answer library, your policies and model change log, flag questions that need a new answer and keep every response consistent.
From $1,000 · About 4 weeks Details
Release notes automation & model change log
Turn merged pull requests, model version changes and eval results into drafted release notes and a change log with approvals, so you can show what changed, when and who signed it off.
From $1,000 · About 4 weeks Details
Founder or model owner decides
-
AI Startup Website Checklist: Crawlers, Trust Pages and UK RulesGuides · 10 min read
-
LLM Evaluation Workflow: Tracking Regressions on Every ChangeAutomation · 8 min read
-
What Are AI Agents? A Plain-English Guide for Small BusinessesAutomation · 10 min read
-
Claude Referral Traffic Is Rising: How to Get Cited by ClaudeSEO · 9 min read
-
Block or Allow AI Crawlers? GPTBot, ClaudeBot and MoreSEO · 10 min read
-
What Is llms.txt and Do I Need One for My Website?Guides · 7 min read
-
Support Ticket Triage and Release Notes: A Workflow for Software CompaniesAutomation · 10 min read
-
SaaS SEO Checklist: 12 Fixes for Software Company WebsitesSEO · 10 min read
Free 20-minute call
Want this built for your ai company?
Book a free 20-minute call. We will look at how the work runs today, tell you honestly whether it is worth building and give you a fixed quote if it is.
Sources we checked
Rules and product facts on this page come from these pages, checked in November 2026. This is general information, not legal advice; confirm the rules that apply to your practice with your own adviser.
- EU AI Act, Article 50 (transparency)
- EU AI Act, Article 53 (general-purpose AI model providers)
- NIST AI Risk Management Framework
- GDPR Article 28 (processors)
- FTC: CAN-SPAM Act compliance guide
- UK PECR regulation 22
- W3C WCAG 2.2
- Stripe: usage-based billing (incl. usage monitoring alerts)
- promptfoo: GitHub Action
- promptfoo: CI/CD integration
- Langfuse: clarifications (MIT core, self-hosting)
- Langfuse documentation
- Attio REST API
- Zendesk Ticketing API
- Intercom developer FAQs
- Linear API and webhooks
- Drata: public API updates for questionnaire tracking
- Vanta: questionnaire automation
- n8n pricing
- n8n self-hosting
- Google autocomplete phrasing checked for keywords (llm evaluation, security questionnaire automation, stripe usage based billing, ai support agent, ai startup website, release notes automation)