E2E testing for LLM SaaS: deterministic tests, goldens, CI/CD

Introduction

LLM features break the testing habits that worked for classic SaaS.

You can ship a search filter, write a few unit tests, and call it done. You can’t do that with a summarizer, a chat assistant, or an “auto fill this form” flow. The output is probabilistic, the model changes, and the same prompt can behave differently depending on context.

So what do you test?

• The product contract (what users must get every time)
• The model behavior envelope (what is acceptable, what is not)
• The system around the model (retrieval, tools, permissions, rate limits, fallbacks)

Insight: Treat LLM output like a UI. You don’t snapshot every pixel. You test the parts that must not break.

In Apptension projects, the teams that move fastest are the ones that create deterministic seams early. Then they add golden datasets and evaluation gates in CI. It’s less exciting than “prompt magic”, but it’s what keeps releases boring.

• This article assumes you already have E2E tests for your SaaS.
• We’ll focus on what changes when you add AI assisted workflows.
• We’ll be blunt about what fails and how we mitigate it.

What we mean by end to end for AI assisted SaaS

For LLM features, end to end testing usually spans:

1. Frontend flow (user intent and inputs)
2. Backend orchestration (prompt building, tools, retrieval)
3. Model call (LLM provider or self hosted model)
4. Post processing (validators, redaction, formatting)
5. Persistence and audit logs
6. User visible output and side effects (tickets created, emails drafted, tasks updated)

If your “E2E” test only asserts that an API returned 200, you’ll miss the failures users actually report.

Why LLM features are hard to test in production like systems

Most teams hit the same failure modes within the first few weeks.

• Non determinism: temperature, sampling, system prompts, and tool selection can shift outputs.
• Hidden dependencies: retrieval results change when new documents land.
• Vendor drift: model versions update, safety filters change, latency varies.
• Prompt coupling: a tiny prompt edit breaks a downstream parser.
• Compliance risk: you can’t “just log everything” in regulated contexts.

Key Stat: 76% of consumers get frustrated when organizations fail to deliver personalized interactions.

That stat is usually cited to justify personalization. In testing, I read it differently: users have a low tolerance for inconsistent experiences. If your assistant is helpful on Monday and sloppy on Tuesday, they don’t care that “the model is stochastic”.

What we measure (or propose measuring) in AI E2E:

• Task success rate (did the user goal complete)
• Critical error rate (hallucinated facts, wrong tool call, permission leak)
• Latency at p95 and p99 for the full flow
• Cost per successful task (token spend plus tool calls)
• Regression rate after prompt or model changes

Common anti patterns we keep seeing

• Snapshot testing full responses. It makes CI noisy and teaches the team to ignore failures.
• Testing only the “happy prompt”. Users don’t type like your product manager.
• No deterministic seams. Everything depends on the live model, so every test is flaky.

A quick comparison: what breaks and what holds

A note on “deterministic” in an LLM system

Deterministic does not mean “the model always says the same sentence”. It means:

• The system has stable contracts at boundaries.
• You can replay the same inputs.
• You can explain why a test failed.

If you can’t do those three, you don’t have a test. You have a demo.

Deterministic seams: what to lock down, what to let float

The fastest path to reliable E2E tests is to decide where you want determinism.

Here’s the split that works in practice:

• Lock down: inputs, retrieval corpus, tool outputs, schemas, permission checks, redaction rules
• Let float: wording, tone, minor ordering, optional details

processSteps: Build deterministic seams in 7 steps

1. Define the user task as a contract (inputs, outputs, side effects).
2. Add a structured output schema (JSON schema, Zod, Pydantic).
3. Validate and repair model output (retry with guardrails, not blind retries).
4. Wrap tool calls with strict interfaces and logging.
5. Freeze retrieval for tests (golden corpora and fixed embeddings where possible).
6. Add policy checks (PII, permissions, allowed sources) as code, not prompts.
7. Make failure visible in UI (don’t hide it behind “try again”).

Insight: If a requirement matters, it can’t live only in the prompt.

Concrete example: schema first generation

A common pattern is “LLM generates a blob of text, then we parse it”. That’s fragile.

Better pattern:

{
  "task": "support_reply",
  "language": "en",
  "tone": "calm",
  "answer": "...",
  "citations": [{
    "source_id": "kb-123",
    "quote": "..."
  }],
  "actions": [{
    "type": "create_ticket",
    "priority": "high"
  }]
}

Now your tests can assert:

• answer is present and non empty
• citations only reference allowed sources
• actions are allowed for the user role

benefits: What deterministic seams buy you

• Fewer flaky tests and fewer “rerun CI” habits
• Faster debugging (you know which boundary failed)
• Safer releases (policy checks are enforced as code)
• Cleaner audit trails for regulated work

Where we’ve used this mindset outside LLMs

This is not unique to AI. In our React Native cryptography work, the tricky part wasn’t only implementing AES and RSA flows. It was testing and proving behavior across mobile and web constraints. The lesson transfers: when the system is complex, you isolate what must be deterministic, then test the boundaries hard.

LLMs just make the need obvious.

Use one JSONL line per test case.

{
  "id": "support_reply_042",
  "input": {
    "message": "I was charged twice. Fix it.",
    "language": "en"
  },
  "context": {
    "user_role": "customer",
    "retrieval_snapshot_id": "kb_snapshot_2026_01_15",
    "allowed_tools": ["refund_lookup", "ticket_create"]
  },
  "assertions": {
    "schema": "support_reply_v3",
    "must_call_tools": [{
      "name": "refund_lookup"
    }],
    "must_not_contain": ["full card number"],
    "must_include": ["apology", "next step"]
  }
}

Keep assertions small. If you need 20 assertions, the contract is unclear.

Golden datasets: your regression suite for behavior, not words

Golden datasets are how you stop arguing about “does this feel better”. They turn it into a repeatable check.

A good golden dataset is not huge. It’s representative.

• 50 to 200 prompts can catch most regressions for a single feature.
• Include real user phrasing, not only “clean” prompts.
• Include adversarial cases: missing context, conflicting instructions, unsafe requests.

Example: When we built Teamdeck (our own resource management SaaS), the product lived or died on predictable workflows: planning, time tracking, reporting. AI assisted features should be tested with the same discipline: the workflow matters more than fancy output.

What goes into a golden item

Each item should include:

• Input (user message, form fields, selected context)
• Context snapshot (retrieved docs, tool availability, user role)
• Expected outcome checks (not exact text)

Outcome checks that work well:

• Schema validity (pass/fail)
• Required facts present (string contains, regex, or entity match)
• Forbidden content absent (PII, policy violations)
• Correct tool calls (name, arguments, count)
• Citation integrity (only from allowed sources)

Comparison: ways to score LLM outputs

faq: Golden dataset questions teams ask

• Q: How often do we update goldens? A: Whenever product requirements change. Not when the model “feels different”. If the contract is the same, the goldens should catch drift.

• Q: Do we store model outputs in git? A: Store inputs and evaluation results. Store outputs only if you can do it safely (PII). In regulated setups, store redacted outputs or hashed artifacts.

• Q: What if we can’t agree on the expected behavior? A: Write down the contract in product terms. Then label a small set with humans. Use that as the baseline.

A practical labeling workflow that doesn’t stall the team

If you’ve read our piece on scaling post MVP, you’ll recognize the pattern: early hustle works, then it stops scaling.

Labeling is the same.

• Start with 20 to 30 examples.
• Do a 45 minute calibration session with PM, QA, and an engineer.
• Agree on 5 to 8 failure categories.
• Label only disagreements and edge cases.

Hypothesis worth testing: if you keep the label taxonomy small, you can keep weekly evaluation under 60 minutes for a feature. Measure it.

Start with a small set. Add categories only when you see repeated ambiguity.

• Policy and safety (PII leak, disallowed advice)
• Permissions (used data the user should not see)
• Tooling (wrong tool, wrong args, missing tool call)
• Retrieval (no citation, wrong citation, out of scope source)
• Formatting (schema invalid, missing required fields)
• UX (tone wrong, too long, unclear next step)

Track counts per category over time. It tells you what to fix next.

CI/CD for LLM features: evaluation gates, not brittle tests

Classic CI expects deterministic pass or fail. LLM features need thresholds.

The pattern we use is simple:

1. Run fast deterministic tests on every PR (schemas, tool routing, permission checks).
2. Run golden evaluation as a separate job (can be slower).
3. Block merges only when key metrics drop below agreed thresholds.

Insight: Your CI should tell you “quality dropped by 6%” not “test expected this exact sentence”.

What to gate on

Pick a small set of metrics that map to user pain.

• Task success rate on goldens (target threshold)
• Critical safety failures (must be zero or near zero)
• Tool call correctness (threshold)
• Latency budget (p95, and sometimes p99)
• Cost budget (tokens per task, tool call count)

A minimal CI pipeline sketch

jobs:
  unit_and_contract_tests:
    steps:
      - run: pnpm test
      - run: pnpm test:schemas
      - run: pnpm test:policy

  golden_eval:
    needs: unit_and_contract_tests
    steps:
      - run: python eval/run.py --dataset goldens/support_reply_v3.jsonl
      - run: python eval/report.py --fail-on "critical_failures>0" \
                               --fail-on "success_rate2500"

  canary_release:
    if: main
    steps:
      - run: deploy --strategy canary --percent 10
      - run: monitor --slo "ai_task_success>0.90" --window 30m

What fails in CI/CD and how we handle it

• Model provider incident: tests fail for reasons unrelated to your change.

  • Mitigation: allow reruns, but also record provider status and error rates.

• Judge model drift: LLM as judge changes its scoring.

  • Mitigation: pin judge model version, keep a small human labeled calibration set.

• Hidden data changes: retrieval corpus updates break goldens.

  • Mitigation: golden corpora snapshots and explicit dataset versioning.

benefits: What a good evaluation gate changes in the team

• Engineers stop fearing prompt edits.
• PMs get a shared language for quality.
• QA stops being the last line of defense.
• Releases become smaller and more frequent.

Where UAT still matters

Automated evaluation does not replace user acceptance testing. It narrows the surface area.

In a fintech UAT process we led, the hard part was stakeholder alignment and regulatory expectations. LLM features add another layer: you’re aligning on what “acceptable” means.

Use UAT for:

• New workflows and UX changes
• High stakes domains (finance, health, legal)
• Anything that touches user trust

Automate what can be automated. Then spend human time where it actually matters.

Conclusion

End to end testing for AI assisted SaaS is not about catching every weird sentence. It’s about building a system you can reason about.

If you want a practical starting point, do this in order:

1. Define contracts for each AI feature (inputs, outputs, side effects).
2. Add deterministic seams: schemas, tool interfaces, policy checks.
3. Build a golden dataset that reflects real user prompts and edge cases.
4. Put evaluation gates in CI/CD with thresholds tied to user pain.
5. Monitor in production with canaries and clear rollback triggers.

Final takeaway: If the only thing keeping your AI feature “working” is that nobody touched the prompt, you don’t have a feature. You have a fragile demo.

Checklist you can copy into your backlog

• Structured output schema and validators
• Tool call contract tests
• Retrieval snapshot for goldens
• Golden dataset versioning
• Evaluation report with success, safety, latency, cost
• CI gate thresholds agreed with product
• Canary release and rollback criteria

When you do this well, you ship faster. Not because the model got smarter, but because your system got more predictable.