Testing & AI

Mock-First Testing for AI Systems: Build Confidence Before Production

Learn how to test AI applications deterministically without calling expensive LLMs during every test run. We cover mock strategies, Pydantic contracts, evaluation gates, and production confidence.

Sep 24, 2026

14 min read

By Brian Shimkus

Mock-First Testing for AI Systems: Build Confidence Before Production

The Problem: Testing AI Is Expensive

You build an AI feature. You want to test it. So you call OpenAI's API in your test suite.

Problem: Every test run costs money. OpenAI charges per token. If your test suite has 50 tests and each one calls GPT-4, you're spending dollars per test run. Your CI/CD pipeline becomes expensive. Developers stop running tests locally.

Worse: LLMs are non-deterministic. The same prompt returns different answers. Your test passes one time and fails the next. You can't trust your test results.

The solution is mock-first testing. Build your tests against mocks. Run them fast, free, and deterministically. Then, separately, run evaluation gates against real LLMs to catch regressions.

Three Layers of Testing

Think of AI testing as three layers stacked on top of each other.

Layer 1: Mock Tests (Every Commit)

Your test suite runs against deterministic mocks. Fast, free, reliable. These run on every commit.

Layer 2: Integration Tests (Before Deploy)

Run tests against real APIs (Salesforce, OpenAI, etc.) in a staging environment. Costs money but catches real-world issues.

Layer 3: Evaluation Gates (Holdout Testing)

Before shipping, run a separate evaluation suite on holdout test cases with real LLMs. Measure quality (accuracy, safety, latency) against thresholds.

Most developers skip straight to layer 3 and wonder why their CI is slow and expensive. Start with layer 1. Only call real LLMs when you have to.

Mock-First in Practice: SupportFlow Mini

SupportFlow Mini is a support ticket router. It takes a customer ticket, calls OpenAI to generate a recommendation, routes it to a human reviewer, and delivers the result to a ticketing system.

Here's how we test it without calling OpenAI in every test run.

Step 1: Define the Contract

First, we define what our AI module should output. We use Pydantic to enforce the shape and types.

from pydantic import BaseModel

class TicketRecommendation(BaseModel):
    category: str  # "urgent", "support", "sales", "billing"
    priority: int  # 1-5
    confidence: float  # 0.0-1.0
    reason: str

# Your AI must always return this shape.
# Pydantic validates it. No surprises.

This contract is the bridge between your mock and your real LLM. Both must return the same Pydantic shape. If they don't, your tests catch it.

Step 2: Build Mock Implementations

Create a mock AI provider that returns fixed, deterministic responses.

class MockAIProvider:
    def recommend(self, ticket: Ticket) -> TicketRecommendation:
        # Deterministic logic, no API calls
        if "urgent" in ticket.title.lower():
            return TicketRecommendation(
                category="urgent",
                priority=5,
                confidence=1.0,
                reason="Urgent keyword detected"
            )
        return TicketRecommendation(
            category="support",
            priority=2,
            confidence=0.8,
            reason="Default routing"
        )

class RealAIProvider:
    def recommend(self, ticket: Ticket) -> TicketRecommendation:
        # Call OpenAI, parse response, validate with Pydantic
        response = openai.ChatCompletion.create(...)
        # Extract JSON, validate schema
        return TicketRecommendation(**parsed_response)

Both providers implement the same interface. Your tests accept a provider as a dependency. Inject the mock in tests, the real one in production.

Step 3: Write Deterministic Tests

Now your tests run against the mock. Fast, free, deterministic.

def test_urgent_ticket_routed_correctly():
    # Arrange
    provider = MockAIProvider()
    router = TicketRouter(ai_provider=provider)
    urgent_ticket = Ticket(title="URGENT: Database down")

    # Act
    result = router.route(urgent_ticket)

    # Assert
    assert result.recommendation.category == "urgent"
    assert result.recommendation.priority == 5
    assert result.routed_to == "escalation_team"

def test_normal_ticket_routed_to_support():
    provider = MockAIProvider()
    router = TicketRouter(ai_provider=provider)
    normal_ticket = Ticket(title="How do I reset my password?")

    result = router.route(normal_ticket)

    assert result.recommendation.category == "support"
    assert result.routed_to == "support_team"

Every test passes or fails consistently. Run them 100 times in a row, same results every time. No flaky tests, no API costs.

Step 4: Evaluation Gates (Production Confidence)

Before shipping to production, you run a separate evaluation suite against real LLMs. This is intentional and measured, not every test run.

# evaluation/test_quality_gates.py
# This runs BEFORE shipping, not on every commit

def test_recommendation_accuracy():
    """Run 50 holdout tickets against real OpenAI."""
    provider = RealAIProvider()
    router = TicketRouter(ai_provider=provider)

    holdout_tickets = load_holdout_set()  # 50 real tickets with human labels
    correct = 0

    for ticket, expected_category in holdout_tickets:
        result = router.route(ticket)
        if result.recommendation.category == expected_category:
            correct += 1

    accuracy = correct / len(holdout_tickets)

    # Gate: block shipping if accuracy drops below 92%
    assert accuracy >= 0.92, f"Accuracy {accuracy} below threshold"

def test_latency():
    """Ensure recommendations come back in <2 seconds."""
    provider = RealAIProvider()
    router = TicketRouter(ai_provider=provider)

    for ticket in load_holdout_set():
        start = time.time()
        router.route(ticket)
        elapsed = time.time() - start
        assert elapsed < 2.0, f"Recommendation took {elapsed}s"

These tests run once before deployment. They're allowed to be slow and expensive because they only run when you're about to ship. They catch regressions. They give you confidence.

The Benefits

✓Fast feedback: Mock tests run in milliseconds. Developers get instant feedback.
✓Cheap CI: No API calls per test run. Your CI bill stays low.
✓Deterministic: No flaky tests. Same input always produces same output.
✓Production gates: Evaluation tests catch regressions before shipping.
✓Clear contracts: Pydantic forces consistent schemas between mock and real.

Common Pitfalls

Mock Diverges from Reality

Your mock returns "urgent" for every ticket with a keyword, but the real LLM is more nuanced. Fix: Make mocks smarter or accept that they're simplified. Run integration tests to catch the gap.

Evaluation Gates Are Too Loose

Your evaluation tests pass even though the model is degrading. Fix: Set thresholds based on production needs, not wishful thinking. If 92% accuracy matters, enforce it.

Skipping Evaluation Entirely

You ship when all mock tests pass. But the real LLM behaves differently. Fix: Always run holdout evaluation before shipping to production.

See It in Action

SupportFlow Mini demonstrates this pattern end-to-end. Check out the case study to see the full architecture, including how mocks feed into evaluation gates and production deployment.

SupportFlow Mini Case Study

Key Takeaways

✓Mock-first testing avoids expensive LLM calls in every test run while keeping tests deterministic.
✓Pydantic contracts ensure mocks and real implementations return the same shape.
✓Evaluation gates run separately before shipping, catching regressions against real LLMs.
✓Three layers: mock tests (fast, every commit), integration tests (staging), evaluation gates (pre-production).

Building AI systems and want to discuss testing strategies? Let me know.

Contact Me