Lecture overview:

• Total time: ~55 minutes (tight!)
• Prerequisites: Students have JUnit basics, testing principles from earlier lectures
• Connects to: A4 (testing with Mockito), prior readability work, mutation testing grading

Key framing:

• We're about to scale up to larger codebases
• As systems grow, testing becomes harder—dependencies multiply
• This lecture gives you tools to keep tests fast, focused, and reliable
• Like L13 (AI coding assistants): tools change, responsibility doesn’t
• If you can’t evaluate a test (or a mock), you can’t trust it—AI or not

Structure:

• Three pillars: Fast, Reliable, Finds Bugs (~10 min)
• Input generation strategies (~5 min)
• Test scope spectrum (unit/integration/E2E) (~10 min)
• The dependency problem (~5 min)
• Test doubles: stubs, fakes, spies (~10 min)
• Mockito framework (~10 min)
• Tradeoffs (~5 min)

→ Transition: Let's start with the learning objectives...

CS 3100: Program Design and Implementation II

Lecture 15: Test Doubles and Isolation

©2025 Jonathan Bell, CC-BY-SA

Context from prior lectures:

• L5: Readability—tests are documentation that must be readable
• L7-8: Design for change—good design makes testing easier
• Assignments: Your test suites are graded on mutation testing—can they detect bugs?

Today's big question: How do we test code that depends on databases, networks, hardware?

→ Transition: Here's what you'll be able to do after today...

Learning Objectives

After this lecture, you will be able to:

1. Distinguish between unit, integration, and end-to-end tests
2. Explain the challenge of testing code with external dependencies
3. Identify properties of high-quality individual tests (hermetic, clear, non-brittle)
4. Differentiate between stubs, fakes, and spies as types of test doubles
5. Apply mocking frameworks to generate test doubles
6. Evaluate the tradeoffs of using test doubles
7. Apply AI coding assistants to generate test plans and test doubles

Time allocation:

• Three pillars + input generation (~10 min)
• Objectives 1-2: Test scope + dependency problem (~15 min)
• Objective 3: properties of good tests (~5 min, embedded)
• Objective 4: Stubs, fakes, spies (~10 min)
• Objective 5: Mockito framework (~10 min)
• Objective 6: Tradeoffs (~10 min)

Why this matters now:

• Next lectures scale up to larger codebases with more dependencies
• A4 requires Mockito for testing services with repository dependencies
• Your test suites are graded on bug detection—tests must actually catch bugs!
• We’ll weave AI throughout: where it helps (boilerplate, edge cases) and where it’s risky (weak assertions, unrealistic mocks)

→ Transition: First, what makes a test suite "good"?

What Makes a Good Test Suite?

Fast, reliable, and finds bugs—and individual tests must be hermetic, clear, and non-brittle.

🎯

Finds Bugs

Actually detects defects in the code

🔒

Is Reliable

Passes when code is correct, fails when it's not

⚡

Is Fast

Runs quickly so you actually run it

Suite goals

• Find bugs
• Run automatically
• Be cheap to run

Individual tests

• Hermetic
• Clear + debuggable
• Not brittle (avoid unspecified behavior)
• Use public APIs only

The three pillars of a good test suite:

1. Finds Bugs:

• A test that passes regardless of bugs is useless
• 100% coverage with no assertions = 0% bug detection
• We need tests that actually verify correct behavior

2. Is Reliable:

• No "flaky" tests that randomly pass or fail
• Deterministic: same code → same result
• When it fails, you trust it found a real bug

3. Is Fast:

• If tests take 10 minutes, you won't run them often
• Fast feedback = catch bugs early
• Slow tests = bugs slip through

The challenge: These goals can conflict! We'll explore each pillar and see how different test types make different tradeoffs.

Module 11 connection (test adequacy):

• A good suite has goals (find bugs, automatic, cheap)
• A good test has properties (hermetic, clear, non-brittle, public APIs only)
• We’ll keep returning to these as we talk about doubles/mocks and AI

→ Transition: Now let's explore each pillar...

⚡ What Makes Tests Fast (or Slow)?

I/O dominates—memory is microseconds, network is milliseconds.

Think about it: what operations take time in a test?

Fast (microseconds)

• Creating objects in memory
• Calling methods
• Arithmetic, string operations

Slow (milliseconds to seconds)

• File system I/O
• Network calls (APIs, databases)
• Starting processes/containers

Pause and ask: "What operations in your code take the most time?"

The answer: I/O dominates. CPU operations are nanoseconds; disk is milliseconds; network is tens to hundreds of milliseconds.

Implication for testing:

• Tests that stay in memory = fast
• Tests that hit disk, network, external services = slow
• If you want fast tests, avoid I/O

→ Transition: Now let's think about reliability...

🔒 What Makes Tests Reliable (or Flaky)?

Control all state → deterministic. Depend on external world → flaky.

A "flaky" test sometimes passes, sometimes fails — same code!

Reliable (deterministic)

• Same inputs → same outputs
• No shared mutable state
• No timing dependencies

Flaky (non-deterministic)

• Network timeouts
• Race conditions
• External service state

Empirical studies find a few causes dominate flakiness (Luo et al., FSE 2014):

• Async wait / timeouts: ~37%

• Test order dependency: ~17%

• Concurrency: ~17%

• Resource leaks: ~10%

• Network: ~9%

• Other: floating point, random, unordered collections

Pause and ask: "Have you ever had a test that passed locally but failed in CI? Why?"

Common causes of flakiness:

• Network: API might be slow, rate-limited, or down
• Timing: "wait 2 seconds" might not be enough
• Shared state: Another test changed the database
• Environment: Different OS, timezone, locale

Module 11 connection:

• Good tests are hermetic (setup/teardown; leave no trace) → prevents test-order dependency
• Avoid Thread.sleep(...)-style tests → async-wait is the #1 flaky cause in empirical studies
• “Test smells” often correlate with flakiness (timing assumptions, shared state)

Implication for testing:

• Tests that control all state = reliable
• Tests that depend on external world = flaky
• If you want reliable tests, isolate from external state

→ Transition: Now let's think about bug detection...

🎯 Measuring Bug-Finding: Coverage

Coverage measures what code ran, not whether it's correct.

Code coverage measures what code your tests execute

• Statement coverage: % of lines executed

• Branch coverage: % of decision paths taken (if/else, loops)

But coverage has a fatal flaw...

Coverage is a starting point:

• Helps find code that's never tested
• If a line is never executed, bugs there can't be found
• Higher coverage = more code exercised

But coverage doesn't guarantee bug detection:

• You can execute code without checking if it's correct
• A test with no assertions achieves coverage but catches nothing

→ Transition: Let's see why coverage alone isn't enough...

Coverage Can Still Miss Bugs

100% branch coverage ≠ 100% tested behaviors.

Even 100% branch coverage does not mean you tested every behavior.

public int magic(int x, int y) {
    int z;
    if (x != 0) { z = x + 10; }
    else { z = 0; }

    if (y > 0) { return y / z; } // BUG: can divide by zero
    else { return x; }
}

@Test void t1() { assertEquals(2, magic(1, 22)); }   // covers x!=0, y>0
@Test void t2() { assertEquals(0, magic(0, -10)); }  // covers x==0, y<=0

// 100% branch coverage… but magic(0, 5) throws ArithmeticException

Module 11 takeaway: "Every branch executed" is not the same as "every behavior tested."

What went wrong? The tests have strong assertions (assertEquals)—but they missed the key input combination: (x == 0) AND (y > 0).

The lesson: Coverage tells you which code ran, but not which inputs you chose. Finding the right inputs is a separate skill.

→ Transition: So where do good test inputs come from?

Where Do Test Inputs Come From?

Spec-driven and code-driven strategies complement each other.

Two fundamental strategies for choosing test inputs:

Spec-Driven

(from requirements)

• What should the code do?
• Boundary values, equivalence classes
• "2 cups → should convert to 32 tbsp"

Code-Driven

(from implementation)

• Look at branches, paths
• Choose inputs to exercise each path
• Coverage guides what's missing

AI-assisted (new reality): ask for equivalence classes + boundaries… then you validate the oracle.

The input selection problem:

• There are infinitely many possible inputs to any method
• We can't test them all—need to choose wisely
• These two strategies complement each other

Spec-driven:

• Read the spec: "converts cups to tablespoons"
• What are interesting inputs? 0, 1, fractional, negative?
• Boundary values: max precision, overflow cases

Code-driven:

• Look at the if-statements and loops
• Pick inputs that go down each branch
• Coverage tells you what code ran, helps find untested paths

Best practice: Start with spec-driven (what SHOULD it do), then use coverage to find gaps (what did I MISS testing).

Module 11 connection (equivalence classes):

• Example: ZIP code lookup has classes like “not 5 digits”, “5 digits but invalid”, “valid with one place name”, “valid with multiple”
• Boundary testing: barely legal / barely illegal inputs (4 digits vs 5 digits vs 6 digits, non-numeric strings, leading zeros)

AI connection (L13):

• AI is good at enumerating plausible equivalence classes and boundary cases quickly
• But the Evaluate step is on you: are these classes actually relevant to the spec? are expected results correct?

Beyoncé rule (Software Engineering at Google):

• “If you liked it you should have put a test on it” — ship features with tests so they keep working during maintenance

→ Transition: Let's start with what you already know...

Example-Based Testing: What You Know

You choose specific inputs from requirements or by examining code.

You write specific test cases with specific inputs

@Test
void shouldConvert2CupsTo32Tablespoons() {
    ExactQuantity twoCups = new ExactQuantity(2, Unit.CUP);

    Quantity result = twoCups.convertTo(Unit.TABLESPOON, registry);

    assertThat(result.getAmount()).isEqualTo(32);  // 2 cups × 16 tbsp/cup
}

You choose inputs from requirements or by examining the code

Example-based testing:

• Most common form of testing
• You pick specific inputs and expected outputs
• Clear documentation of behavior

Where do the inputs come from?

• From requirements: "should convert cups to tablespoons" → try 1, 2, 0.5 cups
• From code: See an if-statement for negative values? Add a test for -1 cups
• From coverage: See an untested branch? Figure out what input reaches it

The challenge: You're limited by your imagination. What cases did you forget?

→ Transition: Parameterized tests help you test more cases efficiently...

Parameterized Tests: Same Logic, Many Inputs

Write the test logic once; run it with many inputs.

From the HW3 handout — test the same behavior with different data

static Stream<Arguments> toStringTestCases() {
    return Stream.of(
        Arguments.of(1, "Preheat oven", "1. Preheat oven"),
        Arguments.of(2, "Mix ingredients", "2. Mix ingredients"),
        Arguments.of(10, "Serve warm", "10. Serve warm"),
        Arguments.of(99, "Final step", "99. Final step"));
}

@ParameterizedTest(name = "step {0}: \"{1}\" -> \"{2}\"")
@MethodSource("toStringTestCases")
void shouldFormatCorrectly(int stepNumber, String text, String expected) {
    Instruction instruction = new Instruction(stepNumber, text, List.of());
    assertThat(instruction.toString()).isEqualTo(expected);
}

Parameterized tests:

• Write the test logic once
• Run it with many different inputs
• JUnit generates a separate test for each input
• Great for boundary conditions: 1, 2, 10, 99...

When to use: Testing the same behavior with different data—boundary values, edge cases, different valid inputs.

→ Transition: What if we could generate inputs automatically?

Property-Based Testing: Describe Properties, Generate Inputs

Describe invariants; let the framework find counterexamples.

JUnit-QuickCheck (out of scope for this class, but worth knowing about)

@Property
void scalingPreservesRatios(
        @ForAll @IntRange(min=1, max=100) int flour,
        @ForAll @IntRange(min=1, max=100) int sugar,
        @ForAll @DoubleRange(min=0.1, max=10) double scaleFactor) {
    Recipe original = recipeWith(flour, sugar);
    Recipe scaled = original.scale(scaleFactor, registry);

    // Property: ratio of flour:sugar should be preserved
    double originalRatio = flour / (double) sugar;
    double scaledRatio = getFlour(scaled) / (double) getSugar(scaled);
    assertThat(scaledRatio).isCloseTo(originalRatio, within(0.001));
}

Framework generates hundreds of random inputs satisfying constraints

Property-based testing:

• Describe a PROPERTY that should hold for ALL inputs
• Framework generates hundreds of random inputs
• Finds edge cases you'd never think of

Example property: "Scaling a recipe preserves ingredient ratios"

Why out of scope: Requires thinking about properties, not just examples. Valuable skill, but we're focusing on fundamentals.

→ Transition: Fuzzing takes automatic generation further...

Fuzzing: Generate LOTS of Inputs

Generate millions of inputs to find crashes you'd never imagine.

Automatically generate inputs to find crashes and bugs

Graybox Fuzzing (used by security researchers):

• Start with sample inputs, randomly mutate them
• Monitor code coverage to guide mutation
• If mutation covers new code → keep it, mutate further
• Run millions of inputs per second

Google's OSS-Fuzz has found 10,000+ bugs in open-source projects

Fuzzing:

• Originally for security testing (finding crashes, buffer overflows)
• Now used for general robustness testing
• "Graybox" means it uses coverage info to be smarter

Why mention it?

• Awareness of professional testing techniques
• If you work on security-critical code, you'll encounter this
• Shows the spectrum: manual → parameterized → property-based → fuzzing

For this class: Focus on example-based and parameterized testing. These are your daily tools.

→ Transition: Now that we know where inputs come from, there's another trap to avoid...

The Testing “Vibe” Trap

Passing tests + high coverage ≠ tests that detect bugs.

The failure mode looks just like “vibe coding” (L13), but for tests:

1. Ask AI to generate tests
2. Tests pass, coverage goes up
3. Ship it

The problem: many generated tests have weak oracles, duplicate coverage (many tests checking the same thing), or are flaky/brittle.

L13 connection: evaluating “it runs” is not enough.\n \n Testing equivalent: evaluating “tests pass” / “coverage is high” is not enough.\n \n How we catch this in CS3100: mutation testing. If a bug can be introduced and your tests still pass, your tests aren’t doing their job.\n \n → Transition: So what does a strong test actually assert?\n

Strong Oracles Beat More Tests

A test is only as good as its oracle.

A test oracle is the rule that decides when a test should fail.

Weak oracles

• “It doesn’t crash”
• “Result is not null”

• Overuse of any() / vague verifications

Strong oracles

• Exact expected output
• All observable effects verified
• Meaningful properties + boundaries checked

Module 11 takeaway: oracle strength is often the difference between “high coverage” and “finds bugs.”\n \n AI implication: AI is great at generating tests, but it will often default to weak oracles unless you specify what correctness means.\n \n L13 principle: if you can’t explain the oracle, you can’t evaluate the test.\n \n → Transition: Good inputs + strong oracles. Now let's see how different test types make different tradeoffs...

The Tradeoff Triangle

Fast + Reliable vs. Finds Real Bugs—different tests make different tradeoffs.

Now we see the challenge: these goals conflict!

	⚡ Fast	🔒 Reliable	🎯 Finds Real Bugs
Test in memory only	✓	✓	Maybe misses I/O bugs
Test with real services	✗	✗	✓ Catches integration bugs

Different test types make different tradeoffs

The fundamental tension:

• Fast + Reliable = stay in memory, avoid I/O → but might miss real integration bugs
• Find real bugs = test real services → but slow and flaky

The solution: Different types of tests for different purposes!

• Some tests optimize for speed/reliability (unit tests)
• Some tests optimize for realism (E2E tests)
• Need a mix of both

→ Transition: Let's look at the different test types...

Unit Tests: ⚡ Fast + 🔒 Reliable

Test one unit in isolation: fast, focused, deterministic.

Test a single "unit" — typically one class — in isolation

• ⚡ Fast: Run in milliseconds (no I/O, no network)

• 🔒 Reliable: No external state = deterministic

• ✓ Focused: When it fails, you know exactly where to look

The challenge: achieving isolation when code has dependencies

Connecting to our pillars:

• ⚡ Fast: No I/O, no network → microseconds per test
• 🔒 Reliable: No external services → always deterministic
• 🎯 Finds bugs: In the unit's logic, but might miss integration bugs

The isolation problem:

• If RecipeService calls OcrPort.extractText()...
• How do we test the service without real OCR?
• This is the central question of today's lecture

Readability connection (L5): Unit tests are documentation. A reader should understand the expected behavior from the test name and structure.

→ Transition: Let's see an example from the handout...

Unit Test Example: From HW3 Handout

If it fails, you know exactly where the bug is.

Testing Instruction.toString() — one class, no dependencies

@ParameterizedTest(name = "step {0}: \"{1}\" -> \"{2}\"")
@DisplayName("should format as stepNumber. text")
void shouldFormatCorrectly(int stepNumber, String text, String expected) {
    Instruction instruction = new Instruction(stepNumber, text, List.of());

    assertThat(instruction.toString()).isEqualTo(expected);
}

// Test cases: (1, "Preheat oven", "1. Preheat oven")
//             (2, "Mix ingredients", "2. Mix ingredients")
//             (10, "Serve warm", "10. Serve warm")

Why this is a unit test:

• Tests ONE class (Instruction) in complete isolation
• No other components, no I/O, no network
• Runs in microseconds
• Zero external dependencies

The test is focused: If it fails, the bug is in Instruction.toString() — nowhere else to look.

→ Transition: Integration tests cover the middle ground...

Integration Tests: The Middle Ground

Test components together to catch seam bugs.

Test multiple components interacting

• ⚡≈ Moderate speed: Some I/O, but local

• 🔒≈ Mostly reliable: Controlled environment

• 🎯✓ Catches seam bugs: Serialization, protocols, formats

Do components communicate correctly? Agree on data formats?

The middle ground:

• ⚡ Moderate: File I/O is slower than memory, but faster than network
• 🔒 Mostly reliable: Local file system is predictable
• 🎯 Catches seam bugs: Issues at boundaries between components

When integration tests shine:

• Real Recipe + real JsonRecipeRepository + real file system
• Testing the "seams" between components
• Catching issues unit tests can't: serialization bugs, file format issues

→ Transition: Let's see a concrete example...

Integration Test Example: From HW3 Handout

Round-trip tests catch serialization and format issues.

Testing Recipe + JsonRecipeRepository + Jackson + file system together

@Test
@DisplayName("round-trip preserves recipe with all fields")
void roundTripPreservesRecipeWithAllFields() {
    Recipe recipe = new Recipe("test-id", "Chocolate Cake",
        new ExactQuantity(8, Unit.WHOLE),
        List.of(new MeasuredIngredient("flour",
            new ExactQuantity(2, Unit.CUP), null, null)),
        List.of(new Instruction(1, "Mix ingredients", List.of())),
        List.of());

    repository.save(recipe);  // Writes JSON to file system
    Optional<Recipe> loaded = repository.findById("test-id");

    assertTrue(loaded.isPresent());
    assertEquals(recipe, loaded.get());
}

Why this is an integration test:

• Tests MULTIPLE components working together
• Recipe → Jackson serialization → file system → Jackson deserialization → Recipe
• Actually writes files (I/O)
• Catches bugs in: JSON annotations, file paths, encoding issues

The tradeoff: Slower than unit tests, but catches real integration bugs.

→ Transition: At the far end of the spectrum...

End-to-End Tests: 🎯 Finds Real Bugs

Test the whole system as users experience it—slow but realistic.

Test the entire system as a user would experience it

• ⚡✗ Slow: Seconds or minutes per test

• 🔒✗ Flaky: Network glitches, timing issues

• 🎯✓ Realistic: Tests what users actually experience

The opposite tradeoff: sacrifice speed and reliability for realism

Connecting to our pillars:

• ⚡ Fast: NO — real I/O, real services, real waiting
• 🔒 Reliable: NO — external services can fail, timeout, change
• 🎯 Finds bugs: YES — catches real integration issues across the whole stack

Why we still need them:

• Catches bugs that only appear when everything works together
• Tests real user journeys
• But too expensive for edge case coverage

We'll cover E2E testing in depth in L30 (GUI Patterns and Testing).

→ Transition: Let's see a real E2E example from your project...

E2E Example: Cook Your Books Import-to-Export

One test, many systems—and many potential failure points.

One test, five systems: Image → OCR → Parser → Repository → Exporter

The Cook Your Books E2E workflow:

1. User photographs a page from their physical cookbook
2. Image sent to OCR service (external API) to extract text
3. IngredientParser converts raw text to structured Recipe objects
4. Recipe saved to JsonRecipeRepository (file system)
5. Recipe exported to Markdown for sharing

What can go wrong:

• OCR API timeout or rate limits
• OCR returns garbage text for low-quality images
• Parser fails on unusual ingredient formats ("a pinch of love")
• File system permissions or disk space issues
• Markdown formatting breaks on special characters

This tests the COMPLETE user journey — but takes seconds and involves external services.

→ Transition: Why not just write E2E tests for everything?

The Practical Mix

E2E tests for critical journeys; unit tests for edge cases.

Why not just write E2E tests for everything?

Imagine testing 15 edge cases for ImportService:

	Unit Tests	E2E Tests
Example	ImportService logic	Full OCR-to-export workflow
Time to run	~150ms total	~30 seconds
Flakiness	None	OCR API, file system
Debug time	Seconds	Minutes to hours

The math doesn't work:

• 15 edge cases × 2 seconds each (OCR call) = 30 seconds (best case)
• Unit tests: 15 edge cases × 10ms = 150ms
• Run unit tests on every save; E2E tests nightly

The right balance:

• Many unit tests for edge cases and logic (IngredientParser, Recipe scaling)
• Some integration tests for component interaction (Repository round-trip)
• Few E2E tests for critical user journeys (import cookbook → export recipe)

→ Transition: This ratio is captured in the test pyramid...

The Test Pyramid

Many unit tests, some integration, few E2E.

The pyramid shape is intentional:

• Many unit tests at the base (fast, cheap)
• Fewer integration tests in the middle
• Very few E2E tests at the top (slow, expensive)

Why this ratio matters:

• Edge cases and detailed logic: unit tests
• Component communication: integration tests
• Critical user journeys: E2E tests

Today's focus: Unit testing—specifically, how to achieve isolation.

→ Transition: So how do we take one big E2E test and break it into many small unit tests?

From Big Tests to Small Tests: Find the Seams

Replace dependencies with doubles to carve unit tests from E2E.

Start with a real user workflow, then carve out unit tests by replacing dependencies.

One big test (E2E)

• Slow + flaky (network, DB, timeouts)
• Hard to debug (“something failed”)
• Good for one critical journey

Many small tests (unit)

• Fast + reliable (in-memory)
• Each test targets one behavior/edge case

• Depends on test doubles for external services

The trick: identify dependencies (GitHub/API/DB/email) and replace them with stubs/fakes/spies/mocks.

Concrete workflow:\n 1) Write (or imagine) the E2E “happy path” once.\n 2) List the dependencies it touches (network, DB, filesystem, clock, randomness).\n 3) For each dependency, create a seam (interface / DI) so it can be replaced.\n 4) Unit-test the logic with doubles; keep a small number of integration/E2E tests to validate reality.\n \n This is why the dependency problem matters: it’s the engineering move that turns “few big tests” into “many small tests.”\n \n → Transition: Let’s look at a concrete dependency-heavy service.\n

The Dependency Problem

How do we test code that depends on databases, networks, hardware?

Consider SubmissionService from Pawtograder:

public class SubmissionService {
    private final GitHubService github;
    private final AutograderRunner autograder;
    private final NotificationService notifier;
    private final Database database;

    public SubmissionService(GitHubService github,
                             AutograderRunner autograder,
                             NotificationService notifier,
                             Database database) {
        this.github = github;
        this.autograder = autograder;
        this.notifier = notifier;
        this.database = database;
    }
    // ...
}

Notice the dependencies:

• GitHubService — fetches code from GitHub API
• AutograderRunner — executes tests in containers
• NotificationService — sends real emails to students
• Database — persists grades

Design for change connection (L7-8): This is dependency injection! We designed for testability.

The question: How do we unit test this without real GitHub, real containers, real email?

→ Transition: Let's see the method we want to test...

The Method Under Test

Testing side effects requires controlling dependencies.

public GradeResult processSubmission(Submission submission) {
    CodeSnapshot code = github.fetchCode(submission.repoUrl());
    TestResult result = autograder.runTests(code);

    if (result.allPassed()) {
        database.saveGrade(submission.studentId(), result.score());
        notifier.send(submission.studentEmail(),
                      "Your submission passed!", NotificationLevel.INFO);
    } else {
        notifier.send(submission.studentEmail(),
                      "Some tests failed", NotificationLevel.WARNING);
    }
    return new GradeResult(submission.studentId(), result);
}

To test this, we'd need real GitHub, real containers, real email!

What we want to test:

• If all tests pass, save the grade and notify success
• If some tests fail, notify failure (don't save grade)
• Correct student is notified

What we DON'T want:

• Real GitHub API calls (rate limits, network issues)
• Real container execution (slow, resource-heavy)
• Real email sending (spam students!)

The solution: Replace dependencies with test doubles

→ Transition: Enter test doubles...

Test Doubles: Stand-Ins for Real Dependencies

Stubs return canned answers; fakes work simply; spies record calls.

Stubs

Return canned answers

Fakes

Simplified implementations

Spies

Record what happened

From simplest to most sophisticated

The spectrum:

• Stubs: Dumbest—just return a value
• Fakes: Actually work, but simpler than real thing
• Spies: Record interactions for verification

Interview note: Some testing purists have precise definitions for these terms. The distinctions matter less than understanding when to use each.

→ Transition: Let's see each in action...

Stubs: Return Canned Answers

Ignore details you don't care about; return what you need.

class StubGitHubService implements GitHubService {
    private final CodeSnapshot fixedCode;

    public StubGitHubService(CodeSnapshot code) {
        this.fixedCode = code;
    }

    @Override
    public CodeSnapshot fetchCode(String repoUrl) {
        return fixedCode;  // Always returns the same code
    }
}

Ignores the repo URL, always returns sample code — that's fine!

Stubs are simple:

• Configure with a value
• Return that value when asked
• Ignore parameters we don't care about

Why this works:

• For testing "save grade when all tests pass"
• We just need some code to feed to the autograder
• Don't care about actual repo URL, authentication, etc.

Readability (L5): The stub makes the test's intent clear—we're testing the "passing" case.

→ Transition: What about verifying side effects?

Spies: Record What Happened (Decorator Pattern)

Wrap, record, delegate—verify interactions after the fact.

class SpyDatabase implements Database {
    private final Database delegate;  // Wraps a real implementation
    private boolean saveGradeCalled = false;
    private String savedStudentId = null;

    public SpyDatabase(Database realDatabase) {
        this.delegate = realDatabase;  // Decorator pattern!
    }

    @Override
    public void saveGrade(String studentId, int score) {
        this.saveGradeCalled = true;       // Record the call
        this.savedStudentId = studentId;
        delegate.saveGrade(studentId, score);  // Delegate to real impl
    }

    // Query methods for tests
    public boolean wasSaveGradeCalled() { return saveGradeCalled; }
    public String getSavedStudentId() { return savedStudentId; }
}

This is the Decorator pattern:

• Spy wraps a real Database instance
• Records the call, then delegates to the real implementation
• Can verify interactions AND have real behavior

Spies record:

• Which methods were called
• With what arguments
• How many times

Why we need this:

• processSubmission() has side effects (saving grade)
• Can't test just by checking return value
• Must verify it called database.saveGrade() with correct args

→ Transition: Now we can write a complete test...

A Complete Test with Hand-Rolled Doubles

Stubs + spies enable fast, focused tests.

@Test
public void savesGradeWhenAllTestsPass() {
    StubGitHubService stubGithub = new StubGitHubService(sampleCode());
    StubAutograderRunner stubAutograder = new StubAutograderRunner(
        new TestResult(true, 100));  // All tests pass, score 100
    SpyDatabase spyDatabase = new SpyDatabase();
    StubNotificationService stubNotifier = new StubNotificationService();

    SubmissionService service = new SubmissionService(
        stubGithub, stubAutograder, stubNotifier, spyDatabase);

    service.processSubmission(new Submission("student123", "repo-url", "email"));

    assertTrue(spyDatabase.wasSaveGradeCalled());
    assertEquals("student123", spyDatabase.getSavedStudentId());
    assertEquals(100, spyDatabase.getSavedScore());
}

Trace through the test:

1. Stub GitHub returns sample code
2. Stub autograder says all tests pass with score 100
3. Service should save the grade
4. Spy records that saveGrade() was called with correct args

This test is:

• Fast (no I/O)
• Focused (tests one behavior)
• Clear (the stubs make the scenario obvious)

But there's a problem...

→ Transition: Look at how much code we wrote!

The Pain of Hand-Rolling Test Doubles

Four classes for one test? That doesn't scale.

We wrote four separate classes just to test one method!

• StubGitHubService
• StubAutograderRunner
• SpyDatabase
• StubNotificationService

And we haven't tested failures, timeouts, edge cases...

Do we need a new stub class for every test result?

The boilerplate explosion:

• Each new test scenario might need different stub values
• Different spy recording for different verifications
• Lots of repetitive code

This is where mocking frameworks shine.

→ Transition: Enter Mockito...

Mockito: Test Doubles Without the Boilerplate

Generate test doubles at runtime with when() and verify().

A mocking framework generates test doubles at runtime

@Test
public void savesGradeWhenAllTestsPass() {
    // Create test doubles — Mockito generates these at runtime
    GitHubService mockGithub = mock(GitHubService.class);
    AutograderRunner mockAutograder = mock(AutograderRunner.class);
    NotificationService mockNotifier = mock(NotificationService.class);
    Database mockDatabase = mock(Database.class);

    // Configure stub behavior
    when(mockGithub.fetchCode(anyString())).thenReturn(sampleCode());
    when(mockAutograder.runTests(any())).thenReturn(new TestResult(true, 100));

    SubmissionService service = new SubmissionService(
        mockGithub, mockAutograder, mockNotifier, mockDatabase);

    service.processSubmission(new Submission("student123", "repo-url", "email"));

    // Verify spy recordings
    verify(mockDatabase).saveGrade("student123", 100);
    verify(mockNotifier).send(eq("email"), contains("passed"), any());
}

Notice the difference:

• No separate stub/spy classes
• Configure behavior inline with when().thenReturn()
• Verify interactions with verify()

A Mockito "mock" is really:

• A stub (when().thenReturn())
• AND a spy (verify())
• Combined into one object

→ Transition: Let's break down the key Mockito operations...

Mockito: Create, Configure, Verify

mock(), when().thenReturn(), verify()—the three operations you'll use most.

mock(Class.class)	Create a test double
when(...).thenReturn(...)	Configure stub behavior
verify(mock).method(...)	Check spy recordings
when(...).thenThrow(...)	Simulate exceptions

Mockito uses reflection to generate implementations at runtime

The four operations you'll use most:

1. mock() — create the double
2. when().thenReturn() — "when someone calls this, return that"
3. verify() — "assert this method was called"
4. when().thenThrow() — simulate failures

How Mockito works (simplified):

1. Uses Java reflection to examine the interface
2. Generates a new class implementing all methods
3. Default: methods return null/0/false
4. Records every call (spy)
5. Lets you configure returns (stub)

→ Transition: Argument matchers add flexibility...

AI for Test Doubles (Especially Mockito)

AI generates boilerplate; you evaluate whether mocks match reality.

AI assistants are great at generating boilerplate if you keep evaluation in the loop.

1. Plan first: write the behaviors you need to simulate and verify

2. Generate: ask AI to produce when(...)/verify(...) scaffolding

3. Review: does the mock reflect real dependency behavior?

4. Break it: introduce a bug — does the test fail?

L13 workflow applied to Mockito: Identify → Engage → Evaluate → Calibrate → Tweak → Finalize.\n \n A good prompt is a spec:\n - “Here is the method under test.”\n - “Here are the side effects I expect (save grade only on pass).”\n - “Here are the interactions that should not happen.”\n \n Common AI failure modes in mocks:\n - Overuse of any() leading to weak verification\n - Mock behavior that is impossible in reality (e.g., returning null when real code never does)\n - Verifying implementation details rather than behavior\n \n → Transition: Mockito matchers help you be precise without being brittle.\n

Argument Matchers: Flexible Matching

Match any value, exact values, or custom predicates.

// Return sample code for ANY repo URL
when(mockGithub.fetchCode(anyString())).thenReturn(sampleCode());

// Verify saveGrade was called with any student ID
verify(mockDatabase).saveGrade(anyString(), anyInt());

// Verify with custom condition
verify(mockNotifier).send(
    anyString(),
    argThat(message -> message.contains("passed")),
    eq(NotificationLevel.INFO)
);

anyString(), any(), eq(), argThat()

Common matchers:

• anyString(), anyInt(), any() — match any value
• eq(value) — match exactly this value
• argThat(predicate) — match if predicate returns true

When to use argThat:

• Don't care about exact message, just that it contains key info
• Testing that a timestamp is "recent" rather than exact

Rule: If you use any matcher in a call, ALL arguments must use matchers.

→ Transition: What about fakes?

Fakes: When You Need Real Behavior

When you need save-then-retrieve, use a working in-memory implementation.

class FakeUserRepository implements UserRepository {
    private final Map<String, User> users = new HashMap<>();

    @Override
    public void save(User user) {
        users.put(user.getId(), user);
    }

    @Override
    public User findById(String id) {
        return users.get(id);
    }

    @Override
    public List<User> findAll() {
        return new ArrayList<>(users.values());
    }
}

A working implementation — just simpler than the real database

When fakes are useful:

• Need to save and then retrieve (can't stub that easily)
• Testing multi-step workflows
• Repository patterns (your A4 uses this!)

Stub vs Fake:

• Stub: returns canned value, doesn't "work"
• Fake: actually stores/retrieves, but in-memory

A4 connection: You'll use fakes for in-memory repositories, mocks for OcrPort.

→ Transition: Let's talk about tradeoffs...

The Good: Why Test Doubles Work

Speed, determinism, isolation, and easy error simulation.

• ✓ Speed: Tests run in milliseconds

• ✓ Determinism: No flaky tests from real APIs

• ✓ Isolation: Failures point to the code under test

• ✓ Edge cases: Easy to simulate errors

// Simulating a GitHub API failure — one line!
when(mockGithub.fetchCode(anyString()))
    .thenThrow(new GitHubException("API rate limit exceeded"));

Speed: Thousands of tests in seconds. Run on every save.

Determinism: A stub always returns the same code. The real GitHub API might timeout or rate-limit you.

Isolation: When the test fails, the bug is in SubmissionService, not in network code.

Edge cases: How do you test GitHub being down? Wait for an outage? With a stub, one line.

But there are dangers...

→ Transition: The dark side of mocking...

The Dangerous: False Confidence

Tests prove code works with your doubles, not with real systems.

The core danger:

• Tests prove code works with YOUR test doubles
• The real GitHub API might rate-limit you
• The real autograder might run out of memory
• The real database might timeout

Example:

• Your stub returns code instantly
• Real GitHub takes 2 seconds and sometimes fails
• Does your code handle that?

AI makes this easier to fall into (L13):

• AI can generate a very convincing mock setup in seconds
• But AI is predicting what mocks usually look like — not what your real dependency actually does
• Your job is to evaluate: does the mock behavior match reality? are we testing behavior or just our own assumptions?

→ Transition: Another danger: brittle tests...

The Dangerous: Brittle Tests

Test behavior, not implementation details.

Brittle tests rely on unspecified behaviors:

// BRITTLE: Assumes Set iteration order (unspecified!)
assertThat(recipe.getTags().toString())
    .isEqualTo("[vegetarian, quick, healthy]");

// BRITTLE: Exact string match on unspecified message format
verify(mockNotifier).send(eq("email@example.com"),
    eq("Your submission passed!"), any());

// BETTER: Test the behavior, not the implementation
assertThat(recipe.getTags()).containsExactlyInAnyOrder(
    "vegetarian", "quick", "healthy");

Brittle tests rely on unspecified behaviors:

• Set/Map iteration order (not guaranteed!)
• Exact string formatting of messages
• Internal method names that could be renamed
• Order of operations that doesn't affect correctness

Brittle test symptoms:

• Tests break when you refactor, even though behavior is unchanged
• Passes on your machine, fails in CI (different JVM, OS)
• Spend more time fixing tests than writing features

The deeper problem:

• Are you testing BEHAVIOR or IMPLEMENTATION?
• "Tags should contain these values" vs "Tags.toString() equals this exact string"
• The first is behavior; the second is implementation detail

Readability connection (L5): Tests are documentation. If you're documenting implementation details, you're documenting the wrong thing.

→ Transition: Guidelines for when to use test doubles...

When to Use Test Doubles

If mock setup is more complex than the code, reconsider.

Use test doubles when:

• Dependency is slow or unreliable
• Need to simulate error conditions
• Dependency has side effects (emails, charges)
• Want to verify interactions

Be cautious when:

• Mock setup is more complex than code
• Verifying implementation, not behavior
• Mocking types you don't own

If mock setup is getting elaborate, consider an integration test instead

Good uses:

• Database, network, hardware
• Simulating timeouts, exceptions
• Verifying notifications were sent
• Avoiding side effects (charges, emails)

Bad signs:

• Mock setup longer than the test
• Mocking every little thing
• Mocking third-party libraries (you don't know their contract!)

The heuristic: If the mock setup is getting complex, that's a sign to write an integration test instead.

→ Transition: How does this connect to your assignments?

Connection to Your Assignments

Mutation testing grades whether your tests catch bugs, not just pass.

Your test suites are graded by mutation testing

• We introduce bugs (mutations) into your code

• Your tests should catch those bugs

• A test that passes with the bug present = weak test

It's not enough for tests to pass — they must detect bugs!

Mutation testing explained:

• We change ">" to ">="
• We delete a line
• We change a constant
• Do your tests catch it?

Why mutation testing is legit (Module 11):

• Research (hundreds of real faults studied) finds mutation score correlates with finding real bugs
• This has been replicated in multiple contexts, including student code

Equivalent mutants (important nuance):

• Some “undetected” mutants aren’t actually bugs — they’re behaviorally equivalent
• Deciding equivalence often requires human judgment (the spec may not care about the mutated detail)

Why this matters:

• You can have 100% coverage with useless tests
• Tests that just call methods without asserting = useless
• Tests that verify the wrong thing = useless

A4 connection:

• Test your services with mocks
• But make sure tests actually verify behavior
• Ask: "If this line were wrong, would my test catch it?"

→ Transition: AI can help with test generation...

AI for Test Planning (Plan Mode)

Plan first, generate drafts, then evaluate.

AI assistants are particularly good at generating drafts — especially when you start from a plan.

• Turning “behaviors to test” into test skeletons
• Enumerating edge cases (equivalence classes, boundaries)

• Generating Mockito boilerplate (when/verify)

• Creating test data and fixtures

Key rule (L13): only ask AI to produce what you can evaluate.

Plan mode idea: reduce your evaluation surface.\n \n Before generating code, write a short plan:\n

• Behaviors to test (pass/fail/edge cases)\n
• The oracle (what exactly proves correctness?)\n
• Doubles needed (what to stub, what to verify)\n \n Then ask AI to draft tests from the plan.

AI weaknesses:

• Might write tests that pass but don't verify correctly
• Mock configurations might not reflect reality
• Can generate tests that look good but test nothing

→ Transition: A quick trick: if you can’t write the test plan, you can’t evaluate the AI’s tests.

The Test-First Evaluation Trick

If you can't describe the oracle, you can't evaluate the AI's tests.

Before you ask AI for tests, answer this yourself:

“What would a good test check?”

• What behavior should change for pass vs fail?

• What’s the oracle (strong, not “doesn’t crash”)?

• What inputs hit boundaries / equivalence classes?
• What side effects must be verified?

If you can’t answer these, you’re in the “low familiarity” danger zone (L13): you can’t evaluate the output.

This is the single most useful self-check for AI-assisted testing.\n \n If you can answer it, then the AI can help you go faster:\n 1) Plan → 2) Generate → 3) Review → 4) Break it → 5) Validate mocks.\n \n → Transition: Let’s summarize.

Summary

• Test scope spectrum: Unit (fast/focused) → Integration → E2E (slow/complete)

• Good tests: hermetic, clear, non-brittle, with strong oracles

• Test doubles stand in for real dependencies

• Stubs return canned answers; spies record interactions; fakes are simplified implementations

• Mockito generates test doubles at runtime

• Tradeoffs: Speed/isolation vs false confidence/brittleness

• Mutation testing: Tests must detect bugs, not just pass (coverage helps find gaps)

• AI for testing: plan first, generate drafts, then evaluate (don’t “vibe test”)

Key takeaways for A4:

• Use Mockito to test services in isolation
• Mock repositories and external services (OcrPort)
• Verify behavior, not implementation details
• Ask: "Would this test catch a bug?"

Connection to larger themes:

• L5 Readability: Tests are documentation
• L7-8 Design for change: DI enables testability
• Mutation testing: Quality over coverage
• L13 AI: evaluation is the step that never changes

→ Transition: What's next...

Next Steps

• Reading: Lecture notes, Mockito documentation (linked on course site)

• Next lecture: Designing for Testability

Coming up:

• Larger codebases with more dependencies
• Design patterns that make testing easier
• GUI testing (L30)