Mocking mocking (or "Why I am learning to hate isolating the unit under test")

Mocking basically sucks.

There. I said it.

Using mocks in your tests almost always results in fragile tests.

Why?

A case against mocking

Let’s say that you’re working on a hypothetical (simplified) financial application:

class AccountTest < ActiveSupport::TestCase
  setup do
    @balance = 42.00
    @account = factory_to_create_a_test_account :with_balance => @balance
    @mock_bank_manager = mock(BankManager)
  end

  def test_overdrawing_account_sends_notification
    @mock_manager.expects(:notify_of, :overdrawn_account, :amount => 1)
    @account.withdraw(@balance + 1)
  end
end

The above example is attempting to illustrate how mocks can be useful for specifying a causal relationship. “Overdrawing the account” results in “notifying the bank manager of the amount over balance”.

Mocking seems like such a natural and even expressive way to design an API. It’s, quite literally, behavior driven development: your tests/specifications, where you are mocking the API, are helping you design the API itself! That’s terrific.

So let’s pretend that, like a good TDDer, I’ve gone and implemented the BankManager class.

class BankManager
  def notify_of(event_type, options = {})
    case event_type:
    when :notify_of
      # send the notification
    when ...
      ...
    end
  end
end

Perhaps I even wrote a test/spec for BankManager. But, then, in a fit of refactoring rage, I decide that I must have a second argument to BankManager#notify_of:

class BankManager
  def notify_of(event_type, account, options = {})
    case event_type:
    when :notify_of
      # send the notification
    when ...
      ...
    end
  end
end

Our old friend AccountTest above will still pass because it’s using a mock. However, the API to BankManager has changed; we want AccountTest#test_overdrawing_account_sends_notification to fail!

So, in short, mocking an internal API is a recipe for pain and (Ni!) woe.

  1. After you implement the API, you still have mocks lying around in the test that helped you mock out the API design in the first place. This test is now fragile. If the API changes, the tests containing the mocks will still pass!
  2. You can double back and replace those mocks with actual calls to the API but you just made more work for yourself

Where mocking makes sense

In my experience, the only place that mocks have served me at all well is when I’m interfacing with an external service from a unit test. I certainly don’t want my unit test invoking services beyond my own system. I usually write an interface layer between my business logic and the external service. In my unit tests, I then mock the interface layer only. Testing integration with the external service, predictably, becomes a chore solely for the integration test.

We have the technology. We can rebuild it.

I believe that there is a better path: one that will let us have our mocking cake but now force us to eat brittle tests. I’ve had some ideas about this on the back burner for about a year now. However, I hope to have something concrete to discuss in a few weeks.

Posted by evan on Monday, January 25, 2010

blog comments powered by Disqus