I’ve written about the test pyramid before. It provides a guideline on the type of tests you need and how many. But recently, I got in a conversation on what the test pyramid actually means. Is it about code coverage? Or is it purely about the number of tests?
A Brief Recap
The test pyramid gives us guidance on how our test suite should look like. We should have less tests that use many components and more that test only one or two components:
The reason we follow this guidline is that the tests higher up the pyramid are usually slower to execute and break more easily. This means a longer feedback loop which slows developers.
The Testing Ice Cream Cone
The Testing Ice Cream Cone is when you have the inverse situation:
Lot’s of slow and brittle tests are detrimental to productivity. The different types of tests are necessary and they all provide value. But you want to keep track of your test suite to enable your developers to stay productive and to give them fast feedback of the changes they’re making.
But would code coverage be a better way of measuring the shape of your test pyramid?
An Extreme Situation
Let’s say you have a large number of unit tests that test only cover half of your application. On the other hand, you have a few end-to-end tests that cover more or less your entire application. This is what this would like in terms of both code coverage and number of tests:
I know, this is putting it extremely. But go with me for now. If we agree that the testing ice cream cone is bad, then we’re fine if we just count the number of tests. If we look at code coverage we have a problem. So which is it?
Let’s look at why the idea of the test pyramid exists again. The tests higher up the pyramid take a long time to run and traditionally break more easily. This slows down our development process.
So in this case, we’re still OK: we only have a few end-to-end tests but many unit tests. And that we have our entire application covered is great! So I would not say this is a problematic situation. The only thing the dev team might find cumbersome is that changes in the code that isn’t covered by unit tests requires them to run the slower end-to-end tests. But that could be an incentive to start writing unit tests for those parts.
The Inverse Extreme Situation
The other extreme would be having a few unit tests that provide great coverage but many end-to-end tests that provide little coverage:
However, I’m not sure how you would end up in a situation like this. It would be difficult to achieve. But if we were speaking hypothetically, it would be a suboptimal situation to be in. Not bad at all, because your unit tests have great coverage. But I would think that the number of the end-to-end tests can be reduced drastically by removing the ones that add little value.
The Answer?
So in the first case, we should start writing more unit tests. And in the other, we should reduce our end-to-end tests. This means that we transform our two pyramids to look alike:
Which is where you want to end up. But to answser the initial question: is the test pyramid about number of tests or about code coverage? Well, it’s basically about both and usually both will remain more or less in sync. Counting is probably easier to do, so if you can track that and use it to keep your pyramid in good shape, you should be good.