Here it is in PyCon US 2023
In this blog post series, we saw a bunch of different practices, and talked about how they will affect us by changing the properties of our tests:
- Strength - how well they catch bugs
- Maintainability - how easy it is to work with them
- Performance - how fast they run
The bug funnel is all about performance.
Testing a cohesive whole (implementation vs behavior) is about maintainability and strength.
Priorities are counter-intuitive
But how do we prioritize what we want to invest in?
First and foremost - the objective of tests is their strength.
We have tests so that they catch bugs.
The unintuitive thing is that this is not what we should prioritize when we work:
Start with making them maintainable, Then make sure they are fast enough
And then make them strong.
Why?
Here’s the thing.
Slow tests are weak, or at least they are EVENTUALLY weak.
Let’s say that, as a team, we decided that we are not willing to have tests that run for more than 30 minutes.
If, at some point the tests reach 30 minutes…
It becomes very difficult to add more tests.
So after enough time, there will be a lot of code that’s not tested well.
And the same thing happens with maintenance.
It’s more subtle, but if tests are not maintainable, it costs more to have them, and we end up creating fewer tests.
So again, they will be eventually weak.
And, maintainability issues can also make it difficult to handle performance.
An example we saw is test isolation and parallelization.
In other words:
Maintainability is a necessary condition for performance, and both are necessary conditions for strength.
Conclusion
So make maintainability the priority.
Testing a single fact, code design and all the other practices we covered.
When you have a choice to make - I suggest to go with the most maintainable option almost always.
Even at the cost of other things.
Because in the long run, that’s how we get tests that let us move fast, and have confidence in our code.
