Cleaning up you test suite with pytest parametrize
I admit I jumped on the pytest ship quite late, after a few years of relying on the good ol’ unittest module, and for a good part they worked reasonably well for me, for little apparent incentive to something else.
Over time I started using pytest more and more as I discovered quite a few interesting approaches available with it, which are out of unittest reach.
Writing a test
A single test function can be divided in three steps:
- Arrange: create the tests conditions and initialize the environment in which you want run the subject of the test
- Act: execute the function you want to test
- Assert: verify the function outcome
The hardest part of writing a test is by far the Arrange one.
If you can get away with a limited setup in purely unit tests, as you move one step up in the testing pyramid, and you write integration or service tests, creating the correct conditions can be non trivial.
The risk with this is to put too many assertions in a single test function to “spare” too many tests sharing the same (or similar) test arrangements.
A slightly better solution is to share the arrangement between a few tests by calling a common function, but this can lead to make the tests less readable in the future due to the extra level of indirection.
There is one thing computers are good at: executing repeatedly the same code against different inputs.
What if we apply this to executing tests?
That’s the idea of parametrize:
Thanks to how python works, you can treat a function as an input object to another function and let the latter alter the former or execute it with additional parameters (a pattern implemented by python decorators).
Parametrize uses this pattern to call our single test function using the list of inputs provided as arguments to effectively create (and the tests log reflects this) a list of test functions “generated” at runtime sharing the same body.
The bonus of this approach is that you can concisely define a set of input and expectations, yet they are very clearly isolated from the test function body; on the contrary calling shared functions in multiple concrete test functions bury the conditions and expectations in the code making it difficult to quickly isolate them.
A more complex example:
There are two things which one must be aware when using parametrize:
- each test function run is executed in isolation, as if hey were different functions (as you would expect), so the arrange part is also execute multiple times. If it’s time consuming this will make your test suite slower very quickly as you add input combinations
- you might to adapt the assertions depending on the expected outcome (if you are testing an email send function you may want to check the email headers of the delivered email the send is successful, which you can’t when testing for send failures): this can get out of hand very quickly and creating a mess of conditions and assertions, I advise against doing more than a single if to distinguish the normal tested function flow and the error handing; take this as signal that you are forcing too many unrelated assertions in a single test and evaluate whether create different test functions or test some of the assertions in other test functions.
Don’t do this!:
Interesting parametrize features
Parametrize allows for very complex scenarios and I encourage you to check the documentation for the details.
Still there are a couple of features worth mentioning
Assign ids to each combination
By default parametrize creates the test function run name by combining the function name and the parameters set (to something like
test_timedistance_v0[a0-b0-expected0]) which might be self explanatory. To customize this you can use the ids argument to provide a terser name
Tests with automatic and manually specified ids:
Generated test names:
Mark parameters combination
One of the great pytest features are marker which allow to decorate test functions for different behavior (parametrize itself is a marker). You can use marks on a parametrize decorated function as you would normally do, but you can also decorate single parameter set, using the pytest.param function, which you can also use to define a single Id without defined one for each combination
One of the signatures of pytest is to completely embrace the dynamic nature of python, which allows it to work on a “meta” level, by enriching the test behaviors working outside the test functions, thus separating very clearly the boundaries and creating a test suite easier to understand and extended.
Parametrize is a perfect example of the pytest approach and I advise you to experiment with it, as it will greatly improve your test writing experience.