We really don’t want to break things that work, it’s super frustrating. We test to make sure that any changes we make do not break another piece of the code in our applications. Of course manual or exploratory tests are important but the dream is to get this stuff fully automated.

We make some changes and boom, these random artifacts appear.

That said, tests have a price, and while developers like to think of more tests as better I prefer to think of them as weight. They add robustness to a well established process but they also make deployments more time consuming and they take time to change. My preference with automated tests is to focus on risk. The more risk the more tests, the less risk, the less tests.

What is PyTest

You could write the test 100% manually but others have already made the job a bit easier. Python has a built in unit test module but most people prefer the 3rd party module named pytest. Pytest will collect all python files named “test_something” and manage the actual run time. This way all you need to do is tell python and pytest where your tests are located with a command like this:

Basic running

Python3 -m pytest -v tests

So in English I am asking for, python version 3, run pytest verbosily so I can see what’s going on and the tests I want to run are located in the “tests” directory. This will give an output something like this:

Collected 4 items
tests/test_pt.py::test_base FAILED [ 0%]
tests/test_pt.py::test_base_many[1-1] PASSED  [ 50%]
tests/test_pt.py::test_base_many[2-1.4142] PASSED [ 75%]
tests/test_pt.py::test_base_many[4-2] PASSED [ 100%]

In the event you are running overnight processes or it’s a big test with a potentially large output, it’s nice to throw a >> test-output.txt option and have an output saved.

Python3 -m pytest -v [the test directory/] >> output.txt

Hint: don’t forget the / to tell pytest not just the name of the directory but to look inside the directory.

Fixtures

My tests often require a specific environment and data to be available prior to running. Pytest allows you to decorate a function, that’s the @ symbol, with a fixture. This is basically an environment which is to be set up prior to running another test. In the case below we are building a simple database as a fixture.

@pytest.fixture
def database():
    conn = sqlite3.connect(':memory:')
    yield conn
    conn.close()

In my case I use this for setting up a new working directory for project outputs and managing my login sessions. ( The yield command is kind of like return except it does not need to be re-run all the time )

Skipping and Marking

When developers think the tests are taking too long they usually begin skipping them; or even worse they are considered untrustworthy or flakey. So it is important to be able to quickly skip and selectively run set tests.

There are two ways to handle selective testing in pytest. Skips and marks. In both cases we can use a decorator to assign skips and marks to our test functions. Here I am setting a skip if there is no continuous integration environment set.

@pytest.mark.skipif(not is_concourse, reason='no concourse environment ready')
def test_something():
    n = 2
    expected = 4
    value = n * 2
    assert expected == n

Test Failures

I’m running a test suite which takes over 10 hours to run. I’ve recently hit a few memory errors which mean I’ve had to re-run the whole thing. However pytest maintains its cache so that you can optionally re-run the tests with only the failures from the previous run. You do this by using the –lf command after pytest.

Python pytest --lf thedirectory/of/the/tests/

Resources

An awesome post about debugging pytest. I was originally going to write some more about this but it is quite a complete post, if your interested I recommend giving it a read.

There is of course more to pytest and the documentation is available here, I hope this was a brief window into testing the builds of geospatial tools.

Unrelated but while fixing up these tests I’ve found this article to be very helpful when making Pull Requests to my companies test suite.