Breaking the Gateway

Defining delivery requirements in the Geospatial domain; and my favourite bug discovery. Communication protocols enable systems to communicate information; these definitions are API’s; Application Programming Interface. These protocols must be well defined to function.

• Datetime formats to be used
• Types such as int, string, doubles etc
• Authentication methods

The list goes on within the computer science world geography isn’t normally in this conversation. While delivering a new API to load geospatial data to a PostGIS enabled DB I strumbled across an interesting problem.

• Geometry string complexity

While defining an API it was never considered that data transmission might be simple but large as is often the case with geometry. Loading the country of South Africa we managed to generate interesting system errors.

It was as if the system received the message, but the message wasn’t the one sent. This isn’t normal. Usually a signal is sent, and it either arrives or not. Typically it arrives in the same shape. I’ve seen a lot of exceptions to this rule with lost packets in AWS cp commands on large data but thats with data loading spanning hours.

Signal Loss

In this case I send a signal, its received immediately but doesn’t progress from the initial hit. I simplified the data and it began to work.

Negative Testing and the Gateway

Increasingly I rely on pytest to take these investigations further. I created 2. One to confirm the passing simply geometry works under idential circumstances. Another to XFAIL and confirm the other doesn’t work.

gateway.py::test_delete_catalog_item PASSED [ 33%]

gateway.py::test_cannot_create_item XFAIL [ 66%]

gateway.py::test_gateway_create_item PASSED [100%]

===== 2 passed, 1 xfailed in 2.72s =====

While I was doing this a colleage identified an error in the system gateway limit on max bytes to buffer : 262144

Delivery Truck

My way of explaining this issue to management has been as follows. Imagine we have agreed that sending mail to our office should be done via the post. We have constructed a mailbox at the front of the building and the postal service has an agreement to deliver mail to this building. We’ve agreed:

• how letters should be addressed
• in what order information should be stored in the mail
• what names are allowed such as title, start_time, end_time

What has NOT been considered because it just a letter is how large the letter can be. Because while we’ve constructed a letter box we should have been constructing a loading bay for UPS delivery vans. Most of this information is small, but geometry is flexible. Unfortunately in this case the geometry cannot be easily simplified.

New API specs

The only practical solution to such a problem is find a way to allow more input as either a payload or in the gateway directly. That can open you up to malicious attacks so we’ve had to re-spec the API endpoint and take a different approach. Lesson learnt, when dealing wiht geometry communications over a network, consider the complexity and length of the data.

p.s if writing your own tests around such problems, I’d recommend naming things more professionally. def test_gateway_can_fit_delivery_truck(auth_code):