How Hard Could It Be?
- programmer epitaph
Here is a story about some things I learned about open street maps and how map tile rendering works, and some reflections on dealing with the unknown as a developer.
The Pride
It all started innocently enough. I was investigating the possibility of using OpenStreetMap (OSM) data to render some maps and perform turn-by-turn navigation in an app I was working on. OSM was interesting because the app needed to work off-line. OSM offered a data-set that could be updated when the device was connected to show new roads, without the multi-month turn around of other mapping providers. The UI platform I was working with didn’t have any libraries for rendering OSM maps that I could find. There were libraries for reading OSM xml files, and the more concise protocol-buffer-based osm.pbf files. I was going to use .osm.pbf files for routing (which I had a nice library for) so after spending a few hours reading up on OSM I decided to try to write something of my own to render maps directly from the .osm file. Armed with some osm.pbf files of my home state, and the OsmSharp library to read them I was able to create a simple renderer in fairly short order.
The Fall
A little while later I had buildings being rendered in colour, and areas of land by “usage”. Unfortunately the rendering was too slow to be useful. All the roads (the main feature I cared about) were all still 2-dimensional poly-lines without any width. I was also starting to appreciate some of the added complexity of maps, such as showing different amounts of detail as you zoom in and out of the map. If only there was a way i could pre-compute and cache the geometries of various things I wanted to render at different zoom levels….
After a little more digging I discovered I wasn’t the only one who had had the bright idea of pre-computing geometries. It turned out there was a file format called MBTiles made by the folks at MapBox which was basically a SQLite database full of map tiles. Originally these had been satellite images, but then they’d added vector tiles, which were the pre-computed polygons for a particular map area and zoom level, serialized using protocol buffers.
Vectors
The discovery of MBTiles and vector tiles quickly led me to two decent vector tile renderer implementations which worked with the existing tile-based map controls for the platform I was using, which made my hand-rolled rendering implementation look as bad as it really was.
Critical Errors
In hindsight a number of things contributed to my confusion which I try to catalogue here.
Incorrect Assumptions
I’d assumed that tile-based map controls weren’t going to be useful to me because I’d also assumed that pre-rendered map tiles of images would take up too much bandwidth to update “over the air”. I’d also under-estimated the computational complexity of rendering a large map area, and naively thought that a dynamic vector rendering of the whole area being displayed would be the best solution. Most of us perform a lot of our day-to-day work using the knowledge and experience we’ve gained over a lifetime of work. Usually these assumptions hold true, and provide a useful framework for us to build on. Except when they don’t. If I’d spent more time investigating the tile-based renderers I might have joined the dots on MBTiles sooner.
Niche Platform
Some of the features I was after like offline maps were at odds with the platform I was running on (“desktop” windows). Investigating in more detail how it was done on other platforms where this was more common might have got me to the answer sooner.
You’re Doing it Wrong
Excessive pain when doing something or using a given technology is good signal of missing something in the bigger picture. If you keep thinking to yourself “there must be a better way of doing this” you’re probably right.
Domain Size
The size of the Open Street Map ecosystem also befuddled me. There are lots of different file formats for map data, and libraries that wrap other libraries across all the different platforms people use. Some libraries work with other proprietary map formats. Understandably a large part of the ecosystem is geared up towards tile serving and “online” use of maps in web applications which wasn’t relevant to me (or so I’d thought). Having discovered vector tiles and the MBTiles format I now understand that vector tile creation itself is a fairly large topic (hello TileMaker). Turtles all the way down.
A Cry for Help
One of the best, and perhaps the only way to discover unknown unknowns is to discuss your problem domain with people from a variety of backgrounds and areas of expertise. In this case I was more than capable of representing the ‘novice’ demographic. Discussion with someone with more digital mapping expertise would have probably uncovered my mistakes sooner. One thing I tend to not do very well is ask questions of others. Even though I was one of the first people invited to use StackOverflow after suggesting the name to Jeff I don’t recall ever asking a question on that platform. This is probably the key learning for me from this exercise - to engage with the network of people I know and the internet at large sooner to try to uncover these blind spots.
The diabolical thing about unknown unknowns is that we are operating without an awareness of them all the time. It’s not as if I can sit back and say “whew! I’m glad there are no more unknown unknowns for me to worry about in digital cartography”. As subsequent levels of the onion are peeled back and we learn more things the number of gaps in our knowledge hopefully diminishes. Diabolically thanks to hindsight bias we quickly forget that we were ever unaware of that “thing”. But the risk always remains that there is something fairly substantial out there that we’re just not aware of.
If you have any strategies for dealing with gaps in knowledge like this I’d love to hear them. Please leave a comment below.