Making a basic map with Mapnik

Objective

Basic tutorial on using XML and python with Mapnik to create a basic map. In this example, I want to create a background map that mimics the one on FlightAware for plotting flight plan routes and flown routes, for example this one.

Flight plan route map of Washington National to Chicago O'Hare airport

This article explains the basics of creating the map image. Part 2 describes how to set the projection.

What is Mapnik?

Mapnik is a library that creates map image files combining data from various sources. To create maps with mapnik, you need three things:

  1. A set of data you want to draw.

    There are two types of data vector data (polygons, lines, points) and raster data. Each data source may be in a separate coordinate reference system.

  2. A set of layer and style descriptions.

    Data sources are incorporated into layers and drawn one on top of the other, like a stack of animation cels. You tell mapnik where the data for each layer comes from (a datasource), how to draw it (with a style), and the order in which to draw the layers. The styles and layers can be specified in an XML-format text file, programmatically in a python, node.js or C++ program, or in a combination of XML and a program.

    Stack of map layers
  3. A front-end program.

    Mapnik itself is a library, not an application. You need to either write a (generally small) program, or use an existing tool. In either case, the program tells mapnik about the layer/style descriptions and the output map parameters, and "calls" the mapnik library to draw and save the map images.

There are a bunch of tools that can use mapnik as a "rendering engine" to produce map images, for example to create "map tiles" at different scales for interactive maps. (Map tiles are requested by a web application, and the server can either send them from a cache or instruct mapnik to create them if they do not exist.)

How the styles and layers are specified to mapnik depends on how much flexibility you need and your programming skill. Instructing mapnik about the data sources, styles and layers programmatically has maximum flexibility, but can be hard to maintain (especially for non-programmers) and is generally laborious to write. On the other hand, an XML file is static and has less flexibility; for example, it can be hard to have styles that depend on the context of the data or to do data manipulation in XML, though there are limited facilities such as conditional style filters. (Using a database such as PostgreSQL as allows some more sophisticated data manipulation in the XML file, but then you need to be able to write SQL.) Fortunately, a hybrid approach can be adopted — you can keep the styles and perhaps some basic background layers in an XML file and write do the more sophisticated data munging and data-dependent styling in a program.

Example Application

To create the background FlightAware-style map, I used the following vector data. The data were from NaturalEarth data [2], and sources are listed in an appendix below datasources.

  1. Background
    What colour to draw on the empty map (or render as transparent). We will select a colour for the sea and draw the land on top.
  2. Country land areas
    Polygons for the land area of each country.
  3. River centrelines
    Line data. There are many rivers of various length and sizes ranging from streams to major waterways over a mile wide. We only want to draw the major rivers. Fortunately, the NaturalEarth data contains a scalerank for each river.
  4. Coastlines
    Line data for coastlines.
  5. Water bodies: rivers/lakes
    Polygon data. Depending on the map scale, when a river is sufficiently wide it can no longer be represented satisfactorily as its centerline (estuaries of major rivers, for examples). Many data sets therefore have polygons for the river banks. Lakes can be regarded as very wide rivers!
  6. Land border lines
    Line data. To delineate countries (administrative level 0) and states/provinces (administrative level 1).

These were drawn as the following layers. The order is important and some trial-and-error may be needed to determine the best æsthetic affect. (Trials of the colours and layer order using a GIS editor such as QGIS before creating the mapnik style files can be quite convenient.)

No. Layer Source Geometry Drawing Style
0 Background None Colour #0e233e
1 Countries Polygon Polygon, filled with colour #23384f
2 River lines Line Line, colour #204059, width 1.25 px, scalerank >= 4
3 Lake outlines Polygon Line around edges, colour #204059, width 3 px
4 Lake bodies Polygon Polygon, filled with colour #0e233e
5 Country borders Polygon Line, colour #656357, width 1.25 px
6 State borders Line Line, colour #3c4f60, width 1.25 px
7 Coastline 1 Line Line, colour grey, width 1.25 px, 75% transparent
8 Coastline 2 Line Line, colour #2d6492, width 0.75 px

(For details on how colours are specified, see [A].)

In our data, lakes are represented as polygons, and a single lake may be broken down into multiple polygons. Large lakes may also be fed by wide rivers, which may themselves be wide enough to have polygons. If we want to draw borders around the lakes, we need to draw the outlines first as thick lines, then draw the filled polygons on top. This will overdraw the borders between adjacent lake polygons, leaving just the shores.

For the coastlines, I wanted to draw a composite line; a wide grey line with a thinner line on top. One way to do this with mapnik is to draw the same dataset twice with two separate layers; a lower layer with a thick line, overlaid with a layer with a thinner line.

Images of each layer mocked up in QGIS are shown below.

1 Countries 5 Country outlines
1_countries 5_country_outlines
2 River centrelines 6 State borders
2_river_centerlines 6_state_borders
3 Lake outlines 7 Coastline 1
3_lake_outlines 7_coastline_1
4 Lakes 8 Coastline 2
4_lakes 8_coastline_2

and the final mockup-composite from QGIS is:

9_composite

Mapnik XML File

We can put these definitions in an XML file for mapnik to process. All drawing styles and layers can be defined in the XML file (as would be the case if the XML file is processed by a software tool that the user did not write), or some can be put in the XML file and some set in a user-written program.

The mapnik XML file format is documented here [1]. The outer level is a <Map> that contains <Style> and <Layer> declarations.

Each <Style> declaration contains one or more <Rule> declarations that define how to draw the data source to which they are applied; <PolygonSymbolizer> for drawing polygons, <LineSymbolizer> for drawing lines, <PointSymbolizer> for drawing points and <TextSymbolizer> for drawing text. Rules can also contain <Filter> statements that select the rule according to some condition (for example, involving certain data attributes), and a style can contain several rules each with a different filter.

Each <Layer> declaration references a <Datasource> (where the data are obtained from) and a style to be applied. <Layers> are rendered in the order they are defined. The basic hierarchy is shown below:

<Map> [background_colour] [projection]
  |
  +----- <Style> [style_name]
  |         |
  |         +----- <Rule>
  |                  |
  |                  +---- <Filter>
  |                  |
  |                  +---- <PolygonSymbolizer>
  |                  |
  |                  +---- <LineSymbolizer>
  |                  |
  |                  +---- <PointSymbolizer>
  |                  |
  |                  +---- <TextSymbolizer>
  |
  +----- <Layer> [layer_name]
            |
            +------ <StyleName>
            |
            +------ <Datasource>

For example, the above definitions for drawing the country polygons on the background colour map with the data taken from a NaturalEarth Shapefile ne_10m_admin_0_countries.shp can be defined by the following XML:

<Map background-color="#0e233e" srs="+proj=latlong +datum=WGS84">
  <!-- Define layer 1 style and call it 'Country_Style' -->
  <Style name="Country_Style">
    <Rule>
      <PolygonSymbolizer fill="#23384f" />
    </Rule>
  </Style>

  <!-- Layer 1: country polygons -->
  <Layer name="Country">
    <StyleName>Country_Style</StyleName>
    <Datasource>
      <Parameter name="type">shape</Parameter>
      <Parameter name="file">ne_10m_admin_0_countries.shp</Parameter>
    </Datasource>
 </Layer>
<Map>

The following style fragment is for rendering the river and lake centrelines. Each river/lake centerline in the NaturalEarth Shapefile ne_10m_rivers_lakes_centerlines_scale_rank.shp has an attribute called scalerank that has a number denoting the "scale" of the river. We can use a <Filter> on a style <Rule> to only trigger that rule for rivers of scalerank of 4 or less with the following <Style> declaration:

<Style name="River_Style">
  <Rule>
    <Filter>[scalerank] &lt;= 4</Filter>
    <LineSymbolizer stroke="#204059" stroke-width="1.25" />
  </Rule>
</Style>

The filter clause specifies that the rule only applies if the value of the scalerank attribute stored with the river linestring (which is written in square brackets in the XML file) is less than or equal to 4. (In other computer languages the relational operator 'less than or equal to' would be written as '<=' but because angle brackets (<>) in XML are used to denote tags, they cannot be used for other purposes and instead, an "entity" code is used; '&lt;' in this case substitutes for the '<' character.)

(In fact, we can be even more sophisticated by writing a filter to use different scaleranks depending on the scale of reproduction of the output map, but I'll leave that as an exercise for the reader!)

Rendering the map

We can use an existing application that takes a mapnik XML style file and outputs a map image, such as nik4, or write a program ourselves.

nik4 is a python program that can produce various maps, including tiles. On the command line:

$ nik4 -b 81 38 -74 45 -x 350 350 map.xmp output.png

where:

-b 81 38 -74 45
specifies the bounds of a rectangular area in the order xmin, ymin, xmax, ymax
-x 350 350
specifies the width and height of the output image
map.xmp
is the mapnik XML style file
output.png
is the name of the output file

Other options allow selection of which layers to render etc.

This produces the image:

nik4_output

A simple python program to produce the same image would be as follows (assuming you have the mapnik python bindings installed):

import mapnik

m = mapnik.Map(350, 350)       # map x,y size in pixels
mapnik.load_map(m, 'map.xml')  # load the style file

# You can write code here to create additional styles
# and layers.

# Set the bounds of the area to render.
bbox=mapnik.Box2D(81, 38, -74, 45)
m.zoom_to_box(bbox)

mapnik.render_to_file(m, 'output.png')

Afterword: XML Tips

If you go the XML style file route, sooner or later you're going to want to make it more manageable by splitting it into separate parts (for example, separating styles and layers, or letting you share parts across different projects), or using symbols with meaningful names instead of the same hard-coded value in several places to make changes easier.

Full-blown XML has some handy features that ease the pain of maintaining large XML documents, such as letting you define symbols that represent certain strings (in XML parlance, entities and corresponding values) and allowing you to break up a single monolithic XML file into several pieces. Unfortunately, the XML parser used by mapnik (a library rather than its own) is not so smart and cannot handle entities. However, a program xmllint, which is a tool that comes with some xml libraries, can be used to process full-blown XML documents containing entities and spit out simplified XML file with no entities which mapnik can digest. (It is also useful for checking the syntax of files.)

Entities (symbols) can be defined in following header that should be prepended to the basic XML mapnik file.

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE Map [
   entity definitions
]>
<Map>
...
</Map>

Entity definition statements are of the form:

<!ENTITY name ... >

and are referenced in the document by

&name;

Two basic types of entity are an external entity that can be used to include an external file when the document is processed, and an internal entity that defines a value that will be substituted for the entity name when it is encountered in the file. These are defined respectively by statements of the form

<!ENTITY name SYSTEM "filename">

and

<!ENTITY name "value">

For example, we want to put the set of <Style> definitions in another file styles.xml and to define the colour of country as an entity called country_fill_colour. Let the main map file be called map-base.xml and the styles be in a file styles.xml. The file map-base.xml would start with:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE Map [
<!ENTITY stylefile SYSTEM "styles.xml">
<!ENTITY country_fill_colour "#0e233e">
]>

<!-- Include style file here -->
&stylefile;

and in the file styles.xml we could have:

<Style name="Country_Style">
  <Rule>
    <PolygonSymbolizer fill="&country_fill_colour;" />
  </Rule>
</Style>

To produce a complete file map.xml that can be used as a mapnik style file, we run xmllint from the command line to perform the entity substitutions are follows:

$ xmllint -noent -o map.xml map-base.xml

where:

-noent
tells xmllint to expand entities
-o map.xml
specifies the output file as map.xml (otherwise the output is written to the standard output)
map-base.xml
is the input filename

One might further wish to include all the colours in a file colours.ent containing just entity statements. Unfortunately, we can't reference this in the same way as other external entities in the header; we have to use a slightly different syntax to include a file containing entity declarations in the header, as in the following example:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE Map [
<!ENTITY stylefile SYSTEM "styles.xml">
<!ENTITY % colours SYSTEM "colours.ent">
%colours;
]>

Now we can define the file colours.ent containing just entity statements:

<!ENTITY country_fill "#0e233e">
<!ENTITY country_border "#656357">
...

Data Sources

The following NaturalEarth data sets were used [2] :

Item Dataset Filename
Country polygons Admin 0 - Countries ne_10m_admin_0_countries
River/lake centrelines Rivers + lake centerlines ne_10m_rivers_lake_centrelines_scale_rank
Lakes Lakes ne_10m_lakes
Coastline Coastline ne_10m_coastline
Country land border lines Admin 0 - Boundary Lines ne_10m_admin_0_boundary_lines_land
State lines Admin 1 - States, Provinces ne_10m_admin_1_states_provinces_lines

Footnotes

[A]In the mapnik XML, colours can be referred to by name (as defined by HTML) or specified as a hash sign followed by six hexadecimal digits in the format #rrggbb, where rr, gg and bb respectively specify the red, green and blue colour components in the range (00,ff). White is #ffffff, black is #000000, red is #ff0000, and so forth.

References

[1]Mapnik configuration XML: https://github.com/mapnik/mapnik/wiki/XMLConfigReference
[2](1, 2) Natural Earth: https://www.naturalearthdata.com

social