Recipes

A recipe is a YAML file that describes how the OSM data of an analysis is made: which extract to download, which layers to read from it and which graph to build from it. pyrosm run carries out a recipe, writes its outputs and records how they were made. The recipe file is then the method: running it again reproduces the data, and the record shows which data and which package versions went into it.

Recipes need PyYAML and pyarrow:

pip install "pyrosm[recipes]"

The recipes on this page are in the examples/recipes folder of the repository.

A first recipe

This recipe reads the walking network, the buildings and the restaurants and cafes of central Helsinki from the extract that comes with pyrosm:

recipe: pyrosm
extract:
  area: {name: helsinki_pbf}
layers:
  walking_network: {read: network, network_type: walking}
  buildings: {read: buildings}
  food: {read: pois, custom_filter: {amenity: [restaurant, cafe]}}
outputs:
  extract: helsinki.osm.pbf
  layers:
    walking_network: walking_network.parquet
    buildings: buildings.parquet
    food: food.parquet

Run it with the output folder data:

pyrosm run helsinki_layers.yaml -o data

The command prints the path of each output, after written or reused:

written /home/me/project/data/helsinki.osm.pbf
written /home/me/project/data/walking_network.parquet
written /home/me/project/data/buildings.parquet
written /home/me/project/data/food.parquet
written /home/me/project/data/helsinki_layers.provenance.json

Each layer is a GeoParquet file:

import geopandas as gpd

buildings = gpd.read_parquet("data/buildings.parquet")

Recipe format

A recipe is a mapping with these keys:

Key	What it holds
recipe	Always pyrosm. Required.
version	The recipe format version. Only 1 exists, the default.
extract	The extract stage: the OSM data to download.
pbf	The PBF file to read when there is no extract stage: {file: path}.
reader	Keywords for pyrosm.OSM, such as bounding_box or keep_metadata.
layers	The layers stage: the layers to read.
graph	The graph stage: the graph to build.
outputs	The file name of each output. Required.

A recipe has at least one stage. Layers and a graph are read from the extract stage’s file or from the pbf file, so they need one of the two, and a recipe cannot have both. The paths of input files are relative to the recipe’s folder.

The other keys of a stage are keywords of the function the stage calls, with their values as YAML gives them. pyrosm checks each keyword’s name against that function before anything runs, so a misspelt keyword is reported by name. The function checks the values when the stage runs.

Extract stage

The area key says what to download, as exactly one of:

• bbox: [minx, miny, maxx, maxy], in longitude and latitude;

• place: "Kallio, Helsinki", a place name, turned into an area with pyrosm.geocode() (with the headers and timeout of extract);

• file: area.gpkg, a GeoPackage, GeoJSON or GeoParquet file whose geometries make up the area (with layer: name for one layer of a GeoPackage);

• name: helsinki_pbf, a dataset name of pyrosm.get_data().

For a bounding box, a place or an area file, the stage calls pyrosm.get_data_by_area(), and the other keys of extract are its keywords, such as strategy, crop, must_cover, workers and directory. must_cover names a file: must_cover: {file: stops.gpkg}. For a dataset name, the stage calls get_data and takes its keywords. The recipe sets output_path itself, and opener cannot be given. directory, the folder that keeps the downloads, is relative to the recipe’s folder and must not lie inside the output folder or contain it.

This recipe downloads the smallest extract that contains Kallio, or, when it is smaller in total, a set of extracts that together cover it, merged (strategy: smallest_total), and crops the data to the district:

recipe: pyrosm
extract:
  area: {place: "Kallio, Helsinki"}
  strategy: smallest_total
  crop: polygon
layers:
  driving_network: {read: network, network_type: driving}
outputs:
  extract: kallio.osm.pbf
  layers: {driving_network: driving_network.parquet}

Layers stage

layers names each layer. A layer’s read key is one of network, buildings, pois, landuse, natural, boundaries or custom, which calls the OSM method of that name (custom calls get_data_by_custom_criteria). Its other keys are that method’s keywords. Each layer is written as GeoParquet; a read that finds nothing is written as an empty layer.

Graph stage

graph.network holds the keywords of pyrosm.OSM.get_network(). The other keys of graph are keywords of pyrosm.graphs.graph_tables(), such as simplify and retain_all, which builds the graph as the graph exporters do. The network type sets the direction of the edges: a walking graph runs both ways, a driving graph keeps one-way streets one-way. force_bidirectional: true makes every edge run both ways.

recipe: pyrosm
extract:
  area: {name: helsinki_pbf}
graph:
  network: {network_type: walking}
  simplify: true
outputs:
  extract: helsinki.osm.pbf
  graph: {nodes: walking_nodes.parquet, edges: walking_edges.parquet}

The nodes and the edges are written as two GeoParquet files. A column that holds lists or dicts, such as the attributes of edges merged by simplify, is written as text, with the lists and dicts as JSON.

Outputs

outputs names every file a stage writes: extract for the extract stage’s PBF, a file for each layer under layers, and nodes and edges under graph. provenance names the provenance record and defaults to <recipe name>.provenance.json. The names are relative to the output folder, which is the folder given with -o or, by default, the recipe’s folder. They end with .pbf, .parquet and .json, and must be names that Windows accepts too.

Provenance record

The provenance record is a JSON file with:

• versions: the versions of pyrosm, geopandas, shapely, pandas, numpy, pyarrow and protobuf;

• recipe: the recipe with every path resolved (the extract’s headers are left out, as they can hold credentials);

• inputs and outputs: the SHA-256 of each input file and each output;

• extract: where the extract came from, such as its provider, extract name, URL, snapshot time and the extracts merged into it;

• layers and graph: the number of features of each layer, and of nodes and edges;

• invocation and written_at: how the run was started, and when it finished.

The outputs are made in a staging folder and moved into place only when every stage has succeeded, so a stage that fails leaves the previous outputs and record as they were. While a recipe runs, a lock file .<record name>.lock in the output folder keeps a second run of it from starting. If a run was killed, delete the lock file once no run is active.

Reruns

A second run reuses a stage when its part of the recipe, the reader keywords, its output names, the data it reads and the package versions are all unchanged, and its outputs are in place with the SHA-256 the record gives them. The command prints reused for the outputs of such a stage. Changing a layer reruns that layer only; changing reader reruns the layers and the graph but keeps the extract.

An extract stage is reused as long as the recipe is unchanged, even when the provider has published newer data since. To download again, run with --force (in Python, force=True), which reruns every stage and downloads the extract again, or set update: true in extract, which reruns the extract stage every time.

Pins

A recipe can pin the data it expects:

extract:
  area: {place: "Kallio, Helsinki"}
  snapshot: 2026-10-01
  sha256: "<the 64 hexadecimal characters of the file's SHA-256>"

sha256 can also be given on the pbf, area file and must_cover inputs. A run whose input or extract differs from a pin stops, and the previous outputs and record stay as they were. A snapshot date matches an extract whose data was taken on that day (UTC); a date and time must match the snapshot time exactly. Put a SHA-256 in quotes, so that YAML reads it as text.

Command line

pyrosm validate RECIPE
pyrosm run RECIPE [-o DIR] [--force]

pyrosm validate checks a recipe without running it. Both commands exit with status 0 on success, 1 when the recipe or its data is refused (the reason goes to stderr) and 2 when the command is used wrongly. They never prompt, so they can run in scripts and workflow tools.

Python

from pyrosm import recipes

recipes.validate("helsinki_layers.yaml")
paths = recipes.run("helsinki_layers.yaml", out_dir="data")

pyrosm.recipes.run() returns the paths of the outputs, the provenance record last.

Snakemake

A recipe fits a Snakemake rule. The rule lists the outputs it needs; pyrosm run writes them, and reruns only the stages that changed:

rule osm_layers:
    input:
        "recipes/helsinki_layers.yaml",
    output:
        "data/walking_network.parquet",
        "data/buildings.parquet",
        "data/helsinki_layers.provenance.json",
    shell:
        "pyrosm run {input} -o data"