Saving, cropping and merging data

Saving, cropping and merging data#

Pyrosm can write .osm.pbf files, not only read them. This lets you crop a large extract down to a smaller area to share or re-read faster (OSM.to_pbf), write modified OSM data back to a valid PBF after editing tags or attributes in pandas, export only selected layers (e.g. just the buildings or the road network) to a new PBF (OSM.write_pbf), or merge overlapping extracts into one file (merge_pbf).

How to?

Crop a PBF to a bounding box#

to_pbf() crops the source file by the OSM object’s bounding_box and writes a valid, re-readable PBF. The crop is complete-ways (a way is kept whole when any of its nodes is inside the box), it streams the file blob-by-blob (so it stays low on memory), and coordinates round-trip exactly.

import os
from pyrosm import OSM, get_data

fp = get_data("helsinki")
orig_size_mb = os.path.getsize(fp) / (1024 * 1024)

# Crop to a bounding box [minx, miny, maxx, maxy] in lon/lat
bbox = [24.93, 60.16, 24.96, 60.18]
cropped = OSM(fp, bounding_box=bbox).to_pbf()

size_mb = os.path.getsize(cropped) / (1024 * 1024)
print(f"Cropped size: {os.path.basename(cropped)} {size_mb:.1f} MiB.")
print(f"Original size: {os.path.basename(fp)} {orig_size_mb:.1f} MiB.")

Cropped size: pyrosm_crop_0zc0d7tr.osm.pbf 7.1 MiB.
Original size: Helsinki.osm.pbf 62.1 MiB.

# The cropped file reads back like any other PBF
OSM(cropped).get_buildings().shape

(1504, 36)

Smaller output: compact and repack#

By default the crop keeps each source block’s string table, which is the fastest option. Two flags trade a little speed for a smaller file:

  • compact=True prunes each block’s string table to only the strings its kept elements reference.

  • repack=True re-packs the kept elements into densely filled blocks (as osmium/Osmosis produce) for the smallest output; it supersedes compact.

The written OSM data is identical for every option.

default = OSM(fp, bounding_box=bbox).to_pbf()
compact = OSM(fp, bounding_box=bbox).to_pbf(compact=True)
repack = OSM(fp, bounding_box=bbox).to_pbf(repack=True)

for label, path in [("default", default), ("compact", compact), ("repack", repack)]:
    size_mb = os.path.getsize(path) / (1024 * 1024)
    print(f"{label:8s} {size_mb:.1f} MiB")

default  7.1 MiB
compact  2.6 MiB
repack   2.1 MiB

As we can see, the compact and repack options drop the size of the output file significantly, but in return, the cropping process is slower. The default behavior is the fastest and recommended approach if the larger output file size is not an issue.

Write modified OSM data back to a PBF#

write_pbf() writes the data you read back to a valid PBF after you edit it in pandas — for example to fill a missing maxspeed or add a computed travel_time tag. Each row updates the matching element (by osm_type + id); rows whose id is not in the source are added as new elements synthesised from their geometry.

import tempfile

osm = OSM(get_data("helsinki"))
edges = osm.get_network("driving")

# Edit/add tags in pandas — here we tag every edge with a travel_time (static 15 kmph)
speed_kmph = 15
edges["travel_time"] = 3.6 * edges["length"] / speed_kmph

out_path = os.path.join(tempfile.gettempdir(), "modified.osm.pbf")
osm.write_pbf(edges, out_path)

size_mb = os.path.getsize(out_path) / (1024 * 1024)
print("wrote", os.path.basename(out_path), size_mb, "MiB.")

wrote modified.osm.pbf 57.212669372558594 MiB.

# Read it back, requesting the new tag as a column
reread = OSM(out_path).get_network("driving", extra_attributes=["travel_time"])
reread["travel_time"].astype(float).mean()

np.float64(22.957565802760012)

Export only selected layers#

By default write_pbf() writes the whole dataset it read, so untouched layers survive. Pass subset_only=True to instead write a PBF containing only the features you give it – for example parse the buildings and export just those, or just the road network. The references each feature needs to stay valid are pulled in automatically (a way’s nodes, a relation’s member ways and nodes), so the output is a valid, re-readable PBF.

You can also combine several layers by passing a list ([buildings, network]); the union of their elements is written.

osm = OSM(get_data("helsinki"))
buildings = osm.get_buildings()
network = osm.get_network()

# A buildings-only subset is much smaller than the whole dataset
whole = os.path.join(tempfile.gettempdir(), "whole.osm.pbf")
osm.write_pbf(buildings, whole)  # default: the whole dataset is written
b_only = os.path.join(tempfile.gettempdir(), "buildings_only.osm.pbf")
osm.write_pbf(buildings, b_only, subset_only=True)  # only the buildings (+ refs)
for label, p in [("whole dataset", whole), ("buildings only", b_only)]:
    print(f"{label:15s} {os.path.getsize(p) / 1024 / 1024:.1f} MiB")

# Combine several layers by passing a list -> the union is written
combined = os.path.join(tempfile.gettempdir(), "buildings_and_network.osm.pbf")
osm.write_pbf([buildings, network], combined, subset_only=True)
both = OSM(combined)
print("combined:", len(both.get_buildings()), "buildings,",
      len(both.get_network()), "network edges")

whole dataset   57.0 MiB
buildings only  13.2 MiB
combined: 187449 buildings, 249318 network edges

Merge several PBF files#

merge_pbf() combines overlapping .osm.pbf files into one file. It helps when the area you need crosses the edge of the published extracts, such as a town on a national border that two country extracts cover only together. An element found in several files is written once, and with bounding_box the merged file is cropped to that box with the same rule as to_pbf(). The result goes to a temporary file unless you give output_path. The inputs must be sorted by element type and id, as the extracts from Geofabrik and the other providers are.

from pyrosm import OSM, get_data, merge_pbf

# Valga (Estonia) and Valka (Latvia) form one town across the border
fp = merge_pbf(
    [get_data("estonia"), get_data("latvia")],
    bounding_box=[25.99, 57.75, 26.10, 57.80],
)
print(f"{len(OSM(fp).get_buildings())} buildings in Valga and Valka")

8264 buildings in Valga and Valka

A small example with the Helsinki test data: two overlapping crops merge into one file that holds each building once.

from pyrosm import OSM, get_data, merge_pbf

fp = get_data("helsinki_pbf")

# Two overlapping crops of the same area
west = OSM(fp, bounding_box=[24.935, 60.164, 24.946, 60.179]).to_pbf()
east = OSM(fp, bounding_box=[24.942, 60.164, 24.953, 60.179]).to_pbf()

merged = merge_pbf([west, east])

for name, path in [("west", west), ("east", east), ("merged", merged)]:
    print(f"{name}: {len(OSM(path).get_buildings())} buildings")

west: 256 buildings
east: 330 buildings
merged: 475 buildings