Upgrading

Upgrading#

This page lists the changes since pyrosm 0.6.2 (2023) that change what a read, a download, a written file or a graph export gives, or that need a change in your code, with what to do about each. Find the release you upgrade from and read the sections of the releases after it, newest first; the releases not listed here changed nothing of this kind. Changes in speed and memory use, new options that leave the defaults as they were, and fixes of calls that failed are left out. The changelog lists every change.

pyrosm 0.24.0#

  • Polygons and lines. A closed way is a Polygon when one of its tags is in the OSM wiki’s list of polygon features, as in OSMnx, and a LineString otherwise; area=no makes it a LineString. Closed coastlines, cliffs, tree rows, streams, railway loops and closed ways without an area tag are LineStrings, where they were Polygons; closed hedges, city walls, retaining walls and motorway service areas are Polygons, where they were LineStrings. In 0.23.0 a closed way with a barrier, highway or route tag was a LineString even when another tag made it an area, such as a fenced sports pitch or school; such areas are Polygons again.

  • Columns and dtypes. Both engines give the same columns with the same dtypes. Every read has a bool visible column; network, landuse, natural and custom reads had the flag in the JSON tags column, so read it from the column instead. Out-of-core node rows have int32 version, uint32 timestamp and int32 changeset, as in-memory ones do. In-memory reads with a bounding box and complete_relations=True no longer keep an empty column of a feature outside the box, and the keys in the JSON tags column of a history read follow the order of the tags in the file.

  • Filter keys as columns. Every reader keeps the keys of its filter as columns, also with tags_to_keep or keep_other_tags=False: get_buildings(tags_to_keep=["name"]) has a building column.

  • custom_filter of get_network. A plain dict keeps the ways it matches: get_network(custom_filter={"highway": ["footway"]}) returns the footways, where it returned every way but the footways. Add filter_type="exclude" to leave them out.

  • get_boundaries(name=...) finds the name as plain text, ignoring case, and returns None with a warning when no name contains it. It was a case-sensitive regular expression; to match a pattern, filter the result of get_boundaries() with boundaries[boundaries["name"].str.contains(pattern, na=False)].

  • get_data_by_custom_criteria(osm_keys_to_keep=...) without a custom_filter returns only the elements that have one of the keys. It returned every element.

  • NetworkX graphs. With osmnx_compatible=True, the default, the edge attributes oneway and reversed are booleans, as in OSMnx, or on a simplified edge whose ways differ the list of their values; the OSM text of the oneway tag is in oneway_tag. Default graphs keep their largest connected part before they are simplified, so they are fully simplified and have fewer nodes and edges than before.

  • Graphs with tags_to_keep. get_graph(tags_to_keep=...) also reads the tags the graph is built from: oneway, junction and, for cycling, oneway:bicycle, and when simplifying the edge_attrs_differ tags. A driving graph was two-way everywhere, and the simplification merged edges across changes of speed limit and road class.

  • Graph exports. Every graph type leaves out the edges whose end nodes are not in the node frame and identifies the nodes by node_id_col. The node x/y of NetworkX, pandarm and pandana graphs come from the node geometry, so a projected network gives a graph with projected coordinates. With simplify=False, the line of an edge travelled against its way’s node order runs from u to v. simplify_kwargs={"node_attrs_include": [...]} finds node tags as well as node columns.

  • Progress functions. A progress function given to OSM, merge_pbf, OSM.write_pbf, OSM.to_graph, the graph exporters, find_extracts or pyrosm.enable_logging is called as progress(done, 100), with the percentage of the whole operation done. It got the bytes read in each pass over the file of a read, crop or merge, and the steps done of a graph export. Downloads still call progress(written, total) with bytes.

  • Extracts by box or place. get_data_by_bbox and get_data_by_geocoding choose among Geofabrik, BBBike and Movisda extracts, so the file can come from another provider than Geofabrik. With download=False they return a URL, or a list of URLs when the area needs a set of extracts. get_data_by_geocoding crops to the place’s polygon instead of its bounding rectangle and names its crops area_<place>.osm.pbf. get_data_by_area(crop=False) with one extract copies the extract to output_path; it ignored output_path.

  • File names of the two Georgias. get_data("usa/georgia") and get_data("europe/georgia") save us-georgia-latest.osm.pbf and europe-georgia-latest.osm.pbf; both were saved as georgia-latest.osm.pbf, so get_data("europe/georgia") could return the US state.

  • OSM.clear_downloads(path) with a path removes exactly the file at that path, and only a .osm.pbf or .osh.pbf file; it removed the file of that name in the download folder. A bare file name, such as "finland-latest.osm.pbf", still removes the file of that name in the download folder.

  • Bounding boxes. Every function that takes a bounding box takes the same forms, and a projected GeoDataFrame or GeoSeries is reprojected to lon/lat; get_data_by_bbox read its metres as degrees. Crops, merges and get_data_by_bbox refuse coordinates outside lon/lat, where crops and merges wrote an empty file. OSM(bounding_box=polygon).to_pbf() crops to the polygon, as a read with it does, instead of the polygon’s bounding rectangle.

  • Written files. OSM.write_pbf merges the rows of an element: a change made in one row of a way read with get_network(nodes=True), or in one of two frames that hold the same element, is written for the whole element, with a UserWarning when other rows of the same frame keep the old value, and rows that give a tag different new values raise a ValueError; the last row was written. It writes its elements sorted by id, writes the metadata that was read and none where none was read (new elements get version 1 and the time of writing), no longer writes the network columns u and v as tags while it keeps real tags such as length, and writes the id_tag column back as the tag id. After a read with a bounding box it writes only the relations with a member among the nodes and ways read; it wrote every relation of the file. With complete_relations=True, OSM.to_pbf and OSM.write_pbf also write the member ways outside the box of the relations that reach into it, and their nodes.

pyrosm 0.23.0#

  • Closed ways are Polygons or LineStrings by all their tags, also those that are not columns of the read: get_buildings() returns a building tagged area=no as a LineString. 0.24.0 replaced this rule with the polygon features described in its section.

  • visible values. A missing visible flag reads as True, as the PBF format reads it. Every node and way of a file that is not a history file had visible=False. Relations have a visible value too.

  • Tags keyed id. A node’s or relation’s tag keyed id comes out as id_tag, as a way’s does, instead of replacing the element’s OSM id. Filters read it as id_tag in every reader (custom_filter={"id_tag": [...]}); the filter key id raises a ValueError.

  • custom_filter of the layer readers. get_boundaries, get_buildings, get_landuse and get_natural narrow their result by a custom_filter: get_boundaries(custom_filter={"admin_level": ["10"]}) returns the boundaries with admin_level=10, where it returned every boundary and everything else with admin_level=10.

  • Driving networks. driving and driving+service leave out highway=busway, ladder and trail ways, which cars cannot use.

  • Simplified graphs sum a travel_time column along a merged edge, as OSMnx does; simplify_kwargs={"edge_attr_aggs": {...}} aggregates other columns.

  • pandarm graphs get the CRS of the nodes; pandarm assumed geographic coordinates.

  • Parallel reads. Out-of-core reads (the default engine), crops and merges run on several threads from about 20 MB of data (workers="auto", the default). Results and written files are the same for every worker count; workers=1 keeps a read on one core. The worker-process backend (PYROSM_DECODE_BACKEND=processes) is deprecated.

pyrosm 0.22.0#

  • Python versions. pyrosm supports Python 3.11–3.15. On Python 3.10, pip installs the last release that supports it.

  • NetworkX attributes. NetworkX graphs store only the attributes an edge or node has a value for, as OSMnx’s graphs do: test whether an attribute is present ("maxspeed" in data) instead of whether it is NaN or None.

  • Node tags. Nodes whose tags all became columns have no tags value.

  • Reads at the same time. Out-of-core reads that run at the same time in several threads of one process keep their own settings. A read that started while another one decoded in the calling process could change the file, filter keys or bounding box of the decoding read, which then returned too few rows without an error or read the wrong file: run such reads again.

  • Multipolygons have the same geometries, but their rings and vertices can come in another order, so a comparison of coordinates or of written bytes can differ.

  • Deprecations. get_path() and the pandana graph backend warn with a FutureWarning that they will be removed in pyrosm 1.0. Use get_data() and graph_type="pandarm" (pyrosm.graphs.to_pandarm).

pyrosm 0.21.0#

  • Written files. Crops, merge_pbf and write_pbf compress their blocks with libdeflate. The files hold the same data in slightly fewer bytes, so they no longer match files written by earlier versions byte for byte, and the SHA-256 of a cropped or merged extract changes: a recipe whose extract.sha256 was pinned with an earlier version needs the new value.

pyrosm 0.19.1#

  • Access rules of the networks. The predefined networks leave out a way that access=no or access=private closes to the public unless a tag of the network’s mode of transport allows the use, following OSM’s access hierarchy (walking: foot, access; cycling: bicycle, vehicle, access; driving: motorcar, motor_vehicle, vehicle, access). A way tagged access=private and foot=yes is now in the walking network.

pyrosm 0.19.0#

  • Out-of-core engine by default. OSM(...) reads with the out-of-core engine, which reads country-sized files with bounded memory and keeps each read’s result in the temporary folder, so an identical read returns it without decoding the file again. OSM.clear_cache() removes the kept results, and OSM(..., engine="in_memory") reads with the earlier reader. Both return the same elements and values, but the order of the rows can differ, so sort both results by the same columns before you compare them.

  • Column order of the out-of-core engine. The out-of-core engine gives the columns in the order of the in-memory reader: timestamp before version, visible and nodes at their place among the other columns, and tags after lat in the node frame of get_network(nodes=True). Select columns by name when the order matters.

  • LineStrings. get_network() returns each edge as one LineString through its way’s nodes instead of a MultiLineString of two-point segments, and the other readers return each line way as one LineString too; relations keep their MultiLineStrings. The coordinates and the length column are the same. Code that went through geometry.geoms of a network edge can use the line itself.

  • Simplified graphs. to_graph and the graph exporters of pyrosm.graphs simplify the graph by default; simplify=False keeps one edge per segment. A chain is not merged across a change of maxspeed or highway, so each edge has one speed limit and one road class; simplify_kwargs={"edge_attrs_differ": None} merges as OSMnx does by default. pandana and pandarm graphs sum their weight columns along a merged chain, as length is summed.

  • Network type of a graph. to_graph takes the network type from the frames it is given; it took the type of the network read last, so driving edges exported after a walking read became a two-way graph.

  • History reads. A read of a history file no longer returns elements deleted before the timestamp, and a bounding_box applies to each element’s state at the timestamp.

  • Untagged nodes of plain (non-dense) node groups have None tags, as dense ones do, instead of an empty dict.

pyrosm 0.18.0#

  • History reads take each element’s latest version whole: a tag that the latest version had removed no longer comes back from an older version.

  • Column order. get_pois, get_natural and get_data_by_custom_criteria give their node columns in one order on every run; the order changed with Python’s hash seed.

  • Written files. write_pbf keeps the tags of the elements a frame matches when the frame has no tag columns, such as one that names elements by id and osm_type only; it wrote them without tags.

pyrosm 0.14.0#

  • Overlapping ways in simplified graphs. Where two OSM ways run along the same nodes, the simplified graph keeps one edge, as OSMnx does; it counted the stretch twice in the edge count and the total length.

  • Dataset names. get_data("madrid"), get_data("beijing") and get_data("victoria") each name both a BBBike city and a Geofabrik region: use "cities/madrid" or "spain/madrid", and so on for "china/beijing" and "australia/victoria". Spain’s Geofabrik regions have ASCII names (castilla_y_leon, pais_vasco). sources.asia.china, and the same attribute of the other countries that gained regions, is the country’s group of regions; the whole-country file stays available through get_data("china") and the group’s country record.

pyrosm 0.13.0#

  • Network types as in OSMnx. The predefined network types take the way filters of OSMnx (0.19.1 and 0.23.0 changed them further, as described above). driving is OSMnx’s drive and no longer includes service roads: use driving+service to keep them. all is filtered instead of returning every way, and the new all_public leaves out privately accessible ways. Every mode-specific network leaves out access=private ways, and walking leaves out ways whose sidewalk is mapped separately. Every network leaves out the highway values of ways that are not built yet or no more (such as proposed, construction and abandoned) and of service areas (rest_area, services); read them with get_data_by_custom_criteria, for example custom_filter={"highway": ["construction"]}. driving keeps ways tagged psv=yes, and the network filters match a tag that holds several values (access=private;delivery).

  • NetworkX graphs work with OSMnx: parallel edges between two nodes are all kept (only the last one was), key is no longer an edge attribute, osmid is the node key and no longer a node attribute, and node tags are a JSON string, as the edge tags are.

  • Unknown network types raise a ValueError in every reader.

pyrosm 0.11.0#

  • Multipolygons. Multipolygon and boundary relations are assembled by the even-odd rule, as osmium and GDAL assemble them, instead of by the roles of their members. Relations with islands in holes, touching inner rings or wrong member roles get different, correct geometries.

pyrosm 0.9.1#

  • Incomplete boundaries. get_boundaries() leaves out a boundary whose member ways run off the extract; it closed the ring with a straight edge across the gap.

  • Ambiguous dataset names. get_data() raises a ValueError for a name that several regions share, such as "georgia": use a region-qualified name, "usa/georgia" or "europe/georgia".

pyrosm 0.8.0#

  • Coordinates are read at full float64 precision, the exact 7-decimal OSM values; they were float32, about 0.1 m off.

  • Ring orientation. Polygon exteriors run counter-clockwise and holes clockwise, by the right-hand rule of GeoJSON and OGC, as in osmium and QGIS. Rings followed the node order of their ways.

  • Relation members. Relations carry their members under the members key of the JSON tags column, each as {member_id, member_type, member_role}.

pyrosm 0.7.0#

  • Python versions. Python 3.9 is no longer supported.

  • Relation ids are the OSM ids of the relations; 0.6.2 decoded them wrongly.

  • Bounding-box reads return the ways that cross the edge of the box whole instead of cut at the edge, so a result can reach a little beyond the box: clip it with GeoPandas (gdf.clip(box)) when you need its exact bounds. The node frame of get_network(nodes=True) keeps every end node of the kept edges, also those outside the box.

  • Tags keyed id. A way’s tag keyed id comes out as id_tag instead of replacing the OSM id of the way.

  • Cycling networks are directed and follow oneway:bicycle.

  • Default columns. A get_* call no longer adds columns to the default tag lists of pyrosm.config.Conf.tags: in 0.6.2 a call with extra_attributes added them to every later read of that layer in the process, so a read’s columns depended on the calls before it.

  • Filters. A custom_filter with a highway key no longer turns closed-way polygons into lines. Network filters check every filter key a way has: a highway=service way tagged service=parking_aisle is left out of the driving network, which kept it.

  • NetworkX graphs no longer hold duplicate nodes.