Hi all!
I am quite new to osm. While working on a research project I wanted to use the docker image of osm2pgsql to import a osm dataset into a postgis database. However, many osm objects were not imported, especially the polygons and also some nodes. To be honest i am clueless on why this happens, and with my limited knowledge in the topic makes it hard to find the problem myself, so I am hoping on finding some help for my problem here.
I will state the Data and commands I used below.
System: MacBook Pro with macOS Sequoia V 15.3.1
Docker version 27.5.1, build 9f9e405
I tried to use a xml import, however I got the Message ERROR: No element inside <member> allowed (which surprised me as I used the Data I got from Overpass so I thought there should not be an error in the Data):
docker run --rm \
--network citydb-net \
-v /Users/Path/to/Data:/osm-data \
-e PGPASSWORD=postgres \
iboates/osm2pgsql:latest \
osm2pgsql \
-H postgis_osm \
-P 5432 \
-d osm_db \
-U postgres \
--create \
--slim \
--hstore \
/osm-data/overpass_result.xml
After that I converted my xml Data into pbf Data with osmosis and my “final” Import query looked like this:
--network citydb-net \
-v /Users/Path/to/Data:/osm-data \
-e PGPASSWORD=postgres \
iboates/osm2pgsql:latest \
osm2pgsql \
-H postgis_osm \
-P 5432 \
-d osm_db \
-U postgres \
--create \
--slim \
--hstore \
/osm-data/export.pbf
I got the following responses:
2025-01-04 16:01:56 Processed 1319 nodes in 0s - 1k/s
2025-01-04 16:01:56 Processed 747 ways in 0s - 747/s
2025-01-04 16:01:56 Processed 74 relations in 0s - 74/s
However, many polygons were missing. This surprised me, because I checked and there were also many complete ways that should have formed a polygon, which did not work.
WITH node_points AS (
SELECT
p.id AS way_id,
ST_SetSRID(ST_Point(n.lon, n.lat), 4326) AS point_geom
FROM
planet_osm_ways p
JOIN
planet_osm_nodes n ON n.id = ANY(p.nodes)
)
SELECT
way_id,
ST_IsClosed(ST_MakeLine(point_geom)) AS is_closed
FROM
node_points
GROUP BY
way_id;
way_id | is_closed
------------+-----------
4362256 | t
4362264 | t
7947538 | t
7947539 | t
7947540 | t
7947555 | t
7947557 | t
7947867 | t
7947956 | t
8084201 | f
13480804 | f
...
As mentioned I used osm dataset I got from overpass. This was my query:
Note: I know the query is redundand in many ways, however I decided to use it in my research project because of various methodical reasons - thats not the point I want to discuss, however I think it only makes sense to include the whole Data I used to get to the bottom of my error.
[out:xml][timeout:90];
(
way["building"]({{bbox}});
nwr["information"="board"] ["board_type"="map"]({{bbox}});
nwr["information"="board"] ["board_type"="public_transport"]({{bbox}});
nwr["information"="board"] ["board_type"="architecture"]({{bbox}});
nwr["kerb"="lowered"]({{bbox}});
nwr["kerb"="flush"]({{bbox}});
nwr["highway"="traffic_signals"] ["traffic_signals:grip"="yes"]({{bbox}});
nwr["highway"="traffic_signals"] ["traffic_signals:sound"="yes"]({{bbox}});
nwr["highway"="traffic_signals"] ["traffic_signals:vibration"="yes"]({{bbox}});
nwr["highway"="traffic_signals"] ["traffic_signals:foot"="yes"]({{bbox}});
nwr["highway"="traffic_signals"] ["traffic_signals:bicycle"="yes"]({{bbox}});
nwr["amenity"="parking"] ["access"="private"]({{bbox}});
nwr["amenity"="parking"] ["residents"="yes"]({{bbox}});
nwr["amenity"="parking"] ["residents"="designated"]({{bbox}});
nwr["amenity"="parking"] ["parking:condition"="residents"]({{bbox}});
nwr["amenity"="parking"] ["traffic_sign"="DE:1020-32"]({{bbox}});
nwr["traffic_sign"="DE:314"] ["access"="private"]({{bbox}});
nwr["traffic_sign"="DE:314"] ["residents"="yes"]({{bbox}});
nwr["traffic_sign"="DE:314"] ["residents"="designated"]({{bbox}});
nwr["traffic_sign"="DE:314"] ["parking:condition"="residents"]({{bbox}});
nwr["traffic_sign"="DE:314"] ["traffic_sign"="DE:1020-32"]({{bbox}});
nwr["traffic_sign"="DE:315"] ["access"="private"]({{bbox}});
nwr["traffic_sign"="DE:315"] ["residents"="yes"]({{bbox}});
nwr["traffic_sign"="DE:315"] ["residents"="designated"]({{bbox}});
nwr["traffic_sign"="DE:315"] ["parking:condition"="residents"]({{bbox}});
nwr["traffic_sign"="DE:315"] ["traffic_sign"="DE:1020-32"]({{bbox}});
nwr["public_transport"="stop_display"] ["accessibility"="yes"]({{bbox}});
nwr["public_transport"="stop_display"] ["speech_output"="yes"]({{bbox}});
nwr["public_transport"="stop_display"] ["speech_output:purpose"="blind"]({{bbox}});
nwr["highway"="crossing"] ["crossing"="marked"]({{bbox}});
nwr["highway"="crossing"] ["crossing_ref"="zebra"]({{bbox}});
nwr["traffic_sign"="DE:350"]({{bbox}});
nwr["highway"="traffic_signals"] ["traffic_signals:foot"="yes"]({{bbox}});
nwr["crossing"="traffic_signals"]({{bbox}});
nwr["highway"="street_lamp"] ["lit"="yes"]({{bbox}});
nwr["highway"]["lamp:type"="street_lamp"]({{bbox}});
nwr["highway"]["lamp:type"="path"]({{bbox}});
nwr["type"="lighting"]({{bbox}});
nwr["public_transport"="platform"] ["bus"="yes"]({{bbox}});
nwr["public_transport"="platform"] ["tram"="yes"]({{bbox}});
nwr["public_transport"="platform"] ["train"="yes"]({{bbox}});
node["information"="route_marker"] ["bicycle"="yes"]({{bbox}});
nwr["amenity"="bench"]({{bbox}});
nwr["highway"="cycleway"] ["segregated"="yes"]({{bbox}});
nwr["highway"="cycleway"] ["cycleway:separation"="kerb"]({{bbox}});
nwr["highway"="cycleway"] ["cycleway:separation"="barrier"]({{bbox}});
nwr["bicycle"="designated"] ["segregated"="yes"]({{bbox}});
nwr["bicycle"="designated"] ["cycleway:separation"="kerb"]({{bbox}});
nwr["bicycle"="designated"] ["cycleway:separation"="barrier"]({{bbox}});
nwr["bicycle"="yes"] ["segregated"="yes"]({{bbox}});
nwr["bicycle"="yes"] ["cycleway:separation"="kerb"]({{bbox}});
nwr["bicycle"="yes"] ["cycleway:separation"="barrier"]({{bbox}});
nwr["foot"="designated"] ["segregated"="yes"]({{bbox}});
nwr["highway"="footway"] ["segregated"="yes"]({{bbox}});
nwr["highway"="footway"] ["cycleway:separation"="kerb"]({{bbox}});
nwr["highway"="footway"] ["cycleway:separation"="barrier"]({{bbox}});
node["natural"="tree"]({{bbox}});
way["natural"="tree_row"]({{bbox}});
nwr["highway"="traffic_signals"]({{bbox}});
way["landuse"="grass"]({{bbox}});
nwr["leisure"="park"]({{bbox}});
way["natural"="tree_row"]({{bbox}});
way["natural"="scrub"]({{bbox}});
way["landuse"="forest"]({{bbox}});
way["landuse"="meadow"]({{bbox}});
node["amenity"="planter"]({{bbox}});
node["natural"="tree"]({{bbox}});
nwr["amenity"="parking_space"] ["parking_space"="disabled"]({{bbox}});
nwr["amenity"="parking"] ["parking_space"="disabled"]({{bbox}});
nwr["amenity"="parking"] ["parking_space"="disabled"]({{bbox}});
node["traffic_sign"="DE:314;DE:1044-10"]({{bbox}});
node["traffic_sign"~"DE:350|DE:239|DE:240|DE:241"] ["lit"="yes"]({{bbox}});
way["taxi:lanes"]({{bbox}});
node["traffic_calming"="cushion"]({{bbox}});
nwr["traffic_calming"="bump"]({{bbox}});
way["highway"="footway"] ["width"="2"]({{bbox}});
way["highway"="footway"] ["width"="2.5"]({{bbox}});
way["highway"="path"] ["foot"="designated"]({{bbox}});
way["highway"="cycleway"]({{bbox}});
way["highway"] ["cycleway"~"lane|track"]({{bbox}});
way["bus:lanes"~"designated"]({{bbox}});
nwr["amenity"="parking_space"] ["parking_space"="car_sharing"]({{bbox}});
nwr["amenity"="parking_space"] ["parking"="car_sharing"]({{bbox}});
nwr["amenity"="car_sharing"]({{bbox}});
nwr["barrier"="block"]({{bbox}});
nwr["barrier"="bollard"]({{bbox}});
way["access"="no"]({{bbox}});
way["motor_vehicle"="no"]({{bbox}});
way["access:conditional"]({{bbox}});
way["access"="destination"]({{bbox}});
way["hgv"="no"]({{bbox}});
way["hgv"="destination"]({{bbox}});
way["hgv:conditional"]({{bbox}});
nwr["highway"="traffic_signals"] ["traffic_signals:bicycle"="yes"]({{bbox}});
nwr["highway"="traffic_signals"] ["traffic_signals:countdown"="yes"]({{bbox}});
nwr["amenity"="charging_station"]({{bbox}});
nwr["amenity"="bicycle_parking"]["ebike"="yes"] ["charging"="yes"]({{bbox}});
nwr["amenity"="charging_station"]["bicycle"="designated"]({{bbox}});
nwr["amenity"="car_sharing"] ["capacity:electric"="0"]({{bbox}});
nwr["amenity"="bicycle_rental"] ["capacity:electric"="0"]({{bbox}});
nwr["amenity"="bicycle_rental"] ["electric_bicycle"]({{bbox}});
nwr["amenity"="bicycle_rental"] ["capacity:cargo"="0"]({{bbox}});
nwr["amenity"="bicycle_rental"] ["cargo_bike"]({{bbox}});
nwr["highway"="crossing"] ["traffic_signals:foot"="yes"]({{bbox}});
nwr["highway"="crossing"] ["kerb"="lowered"]({{bbox}});
nwr["highway"="crossing"] ["kerb"="flush"]({{bbox}});
way["oneway"="yes"] ["oneway:bicycle"="no"]({{bbox}});
nwr["barrier"="bollard"] ["bollard"="automatic"]({{bbox}});
nwr["highway"="cycleway"] ["crossing"="raised"]({{bbox}});
nwr["highway"="crossing"] ["crossing"="raised"]({{bbox}});
nwr["traffic_calming"="choker"]({{bbox}});
nwr["traffic_calming"="narrow"]({{bbox}});
nwr["traffic_calming"="island"]({{bbox}});
nwr["information"="board"] ["board_type"="public_transport"]({{bbox}});
nwr["information"="board"] ["announcement"="yes"]({{bbox}});
nwr["information"="board"] ["board_type"="public_transport"]({{bbox}});
nwr["information"="board"] ["announcement:automatic"="yes"]({{bbox}});
nwr["service:bicycle:pump"="yes"]({{bbox}});
nwr["service:bicycle:repair"="yes"]({{bbox}});
nwr["amenity"="bicycle_parking"]({{bbox}});
nwr["amenity"="bicycle_parking"] ["bicycle_parking"~"lockers|building"]({{bbox}});
nwr["bicycle_road"="yes"]({{bbox}});
node["highway"="crossing"] (around:10)["public_transport"="platform"]({{bbox}});
nwr["highway"="pedestrian"]({{bbox}});
node["crossing"="traffic_signals"] ["traffic_signals:foot"="yes"]({{bbox}});
node["crossing"="traffic_signals"] ["traffic_signals:bicycle"="no"]({{bbox}});
node["highway"="traffic_signals"] ["traffic_signals:foot"="yes"]({{bbox}});
node["highway"="traffic_signals"](around:10)["public_transport"="platform"]({{bbox}});
node["information"="guidepost"] ["foot"="yes"]({{bbox}});
nwr["oneway"="yes"] [~"^parking:lane:(left|right|both)$"~"sidewalk"]({{bbox}});
);
out tags meta geom;
I would be thankful for any input on what I did wrong and how I can fix my import. Of course there are many ways to improve my process, the goal of my project was first to get it to work and the optimization has a lower priority.
Thanks in advance to everyone who tries to help!