If using "osm2pgsql-replication init" with "--server https://planet.openstreetmap.org/replication/minute", what determines the replication start date and how to I change it?

Help and support
,
1

I’m the using osm2pgsql version 1.6.0 that is packaged within Ubuntu 22.04.

After following https://switch2osm.org/serving-tiles/manually-building-a-tile-server-ubuntu-22-04-lts/ If I immediately do:

sudo -u _renderd osm2pgsql-replication init -d gis  --server https://planet.openstreetmap.org/replication/minute

I get

2022-04-24 10:27:36 [INFO]: Initialised updates for service 'https://planet.openstreetmap.org/replication/minute'.
2022-04-24 10:27:36 [INFO]: Starting at sequence 5029284 (2022-04-24 07:04:37+00:00).

Where the “starting at sequence and date” seems to be “some variable time in the past”. What determines the starting sequence number and date, and how do I change it (without just editing the resulting values in the database after the “–init”).

2

The actual timestamp is calculated using the most recent way id found in your database (btw: is it expected that your database already contains some data?):

Based on the timestamp, the sequence number is then calculated using a timestamp_to_sequence function which is documented here: replication - Handling Updates of OSM Data — Pyosmium 3.2.0 documentation

3

Whilst that answers the “what determines the replication start date” part it doesn’t answer “how do I change it”.

For example, if I try and use

sudo -u _renderd osm2pgsql-replication init -d gis  -v --server https://planet.openstreetmap.org/replication/minute

I get a python stacktrace that seems to suggest an http 404 error. The debug produced by the -v flag shows what’s wrong:

2022-06-10 10:27:59 [DEBUG]: Using way id 9200014014 for timestamp lookup

That way isn’t from OSM at all, it’s from here. Trying to specify both server and file doesn’t work:

sudo -u _renderd osm2pgsql-replication init -d gis  -v --server https://planet.openstreetmap.org/replication/minute --osm-file ~/data/north-yorkshire-latest.osm.pbf
usage: osm2pgsql-replication init [-h] [-q] [-v] [-d DB] [-U NAME] [-H HOST]
                              [-P PORT] [-p PREFIX]
                              [--osm-file FILE | --server URL]
osm2pgsql-replication init: error: argument --osm-file: not allowed with argument --server

How do I specify a date?

4

Did you import some fake/non-existing OSM data into your database? I think osm2pgsql-replication wasn’t designed to deal with such data. Maybe you can take a look at the compute_database_date function in osm2pgsql/osm2pgsql-replication at master · openstreetmap/osm2pgsql · GitHub and adjust it to your needs.

Alternative option would be to provide the replication state using the --osm-file command line parameter. Maybe take a look at the osm2pgsql-replication manual page for more details.

5

Yes - it was the way that I linked to above from the legend that’s loaded into the database after any OSM data.

Ideally you’d be able to specify where to apply updates from and when to when to apply updates from separately (just like you can with osmosis, which this presumably is designed to replace).

I’m looking at how best to change openstreetmap-replication currently …

Edit:

The problem is that there’s no try/catch (or python equivalent) around:

with urlrequest.urlopen(urlrequest.Request(url, headers=headers)) as response:
    data = json.loads(response.read().decode('utf-8'))

so it never gets to:

if not data.get('elements') or not 'timestamp' in data['elements'][0]:
    LOG.fatal("The way data downloaded from the API does not contain valid data.\n"
              "URL used: %s", url)
    return None
6

Maybe it would make more sense to have this discussion over at Discussions · openstreetmap/osm2pgsql · GitHub

7

osm2pgsql-replication is meant as a simple tool for setting up updates in the very generic case that you import the planet/from extracts and want to keep them up to date without thinking about such things as time of update.

If you have special requirements, you should use pyosmium-get-changes. This is part of the pyosmium and meant to be the direct the replacement for osmosis (without the inconvenience of having to compute sequence IDs yourself).

1 Like