Minutely replication – timeout when trying to download change files from planet.openstreetmap.org

Help and support
, , , ,
1

Hi,

I’m trying to keep an OSM POI database up to date using the minutely diffs.

This command is run in a loop every minute:

osmosis -v --read-replication-interval workingDirectory=/workosmdir --buffer-change --simplify-change --buffer-change --write-xml-change changes.osc

osmosis is configured with

baseUrl=https://planet.openstreetmap.org/replication/minute

The process has been running well for weeks and months, but is now failing:

org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to read the changeset file 005/751/336.osc.gz from the server.
at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.downloadReplicationFile(BaseReplicationDownloader.java:119)
at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.download(BaseReplicationDownloader.java:232)
at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader.java:271)
at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.java:350)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:285)
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)
at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.downloadReplicationFile(BaseReplicationDownloader.java:102)
... 4 more

The file in the error message https://planet.openstreetmap.org/replication/minute/005/751/336.osc.gz exists and is quick for me to download.

Is there some rate limiting in place? I’m wondering, since the timeout happens during the SSL connection already and it looks like a network/firewall issue?

If yes, how should I be calling osmosis to stay within the limits? Or is there a mirror I should use?

2

Both of the popular changeset tracking sites, osmcha and Find Suspicious OpenStreetMap Changeset, haven’t been working for the past 24h. I suspect there the root cause is the same - issues with replication servers.

3

Switched to https://download.openstreetmap.fr/replication/planet/minute which solved the issue.

4

Since Saturday to Sunday night I also get a lot of timeouts when OSMOSIS and other services try to download the minutley diff files from here: Index of /replication/minute

Usually it gets better after 24 hours, but now the problem persists. Any ideas? So far I do not want to switch to another replication site.