What is the preferred way to download planet diff files?

dieterdreist · February 3, 2024, 11:30am

I am keeping a copy of the planet file which I update every few weeks with the daily diff files. This used to work quite well until some time ago, when I guess some sort of rate limiting was implemented. Now it takes more time to download the diffs (without applying them) than it would take to download the whole planet dump again. In order to keep transfer sizes low for the servers, I still continue to apply daily diffs, but I wonder if this slow downloading behaviour is maybe a hint to operate in a different way?

I am seeing lots of timeouts, like this:

--2024-02-03 12:28:36--  (try: 3)  https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/day/000/004/139.osc.gz
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:8651:34db:a9a6::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:85c9:34db:ab6e::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:441:34db:2fe5::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:181:34db:2e24::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:180:34db:2e20::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404c:450:34db:8cb9::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:8541:34db:ab7e::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:84c9:34db:a926::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.169.21|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.133|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.105|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.72.139|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.46.35|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.140.183|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.170.177|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.25|:443... failed: Operation timed out.
Retrying.

here is the result of a traceroute to amazonaws.com

$ traceroute osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com
traceroute: Warning: osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com has multiple addresses; using 52.219.170.169
traceroute to s3.dualstack.eu-central-1.amazonaws.com (52.219.170.169), 64 hops max, 52 byte packets
 1  myfastgate.lan (192.168.1.254)  0.969 ms  0.689 ms  0.672 ms
 2  10.103.123.27 (10.103.123.27)  6.204 ms  1.637 ms  4.147 ms
 3  10.103.72.2 (10.103.72.2)  1.765 ms
    10.1.209.250 (10.1.209.250)  2.076 ms  2.041 ms
 4  10.250.132.213 (10.250.132.213)  5.362 ms  5.792 ms  4.308 ms
 5  10.254.20.5 (10.254.20.5)  6.099 ms  4.732 ms
    10.254.20.9 (10.254.20.9)  5.431 ms
 6  93-57-68-98.ip163.fastwebnet.it (93.57.68.98)  6.431 ms  4.623 ms  7.809 ms
 7  93-57-68-137.ip163.fastwebnet.it (93.57.68.137)  4.172 ms  8.787 ms
    93-57-68-141.ip163.fastwebnet.it (93.57.68.141)  5.929 ms
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  s3.eu-central-1.amazonaws.com (52.219.170.169)  24.379 ms  22.298 ms  23.309 ms

here another one to planet.osm.org:

$ traceroute planet.osm.org
traceroute to planet.osm.org (184.104.179.145), 64 hops max, 52 byte packets
 1  myfastgate.lan (192.168.1.254)  1.130 ms  0.731 ms  0.675 ms
 2  10.103.123.27 (10.103.123.27)  1.580 ms  1.367 ms  2.501 ms
 3  10.1.209.250 (10.1.209.250)  3.329 ms
    10.103.72.2 (10.103.72.2)  2.221 ms
    10.1.209.250 (10.1.209.250)  2.079 ms
 4  10.250.132.209 (10.250.132.209)  6.542 ms
    10.250.132.213 (10.250.132.213)  5.759 ms  3.952 ms
 5  10.254.20.9 (10.254.20.9)  7.082 ms  7.101 ms
    10.254.20.5 (10.254.20.5)  24.009 ms
 6  93-57-68-98.ip163.fastwebnet.it (93.57.68.98)  3.272 ms
    93-57-68-102.ip163.fastwebnet.it (93.57.68.102)  2.867 ms
    93-57-68-133.ip163.fastwebnet.it (93.57.68.133)  3.017 ms
 7  93-57-68-141.ip163.fastwebnet.it (93.57.68.141)  9.099 ms  3.978 ms
    93-57-68-137.ip163.fastwebnet.it (93.57.68.137)  5.349 ms
 8  100ge0-36.core2.rom1.he.net (184.104.194.10)  12.309 ms  14.115 ms  15.650 ms
 9  * * he.minap.it (185.1.114.20)  15.138 ms
10  100ge0-76.core2.zrh2.he.net (184.105.65.33)  19.968 ms  17.258 ms  18.535 ms
11  100ge0-49.core3.fra1.he.net (184.105.65.30)  35.361 ms  26.365 ms  25.165 ms
12  port-channel2.core3.fra2.he.net (72.52.92.70)  27.410 ms *  23.985 ms
13  100ge0-76.core1.ams2.he.net (184.104.193.194)  37.504 ms * *
14  openstreetmap-foundation.port-channel7.switch2.ams2.he.net (184.104.202.70)  35.134 ms  35.818 ms  37.621 ms
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
31  * * *
32  * * *
33  * * *
34  * * *
35  * * *
36  * * *
37  * * *
38  * * *
39  * * *
40  * * *
41  * * *
42  * * *
43  * * *
44  * * *
45  * * *
46  * * *
47  * * *
48  * * *
49  * * *
50  * * *
51  * * *
52  * * *
53  * * *
54  * * *
55  * * *
56  * * *
57  * * *
58  * * *
59  * * *
60  * * *
61  * * *
62  * * *
63  * * *
64  * * *

Firefishy · February 3, 2024, 5:50pm

Those timeouts are not normal. There is no rate limiting or blocks that I am aware of.

Firefishy · May 23, 2024, 1:53pm

Reading the messages posted, it seems like you have a local dual stack (IPv4 / IPv6) networking issue.

The service is up and running and has 100s of people downloading from it.

failed: Network is down. is an issue with networking on your machine.

Maybe try disable IPv6 or force a IPv4 connection for the downloads.

SomeoneElse · March 5, 2024, 1:18pm

The problem is local to you. Other people are not reporting this (server stats for me are here - the only “extended delays” there are me reloading the database).

I suggest that you try again, but on different network infrastructure.

pitscheplatsch · June 10, 2024, 12:37pm

The issue no longer occurs for me. A few more details can be found in the github tracker: Replication connection timeout · Issue #1097 · openstreetmap/operations · GitHub

Thanks @Firefishy !

GeoDeskTeam · February 3, 2024, 4:58pm

@Firefishy or @pnorman may be able to answer.
I’m curious about this as well.

dieterdreist · February 6, 2024, 9:29am

Thank you, I have amended the question with the traceroute logs

dieterdreist · February 16, 2024, 10:53am

The issue is persisting:

--2024-02-15 21:27:56--  (try: 3)  https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/day/000/004/164.osc.gz
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:8490:34db:a912::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40a0:189:34db:a80d::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:150:34db:2e49::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40a0:c9:34db:a89d::|:443... failed: Network is unreachable.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404c:289:34db:8c04::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:8480:34db:ab02::|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:8540:34db:ab7a::|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404c:450:34db:8cb9::|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.170.113|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|3.5.137.131|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.170.81|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.168.76|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.169.97|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.73|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.85|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.170.57|:443... failed: Network is down.
Retrying.

dieterdreist · March 5, 2024, 9:29am

I still keep getting time out problems.

Resolving planet.osm.org (planet.osm.org)... 2001:470:1:fa1::11, 184.104.179.145
Connecting to planet.osm.org (planet.osm.org)|2001:470:1:fa1::11|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/day/000/004/175.osc.gz [following]
--2024-03-05 10:16:07--  https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/day/000/004/175.osc.gz
Resolving osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)... 2a05:d050:40af:8651:34db:a9a6::, 2a05:d050:402f:8580:34db:aa4a::, 2a05:d050:40a0:3d1:34db:4bb4::, ...
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:8651:34db:a9a6::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:402f:8580:34db:aa4a::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40a0:3d1:34db:4bb4::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:402f:8549:34db:aa86::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:1d0:34db:2e10::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:1d1:34db:2e14::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404c:441:34db:8cad::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:8701:34db:a9fe::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.72.211|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 120821635 (115M) [binary/octet-stream]
Saving to: ‘daily/175.osc.gz’

daily/175.osc.gz                                         100%[===============================================================================================================================>] 115,22M  8,62MB/s    in 16s     

2024-03-05 10:26:23 (7,19 MB/s) - ‘daily/175.osc.gz’ saved [120821635/120821635]

Maybe this topic is related: Minutely replication – timeout when trying to download change files from planet.openstreetmap.org

his “solution” was changing the replication url to download.openstreetmap.fr

dieterdreist · March 5, 2024, 4:37pm

pitscheplatsch · June 10, 2024, 5:28am

Since Saturday to Sunday night I also get a lot of timeouts when OSMOSIS and other services try to download the minutley diff files from here: Index of /replication/minute

Usually it gets better after 24 hours, but now the problem persists. Any ideas? So far I do not want to switch to another replication site.

(Cross posting: Minutely replication – timeout when trying to download change files from planet.openstreetmap.org - #4 by pitscheplatsch)