I am keeping a copy of the planet file which I update every few weeks with the daily diff files. This used to work quite well until some time ago, when I guess some sort of rate limiting was implemented. Now it takes more time to download the diffs (without applying them) than it would take to download the whole planet dump again. In order to keep transfer sizes low for the servers, I still continue to apply daily diffs, but I wonder if this slow downloading behaviour is maybe a hint to operate in a different way?
I am seeing lots of timeouts, like this:
--2024-02-03 12:28:36-- (try: 3) https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/day/000/004/139.osc.gz
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:8651:34db:a9a6::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:85c9:34db:ab6e::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:441:34db:2fe5::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:181:34db:2e24::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:180:34db:2e20::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404c:450:34db:8cb9::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:8541:34db:ab7e::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:84c9:34db:a926::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.169.21|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.133|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.105|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.72.139|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.46.35|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.140.183|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.170.177|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.25|:443... failed: Operation timed out.
Retrying.
here is the result of a traceroute to amazonaws.com
$ traceroute osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com
traceroute: Warning: osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com has multiple addresses; using 52.219.170.169
traceroute to s3.dualstack.eu-central-1.amazonaws.com (52.219.170.169), 64 hops max, 52 byte packets
1 myfastgate.lan (192.168.1.254) 0.969 ms 0.689 ms 0.672 ms
2 10.103.123.27 (10.103.123.27) 6.204 ms 1.637 ms 4.147 ms
3 10.103.72.2 (10.103.72.2) 1.765 ms
10.1.209.250 (10.1.209.250) 2.076 ms 2.041 ms
4 10.250.132.213 (10.250.132.213) 5.362 ms 5.792 ms 4.308 ms
5 10.254.20.5 (10.254.20.5) 6.099 ms 4.732 ms
10.254.20.9 (10.254.20.9) 5.431 ms
6 93-57-68-98.ip163.fastwebnet.it (93.57.68.98) 6.431 ms 4.623 ms 7.809 ms
7 93-57-68-137.ip163.fastwebnet.it (93.57.68.137) 4.172 ms 8.787 ms
93-57-68-141.ip163.fastwebnet.it (93.57.68.141) 5.929 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 s3.eu-central-1.amazonaws.com (52.219.170.169) 24.379 ms 22.298 ms 23.309 ms
here another one to planet.osm.org:
$ traceroute planet.osm.org
traceroute to planet.osm.org (184.104.179.145), 64 hops max, 52 byte packets
1 myfastgate.lan (192.168.1.254) 1.130 ms 0.731 ms 0.675 ms
2 10.103.123.27 (10.103.123.27) 1.580 ms 1.367 ms 2.501 ms
3 10.1.209.250 (10.1.209.250) 3.329 ms
10.103.72.2 (10.103.72.2) 2.221 ms
10.1.209.250 (10.1.209.250) 2.079 ms
4 10.250.132.209 (10.250.132.209) 6.542 ms
10.250.132.213 (10.250.132.213) 5.759 ms 3.952 ms
5 10.254.20.9 (10.254.20.9) 7.082 ms 7.101 ms
10.254.20.5 (10.254.20.5) 24.009 ms
6 93-57-68-98.ip163.fastwebnet.it (93.57.68.98) 3.272 ms
93-57-68-102.ip163.fastwebnet.it (93.57.68.102) 2.867 ms
93-57-68-133.ip163.fastwebnet.it (93.57.68.133) 3.017 ms
7 93-57-68-141.ip163.fastwebnet.it (93.57.68.141) 9.099 ms 3.978 ms
93-57-68-137.ip163.fastwebnet.it (93.57.68.137) 5.349 ms
8 100ge0-36.core2.rom1.he.net (184.104.194.10) 12.309 ms 14.115 ms 15.650 ms
9 * * he.minap.it (185.1.114.20) 15.138 ms
10 100ge0-76.core2.zrh2.he.net (184.105.65.33) 19.968 ms 17.258 ms 18.535 ms
11 100ge0-49.core3.fra1.he.net (184.105.65.30) 35.361 ms 26.365 ms 25.165 ms
12 port-channel2.core3.fra2.he.net (72.52.92.70) 27.410 ms * 23.985 ms
13 100ge0-76.core1.ams2.he.net (184.104.193.194) 37.504 ms * *
14 openstreetmap-foundation.port-channel7.switch2.ams2.he.net (184.104.202.70) 35.134 ms 35.818 ms 37.621 ms
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
31 * * *
32 * * *
33 * * *
34 * * *
35 * * *
36 * * *
37 * * *
38 * * *
39 * * *
40 * * *
41 * * *
42 * * *
43 * * *
44 * * *
45 * * *
46 * * *
47 * * *
48 * * *
49 * * *
50 * * *
51 * * *
52 * * *
53 * * *
54 * * *
55 * * *
56 * * *
57 * * *
58 * * *
59 * * *
60 * * *
61 * * *
62 * * *
63 * * *
64 * * *
1 Like
Firefishy
(Firefishy)
3
Those timeouts are not normal. There is no rate limiting or blocks that I am aware of.
3 Likes
Reading the messages posted, it seems like you have a local dual stack (IPv4 / IPv6) networking issue.
The service is up and running and has 100s of people downloading from it.
failed: Network is down. is an issue with networking on your machine.
Maybe try disable IPv6 or force a IPv4 connection for the downloads.
The problem is local to you. Other people are not reporting this (server stats for me are here - the only âextended delaysâ there are me reloading the database).
I suggest that you try again, but on different network infrastructure.
The issue no longer occurs for me. A few more details can be found in the github tracker: Replication connection timeout · Issue #1097 · openstreetmap/operations · GitHub
Thanks @Firefishy !
@Firefishy or @pnorman may be able to answer.
Iâm curious about this as well.
Thank you, I have amended the question with the traceroute logs
1 Like
The issue is persisting:
--2024-02-15 21:27:56-- (try: 3) https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/day/000/004/164.osc.gz
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:8490:34db:a912::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40a0:189:34db:a80d::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:150:34db:2e49::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40a0:c9:34db:a89d::|:443... failed: Network is unreachable.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404c:289:34db:8c04::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:8480:34db:ab02::|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:8540:34db:ab7a::|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404c:450:34db:8cb9::|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.170.113|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|3.5.137.131|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.170.81|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.168.76|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.169.97|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.73|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.85|:443... failed: Network is down.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.170.57|:443... failed: Network is down.
Retrying.
I still keep getting time out problems.
Resolving planet.osm.org (planet.osm.org)... 2001:470:1:fa1::11, 184.104.179.145
Connecting to planet.osm.org (planet.osm.org)|2001:470:1:fa1::11|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/day/000/004/175.osc.gz [following]
--2024-03-05 10:16:07-- https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/day/000/004/175.osc.gz
Resolving osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)... 2a05:d050:40af:8651:34db:a9a6::, 2a05:d050:402f:8580:34db:aa4a::, 2a05:d050:40a0:3d1:34db:4bb4::, ...
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:8651:34db:a9a6::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:402f:8580:34db:aa4a::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40a0:3d1:34db:4bb4::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:402f:8549:34db:aa86::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:1d0:34db:2e10::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:1d1:34db:2e14::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404c:441:34db:8cad::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:8701:34db:a9fe::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.72.211|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 120821635 (115M) [binary/octet-stream]
Saving to: âdaily/175.osc.gzâ
daily/175.osc.gz 100%[===============================================================================================================================>] 115,22M 8,62MB/s in 16s
2024-03-05 10:26:23 (7,19 MB/s) - âdaily/175.osc.gzâ saved [120821635/120821635]
Maybe this topic is related: Minutely replication â timeout when trying to download change files from planet.openstreetmap.org
his âsolutionâ was changing the replication url to download.openstreetmap.fr
Since Saturday to Sunday night I also get a lot of timeouts when OSMOSIS and other services try to download the minutley diff files from here: Index of /replication/minute
Usually it gets better after 24 hours, but now the problem persists. Any ideas? So far I do not want to switch to another replication site.
(Cross posting: Minutely replication â timeout when trying to download change files from planet.openstreetmap.org - #4 by pitscheplatsch)
