I am keeping a copy of the planet file which I update every few weeks with the daily diff files. This used to work quite well until some time ago, when I guess some sort of rate limiting was implemented. Now it takes more time to download the diffs (without applying them) than it would take to download the whole planet dump again. In order to keep transfer sizes low for the servers, I still continue to apply daily diffs, but I wonder if this slow downloading behaviour is maybe a hint to operate in a different way?
I am seeing lots of timeouts, like this:
--2024-02-03 12:28:36-- (try: 3) https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/day/000/004/139.osc.gz
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:8651:34db:a9a6::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:85c9:34db:ab6e::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:441:34db:2fe5::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:181:34db:2e24::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:180:34db:2e20::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404c:450:34db:8cb9::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:8541:34db:ab7e::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:84c9:34db:a926::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.169.21|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.133|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.105|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.72.139|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.46.35|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.140.183|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.170.177|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.25|:443... failed: Operation timed out.
Retrying.
here is the result of a traceroute to amazonaws.com
$ traceroute osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com
traceroute: Warning: osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com has multiple addresses; using 52.219.170.169
traceroute to s3.dualstack.eu-central-1.amazonaws.com (52.219.170.169), 64 hops max, 52 byte packets
1 myfastgate.lan (192.168.1.254) 0.969 ms 0.689 ms 0.672 ms
2 10.103.123.27 (10.103.123.27) 6.204 ms 1.637 ms 4.147 ms
3 10.103.72.2 (10.103.72.2) 1.765 ms
10.1.209.250 (10.1.209.250) 2.076 ms 2.041 ms
4 10.250.132.213 (10.250.132.213) 5.362 ms 5.792 ms 4.308 ms
5 10.254.20.5 (10.254.20.5) 6.099 ms 4.732 ms
10.254.20.9 (10.254.20.9) 5.431 ms
6 93-57-68-98.ip163.fastwebnet.it (93.57.68.98) 6.431 ms 4.623 ms 7.809 ms
7 93-57-68-137.ip163.fastwebnet.it (93.57.68.137) 4.172 ms 8.787 ms
93-57-68-141.ip163.fastwebnet.it (93.57.68.141) 5.929 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 s3.eu-central-1.amazonaws.com (52.219.170.169) 24.379 ms 22.298 ms 23.309 ms
here another one to planet.osm.org:
$ traceroute planet.osm.org
traceroute to planet.osm.org (184.104.179.145), 64 hops max, 52 byte packets
1 myfastgate.lan (192.168.1.254) 1.130 ms 0.731 ms 0.675 ms
2 10.103.123.27 (10.103.123.27) 1.580 ms 1.367 ms 2.501 ms
3 10.1.209.250 (10.1.209.250) 3.329 ms
10.103.72.2 (10.103.72.2) 2.221 ms
10.1.209.250 (10.1.209.250) 2.079 ms
4 10.250.132.209 (10.250.132.209) 6.542 ms
10.250.132.213 (10.250.132.213) 5.759 ms 3.952 ms
5 10.254.20.9 (10.254.20.9) 7.082 ms 7.101 ms
10.254.20.5 (10.254.20.5) 24.009 ms
6 93-57-68-98.ip163.fastwebnet.it (93.57.68.98) 3.272 ms
93-57-68-102.ip163.fastwebnet.it (93.57.68.102) 2.867 ms
93-57-68-133.ip163.fastwebnet.it (93.57.68.133) 3.017 ms
7 93-57-68-141.ip163.fastwebnet.it (93.57.68.141) 9.099 ms 3.978 ms
93-57-68-137.ip163.fastwebnet.it (93.57.68.137) 5.349 ms
8 100ge0-36.core2.rom1.he.net (184.104.194.10) 12.309 ms 14.115 ms 15.650 ms
9 * * he.minap.it (185.1.114.20) 15.138 ms
10 100ge0-76.core2.zrh2.he.net (184.105.65.33) 19.968 ms 17.258 ms 18.535 ms
11 100ge0-49.core3.fra1.he.net (184.105.65.30) 35.361 ms 26.365 ms 25.165 ms
12 port-channel2.core3.fra2.he.net (72.52.92.70) 27.410 ms * 23.985 ms
13 100ge0-76.core1.ams2.he.net (184.104.193.194) 37.504 ms * *
14 openstreetmap-foundation.port-channel7.switch2.ams2.he.net (184.104.202.70) 35.134 ms 35.818 ms 37.621 ms
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
31 * * *
32 * * *
33 * * *
34 * * *
35 * * *
36 * * *
37 * * *
38 * * *
39 * * *
40 * * *
41 * * *
42 * * *
43 * * *
44 * * *
45 * * *
46 * * *
47 * * *
48 * * *
49 * * *
50 * * *
51 * * *
52 * * *
53 * * *
54 * * *
55 * * *
56 * * *
57 * * *
58 * * *
59 * * *
60 * * *
61 * * *
62 * * *
63 * * *
64 * * *