If you are interested to discuss details with them.
See OpenStreetMap tile request 403r Access Blocked error message · Issue #40581 · dbeaver/dbeaver · GitHub
That part is known, but this does not change that enforcement of this rule is necessary.
It is sad that it is causing unwanted side-effects but steps to fix it needs to be taken by affected days consumers. Note that Tile Usage Policy has not changed and they were violating it, maybe on basis of “it is currently not enforced”.
Good faith usage is welcome but some cooperation is needed now to block abusive users.
If someone is not interested in diagnosing and reporting problems to users of tiles, and someone is not reporting problem where they follow Tile Usage Policy and get blocked anyway…
Then posting info that blocks on tiles requested without referer header are working as intended is not very useful.
And problem with opening local file in browser is known, if anyone has idea how it can be solved it would be nice.
4 Likes
I use the OpenLayers mapping library. They have a solution for people requesting maps via a web server. The challenge I face is:
- I use OpenStreetMaps via a local HTML file with the file:// protocol.
- I cannot set the User-Agent as web browsers do not permit javascript in local HTML files to override this value.
- It appears that the referrer method is only available for requests coming from a web-server. I have tried this method and I’m getting loads of Access blocked messages
- Changing the code to run within a node shell is not viable for the charity where I volunteer.
1 Like
The 403r tiles are not expected to be served to scrapers. They are being served to sites or apps that do not comply with the tile usage policy.
There are no changes to the policy. We are being more systematic about enforcing existing requirements. What DBeaver is doing has never complied with the policy, going back to at least 2011.
Is there a way to identify DBeaver traffic to give them more time to switch?
We’re aware of the difficulties in making a local file comply with the tile usage policy. We don’t yet have a solution.
2 Likes
Thanks in acknowledging the problem I have and please let me know when you have a potential solution you would like me to test - eg could we add some identification information in the HTML URL requests to the OSM servers, or pass information in the header other than using the User-Agent field?
The firefox webbrowser can be configured to send an aribtrary user-agent header. Go to about:config, paste general.useragent.override into the search field, tap the plus icon, choose string and enter some fantasy text. From then on opening the local file will not show you the 403r tiles.
Beware that this will make lots of websites look quite different from how you knew them. If you do not use firefox otherwise might be fine. Maybe chrome can do similar?
PS: Some investigation what does NOT work for file:/// URIs.
<!DOCTYPE html>
<html>
<head>
<script>
navigator.__defineGetter__('userAgent', function () {
return "LibCurl"
});
navigator.__defineGetter__('appName', function () {
return "Curl"
});
async function getData() {
const url = "https://ifconfig.me/all.json";
try {
const response = await fetch(url);
if (!response.ok) {
throw new Error(`Response status: ${response.status}`);
}
const result = await response.json();
console.log(result);
} catch (error) {
console.error(error.message);
}
}
</script>
</head>
<body>
<script>
console.log(navigator.userAgent);
getData();
window.history.replaceState(null, '', 'https://www.openstreetmap.org/');
console.log("Success");
</script>
<a href="https://ifconfig.me/">Info</a><br>
<img src="https://tile.openstreetmap.org/16/34842/22974.png">
<small>Working: Set general.useragent.override to a string in about:config</small>
</body>
</html>
I have just tested the above script and unfortunately it does not work for me using the Firefox browser. Whilst the useragent appears to be set, the object returned from ifconfig indicates the browser has overwritten the useragent field.
I’ve copied the above script into a temporary .html file on my Windows desktop and opened the file in Firefox. Here is the console log
LibCurl test.html:32:9
Uncaught DOMException: The operation is insecure.
<anonymous> file:///C:/Users/****/Desktop/test.html:34
test.html:34:16
Object { ip_addr: "*.*.*.*", user_agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/20100101 Firefox/148.0", port: "58656", method: "GET", encoding: "gzip, deflate, br, zstd", mime: "*/*", via: "1.1 google", forwarded: "*.*.*.*,*.*.*.*", language: "en-GB,en;q=0.9" }
encoding: "gzip, deflate, br, zstd"
forwarded: "*.*.*.*,*.*.*.*"
ip_addr: "*.*.*.*"
language: "en-GB,en;q=0.9"
method: "GET"
mime: "*/*"
port: "58656"
user_agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/20100101 Firefox/148.0"
via: "1.1 google"
<prototype>: Object { … }
Another couple of things that you could try at your end:
- Try a different source of tiles. Whether this will work for you will depend on where you are, where you want raster tiles of, and what detail you want in them
- Create your own tiles and point at those. Likely only an option if you either want a small geographic area or relatively little detail.
- Run a web aerver (serving only to localhost) as part of your solution - either IIS or Apache under WSL2. Likely more work than the above two options.
1 Like
That is what I wrote above, NOT in capitals. The code does not even reach the “Success” line, errors out before. I guess there is no way to programmatically make file:// documents comply with the policy.
Did you try the change in about:config?
PS: You can always undo the change by clicking the trash icon.
1 Like
As mentioned before
1 Like
Solved the problem bij using another provider. We don’t use OpenStreetMap anymore and stopped contributing. we understand the policy.
Sad news you could not fix adding headers. After I opened an issue with Dbeaver to correct headers for tile request, their last Release published today fixed the problem using a local web server. I simply had to to confirm to my firewall to accept this connection.
1 Like