I think this first post by me on these forums, may actually send a minor shock wave through the OSM community, that may reverberate for a while… (but I may well be grossly over-estimating this ;))
During the past half year I have been working zealously on an “OSM Renderer for ArcGIS”. I think the results are now so good, that they are finally presentable to the world.
***** Please note this is an early announcement, the actual toolbox, is not yet available online. ***** See entirely below also.
The OSM Renderer for ArcGIS depends on ESRI’s ArcGIS for Desktop, and especially ESRI’s Open Source “ArcGIS Editor for OpenStreetMap” as hosted on GitHub (https://github.com/Esri/arcgis-osm-editor). It extends ESRI’s editor with an option to create “Topographic” style maps of OpenStreetMap data, and output these using the standard options of ArcGIS for Desktop to PDF as 100% vector based output for high quality professional offset printing, or just for fun to your own desktop color inkjet printer… While the idea of a “Topographic” style map based on OpenStreetMap data may seem outrageous for those of you living in countries where hardly anything besides roads has been mapped yet, many parts of the world, and especially in Europe and in the vicinity of larger cities and towns world-wide, do provide more than enough “content” to create a convincing “Topographic” map.
Please note that I build this renderer from scratch as a personal project in my spare time, but this is not an “afternoon-tea” project - far from it! For anyone who ever attempted to do this (and I think there aren’t that many out there…), you know how hard this can be, and the plethora of issues to deal with considering the open data model of OpenStreetMap. Kudos to those of you who were, and are, involved in the development of the default OpenStreetMap CartoCSS / Mapnik renderer of OpenStreetMap, despite sometimes regretted but understandable issues once you get down to the technicalities, I still think it is a phenomenal job! It is hard to deal with the specifics of the OSM data model in rendering at times, and to “de-clutter” the map at a specific scale for rendering, or to prevent labels from appearing quadruple or more because features may appear in multiple “layers” worth labeling (historic, landuse, etc.)… To build this renderer, I have therefor read many Wikipedia pages to understand OpenStreetMap’s data model, and make conscious decisions about how to tackle rendering issues. In addition, I had many excursions to the OpenStreetMap website, Tag Info and Overpass Turbo (these are some freaky powerful tools by the way…) to compare my rendering results with those of Mapnik / CartoCSS and info about tag usage, and adjust rendering rules based on observations and mismatches.
The OSM Renderer is implemented as a combined ModelBuilder / Python toolbox. It depends on the above mentioned ESRI toolbox for preliminary data preparation and extraction of attributes from *.OSM files and creation of an ESRI Geodatabase, but enhances this with OSM rendering rules and automated data processing, to build the topographic styled map. It makes extensive use of SQL Definition Queries, Symbol Level Drawing and (Maplex) labeling options, including label expressions. The OSM Renderer for ArcGIS cartographic styling is loosely based on CartoCSS / Mapnik rendering, but there are some major adjustments. I have made a serious attempt though, to stay “true” to the default OpenStreetMap website rendering, so things should look quite familiar to all of you… There is no usage of web-components or pre-existing style sheets (I “copied” OSM standard colors by hand using RGB values), and it is therefore a pure Desktop solution not in need of any ArcGIS for Server components. Processed data is stored as Feature Classes in either an ESRI File Geodatabase, or an Enterprise Geodatabase (PostGIS, Oracle, SQL Server etc.). But I still need to test this latter option, although I don’t see a reason why it shouldn’t work, except possibly an IO performance bottle neck.
To extent on this, and very importantly, the “renderer” deals with the most difficult parts of OSM rendering, that is, for example, proper display of roads and railways in terms of what runs below or over other roads or railways (viaducts, bridges, tunnels). Another of such rendering examples, is that it will filter out any below ground buildings and stations based on the OSM “level”, “layer” or “location” key, only displaying above ground ones.
I spend countless, in fact inordinate hours, fine tuning all of this. Despite this, as with any rendering of a data source like OpenStreetMap or for that matter any geodatabase, ultimately, major comprises have to be made. It is just impossible to render everything, nor is it desirable, and rendering issues do exist, most notably the appearance of outline buildings, due to the familiar “tag-on-relationship” versus “tag-on-way” issue. Still, it goes a long way in representing a base selection of the rich data available in the OSM database…
Since my main goal was to create a renderer suitable for creating maps with a topographic style “look-and-feel” at a base scale of about 1:10,000 (extend-able to about 1:50,000), suitable for navigation purposes “in the field” or as touristic base map, I specifically chose not to render any objects to volatile or numerous to make a sensible map at these scales. E.g. while the actual retail function of a building can be safely rendered on a “static” topographic map, as buildings generally maintain such a role for extended / decades long periods, I specifically refrained from trying to depict specific shop types (e.g. supermarket, clothing, grocery) or shop names on the map. There are just to many of them, and they come and go by the dozens in an average city in an average year. Nor do you ever see them for similar reasons on official topographic maps, which aim for a readable map suitable for navigation, which is also the goal of my OSM Renderer for ArcGIS.
The renderer is not a full “multi-scale” renderer as the default rendering of the OpenStreetMap website is, I specifically developed the tools for printed output in a specific scale range (although a future web service based on all of this, isn’t a phantom). That said, the maps of Rome (1:10000), St. Petersburg (1:25000) and Interlaken (1:50000) actually show that, with some caveats, e.g. overlapping and not always so nicely rendered railway / highway lines, the rendering can serve up the most wanted and used “topographic” scales. I implemented some limited measures, e.g. setting display scales for a few specific Feature Layers or labels in ArcMap, to facilitate this.
The styling, as said before, is loosely based on CartoCSS / Mapnik, but I did take the freedom though to include some enhancements or changes compared to the basic OpenStreetMap rendering. Some features of the OSM Renderer for ArcGIS that you may note once you start exploring the maps I included in this post:
- More distinction in building types. Buildings with retail function, offices, industrial, warehouses, garages, sheds, greenhouses / glasshouses etc., now all have a specific rendering, allowing you to for example “spot” the main “Kalverstraat” shopping street in a city like Amsterdam, or the huge garages complexes in Saint Petersburg at the borders of quarters of the city.
- Specific emphasis for many historic and tourism related features. Things like attractions, museums, castles and palaces, now pop-up on the map due to the use of a distinct color (salmon-dark red for attractions, purple for historic). Especially see the “Rome” or “Paris” maps for this.
- Rendition of many of the “ski” / “hike” features in alpine regions, including different symbols for different types of aerialways, pistes for downhill and nordic etc. In addition information elements like guideposts and information boards / maps for hikers. See the “Interlaken” map for this.
- More distinction in railway types. Sidings, spurs and yards are depicted differently from main lines, while subway and light rail also feature specific symbology compared to main line.
- Support for many of the features of the approved new “Power” scheme, see the “Rotterdam Harbour Mouth” map and the E.ON power station there, that implements some of this in its OSM tagging. I am still working on some aspects.
- A possible revelation, and maybe world’s first?, is a comprehensive legend of all features rendered by an OSM renderer. It certainly reveals the complexity of the OSM data model, and why questions by users like “Why does Mapnik not render X/Y?!/ Why don’t I see this new tag X/Y that I entered in my OSM editor immediately displayed on www.openstreetmap.org?”, are not so easily answered…
This is just a glimpse of the work I have done, start exploring the maps and let me know what you think! I recommend you to zoom in at 125% in Adobe Reader. This will let you both appreciate the detail rendered, and keep a bit of an overview. The renderer literally renders hundreds of different features. One note though: although you may think that the roads are displayed “thin” if you are used to OpenStreetMap web map rendering only, they actually are close to real world widths (disregarding the “boulevard” type city roads like in Paris, or six lane highways encircling cities), as in true 1:10,000 scale topographic maps, and they will look fine and normal in a printed map. I have included both two screenshots here to “whet the appetite” and a direct link to one PDF, and the other maps in ZIP files for an easier and more reliable download. They cover a number of well-known cities and areas around the globe and are the real, direct output, of my toolbox (well, the rendering part of it, not the map layout with legend and such). The “Interlaken” map is a special case though, it has been enhanced with the addition of ASTER satellite GDEM V2 based contours, that I created myself from the raw raster data as download-able from NASA, similar to OpenCycleMap.
Some of you may regret the fact that I didn’t build this in Open Source QGIS, but although I have recently begun to explore QGIS and really liked what I saw, the fact remains I have vastly more experience with ArcGIS, than QGIS. I leave it to someone else to come up with a QGIS equivalent… In addition, my tool focuses on the actual rendering aspect of OpenStreetMap data in ArcGIS, and leaves the heavy job of processing and creating the base Polygon / Polyline / Point ArcGIS Feature Classes from OpenStreetMap data, to the already existing tools ESRI provided to the community. My renderer takes these as input.
Lastly, I think once you see the results, you will love it whatever your personal choices and uses of OpenStreetMap data and related editing and web application software are (well, at least I hope so… ;)).
Yes, you may print these maps, it is your (or should I say “our”?) data… Please do not(!) ask me to render your city, town or area. I can not possibly provide such a service on my own.
I am currently in direct contact with ESRI Redlands, to see what we can do to make this available to the entire OSM community, and possibly benefit from cooperative development to finish this personal project and optimize the needed data processing and integration of these toolboxes. These contacts are still in a pre-liminary stage though, so bear with me, it will take some time.
Whatever happens, I think however, that there is a place for both a “topographic” rendering like mine, and the rendering by CartoCSS / Mapnik. Where the default OpenStreetMap rendering excels in multi scale display, and display of objects and labels to the very finest detail, gives my map renderer a more “faithful” rendition in terms of real world size and scale of objects, as expected from topographic maps.
Lastly: have patience… Despite the current results, it will take some serious time before this can be finally “released”. There is still a significant “TO DO” list, and documentation needs to be written. These are absolutely not things to rush, I don’t want to release something half-finished or with significant “rough edges”, nor could I ever do them overnight.
Please post any comments and questions you may have here on the forums!
I hope you will enjoy exploring these maps as much as I did, it was a real pleasure and excitement to see maps of previously unexplored areas being rendered of the downloaded data, and appear on screen in full glory in the PDFs. Note that the PDFs are big, between 6-20MB, consider this before downloading if you are on a severely limited internet connection. They are full size A0(!) layouts, so extremely rich in topographic detail.
***** WARNING - Adobe Reader - WARNING *****
Adobe Reader, needed for viewing these PDFs, has an option for “enhancing” thin lines. This option causes especially railway tunnels, but also other thin line objects, in the below linked maps to show up very ugly on screen, with much over-emphasized appearance. Unfortunately, Adobe has made this bad option of “enhancing” thin lines default. I strongly recommend disabling it before reviewing these maps. The option can be found under Edit / Preferences / Page Display and than under Rendering / Enhance thin lines. Switch this option off before reviewing the maps.
I also strongly urge you to download the maps locally for the best viewing experience both for performance and quality reasons. Even though the Roma map can be viewed directly in the Adobe Reader in your browser using the link below, its display will suffer as the plugin handles things differently (and worse) as compared to the offline viewer.
United States-San Francisco
The Netherlands-Amsterdam Schiphol Airport
The Netherlands-Rotterdam Harbour Mouth
The Netherlands-Zuid Kennemerland
***** New maps (06-09-2014) *:
The Netherlands - Nieuwe Hollandse Waterlinie - Utrecht
The Netherlands - Nieuwe Hollandse Waterlinie - Gorinchem