Looking for an OSM building data service

General talk
1

For my project OSM Buildings http://osmbuildings.org I’m looking for a reliable and available data source.

Right now, to test the engine I’m downloading osm files from geofabrik, extract just buildings into a database (using node.js).
As the whole project grows, it needs either a REST data service providing whole world coverage or at least selected area coverage.
For scalability, creating data tiles could be an option.

For a while, I have been talking to Tordanik (Hi!). I got a bit calm probably because of the huge amount of adaptions he would need to do for the project.
Maybe we can continue.

Also since a while, I was talking to OSM-3D from Uni Heidelberg. We made small steps but since Marcus Götz left, I’m basically running in circles there.
Previously, I was promised full access to their buildings database which is perfect for the beginning.

There are also promising options with commercial services, but I’d like to try it once more asking the community.

Any suggestions are very welcome.

Regards,
Jan

2

Hello Jan,

I’m still interested in helping you by building a service based on OSM2World. As you know an early prototype already exists and I could probably finish the project according to your final requirements.

What I cannot promise yet is whether I’d manage to provide worldwide coverage in the short term, but I’d be willing to at least try whether it works out.

To clarify, are you hoping to have this service hosted for you by someone else? If so, what kind of load is to be expected? This is probably going to be the main obstacle if you choose OSM2World: I’m unlikely to have access to more powerful server resources than you have available yourself.

I do have another suggestion, though, even if this means that I’m advertising an alternative solution from what I can offer myself: Have you looked into Overpass API? It does not have any building-specific intelligence, but is instead a general-purpose API. If you are willing to do the rest on the client side, though (and I remember you were already experimenting with that for the simple shapes?), it could easily give you all building polygons in an area. Availability and stability are pretty good, too.

We could even combine the approaches: Get all buildings where you have client code for rendering from Overpass, and fall back on a backend based on OSM2World or one of the other 3D renderers or the more “interesting” buildings only. This might make it a lot easier to achieve worldwide coverage.

3

This post is quiet interesting to me, as I read osmbuildings problems at twitter.

IMHO this can be a a huge step forward and as such it’s for a long time on the general 3D at OSM roadmap. This would change that mappers aren’t no more just tagging for personal renderings, but for a public available visualisation, that they can consume via various platforms. To get slowly to a solution, I asked the KDE Marble globe, if they might be interested in adding real 3D objects…

If we talking about** a new open service**, IMHO we need to make clear, that it needs to be modular and open in every step. But (IMHO) each step would need still a lot of time:

  • infrastructure - you would need clusters and software that scales well

  • storage backend - I’m not sure if PostGIS or any other current ready to use software could deal with the amount of data,

  • transport protocol - starting from scratch? Beside opensim/Second life I don’t know any ready to use 3D streaming service. Or using enriched content on the back of WFS? Difficult to find a solution, that works for destop and embedded clients…

  • data format - 3D has IMHO a 3D format hell all with pros and cons. Or developing something new?

I fear, that if you see this huge amount of development work that needs to be done, this approach wouldn't work in a measurable amount of time without a sponsoring for 5...10 developers :( Even if I really like OSM2World and agree that currently it provides a much more variety of 3D visualisations and is more open, currently this seem to be a big rock to climp on :/ (but plz call me wrong!) So if we want to go that way, we should talk about the money/ ressources point. But there might be another approach:

opening OSM-3D services:
I don’t know anything about the concerns of the Heidelberg staff. Beside the social aspect, that I don’t want to blame anybody by reinventing a very similar solution, I think they already got a solution that works:

  • infrastructure - is already there, even if centralized, they might get deployed at a donated cluster (user:vvvooovvv told something at the 2. 3D workshop)

  • storage backend - seems to work at least ok

  • transport protocol - Was used to demo/promote the W3DS which they publish as draft to OGC. (I can’t tell anything about it, as I didn’t read the specs)

  • data format - They support a variety of open formats as VRML and Collada

  • features - they already process SRTM terrain data and embed detailed buildings of OpenBuildingsModels (IMHO very essential aspects, to have a good user experience and to attract artists to contribute using their traditional 3D tools)

Till now, nobody seems to have detailed information about why they should block the process of opening the infrastructure for third party projects. In my personal mind, they finished the "scientific utilisation" and now this might become the next step, towards opening the development for the OSM community again. IMHO this would be logical, as they already released their Xnavigator client as open source, they might do it for the server components as well, to make sure that the evolution of the project persists and that their work results in a maximum use of data and technologies? I will ask them to the roadmap for OSM-3D and if they are willing to contribute the code :)
4

I think that the challenge to provide content for osmbuildings is not as hard as you suggest. Due to the simple rendering style and the slippymap user interface, this is a lot easier to build than an hypothetical feature-complete 3D streaming service. It doesn’t need terrain meshes, textures, shaders, arbitrary camera viewpoints, and it can completely ignore any object which is not a building. Even the number of buildings potentially visible at the same time is limited by the top-down perspective. There isn’t any true “streaming” required, nor does the problem call for a sophisticated 3D format or novel storage backends. (Jan, please correct me if I’m wrong you would like some of these features.)

A 3D streaming service for photo-realistic virtual realities along the lines of Second Life would indeed require a development team with considerable manpower - and I’d be happy to be part of that once we find someone willing to pay for it. :wink: But aiming for that level of service and features would be overkill for the task at hand, i.e. showing building wireframes on a slippymap.

The only real challenge I indeed see with that is scaling, both geographically and for server load. That’s why I’m particularly curious about the expected load.

5

Hi Tobias & Matthias,

Some info, what OSM Buildings is capable of right now and in short term.

backend now

  • data is very custom geo to pixel, pre-rendered @ some zoom, in JSON, just footprint polygons + height
  • data is served by PHP out of a MySQL (or PostGIS) database containing just geo polygons + height
  • served area is fully dynamic - thus heavier load
  • just some data is stored, updated manually, very complicated upload

backend asap

  • move pre-rendering geo to pixel onto the client, just serve geo polygons + height
  • add a special db index like ‘map tiles @ zoom 14’ in order to have quick lookups → tile-like referencing
  • enhance JSON format for next client’s requiremens (see below)
  • if it makes any sense, create JSON data tiles (files)
  • ideally: allow access to everyone who wants to embed OSM Buildings
    => as this is the key point for costs and scalability, I could imagine either to charge users
    OR to let everybody cerate his own tile set for his web server
    OR allow people just to clone the server
    Point is: as less obstacles as possible!
  • updates should happen very often, coverage global (maybe Overpass fits here)

frontend now

  • like it is now… doesn’t create much traffic on the data side. map tile usage is much heavier

frontend asap

  • support for individual building and roof colors
  • some color determination by building type (living, religious, other…)
  • support for cylinders, domes, spheres, still in cell shading style
  • handling buildung parts as one object (for selections)
  • handling multipolygons
  • maybe support for one very simple roof shape (Tobias, this is something I would need your help)
    => all these will cause more data and will make it more attractive, I hope

not required

  • Cool details like terrain, trees, streets etc. are not planned to be part of the engine.
  • No heavy infrastructure, unless demand exists and its paid.
  • No 3d streaming needed yet, although it’s cool and maybe worth it.
  • No new standard needed. It’s basically creating it’s own. Full process of estaiblishing through the community takes too long.

Polygonal building models are a bit aside. I’d like to have a simple version, but it needs lots of performance tuning.
Therefore I’d like to postpone this a bit. And Tobias, thats also a topic to discuss in detail.

State from OSM-3D is, that Dr.Zipf is too busy to handle it and Marcus Götz, who helped me a lot, left the University.
I’m now talking to an assistand who is raising security concerns and currently not responding anymore.
As Heidelberg has every infrastructure sufficient for the next steps, would be a natural fit.

6

As I understand at this step you need only polygons and height at client site, so overpass sound ideal for this. For big zooms you can even resign from you server site and only parse data directly from overpass in web-browser JavaScript client. This can made your application independent from database.

7

Hi kendzi,
I’m going to give the API a try! If it works as expected, then it’s the ideal solution for now.
Being independent and up to date… I just hope, the service is at comparable speed delivering data.
But in worst case there will be a small caching instance.
Good news, thanks!

And one more thing which I blindly ignored: I was always expecting to have data quality similar to Germany. Testing New York City yesterday, I just woke up.
Doesn’t make sense to have just ine data source and OSM is not an ideal choice everywhere.
But no worries, the library might be open to different sources. I just shouldnt try to server the whole world from one service :wink:

8

Hi all,

Just a few informations on what we are currently working on at Oslandia, which could be of interest for the discussion.
We are busy with PostGIS 3D implementation, and we have some progress done.

You can read more about it here :
http://www.oslandia.com/tech/?p=1235

Some more insights into PostGIS 2.x and 3D are in those presentations :
http://oslandia.github.com/presentations/

And find some code here :
https://github.com/Oslandia/postgis-sfcgal

Our aim is :

  • to build a full 3D support for PostGIS, with 3D object, and textures as a second step. It will allow for storage (already in) and processing of 3D objects, just as you can do it with 2D data.
  • to build on top of that a «streaming» server for 3D objects. It is not streaming per se, but it’s a 3D vectorial data access webservice. It will be implemented using PostGIS 3D features and TinyOWS (part of Mapserver) WFS-T fast server. It will probably output CityGML first, but collada (KMZ) and X3D are candidates we will for sure consider. REST webservices would be great too, even if our first target is OGC webservices compliancy (yes, I know…).
  • Have tools to import / export data to and from postgis 3D (from collada/KMZ, CityGML)

As for now, PostGIS 3D foundations seem good. The next steps are :

  • discuss integration of the 3D codebase into postgis core (at OSGeo Boston code sprint)
  • improve available 3D operations in the database
  • start implementing the webservice part

Among the operations to be found, we have extrusion, straight skeleton (for roof generation) and much more (see current CGAL features).

The following steps will be :

  • integration of 3D objects from webservices into 3D web clients (cesium, opengeoglobe, openlayers 3)
  • data edition

Note that 3D objects are in the Cesium Roadmap, and Openlayers 3 will probably use Cesium for 3D. As for data edition, it needs to be thought about…

We will work hard this year to have as much of this stack as possible implemented. We are open to any contribution and help, just follow the projects on their respective source code repository and send pull requests, or write to respective mailing lists.
If you are interested in collaborating, do not hesitate to send a direct email too, as we do not follow this forum closely.

We will have to think about scalability as well, but PostgreSQL/PostGIS has made great progress lately, and probably with decent hardware, sharding and replication we can handle quite large load. If it’s not enough, caching 3D objects in a NoSQL database (memcache, redis…) could be an option. This would depend on the exact use case and will probably need prototyping and testing.

All that do not answer right now to the OP needs, which are different, but I hope it gives you some matter to think about for the future, and answer !i! comments on a full-featured 3D object streaming stack.

9

This could be of interest to use as a 3d database: http://www.3dcitydb.net/index.php?id=1880

Interesting paper on preparing data to CityGML standard: http://3u3d.edpsciences.org/index.php?option=com_article&access=standard&Itemid=129&url=/articles/3u3d/pdf/2012/01/3u3d_03004.pdf

10

Hi guys. Tordanik alerted me of this thread. I’m the developer of WikiMiniAtlas (http://wiki.openstreetmap.org/wiki/WikiMiniAtlas) the slippymap gadget on Wikipedia. WikiMiniAtlas has displayed 3D buildings for a few months now, but until last week only as wireframes rendered on a 2D canvas overlay. I’ve just implemented nice 3D webgl-based buildings. It is live on Wikipedia. My tool needs almost no preprocessing of the OSM data. It accesses a copy of the mapnik rendering DB that we maintain on the wikimedia toolserver. I have a small PHP script that generates JSON tiles from the db (on demand, but cached). At high zoomlevels those are are rendered in the browser onto canvas elements. The JSON data contains a large selection of OSM tag data, among it everything that is necessary to render 3D buildings. The code is basically using raw WebGL to keep everything small and fast. I use poly2tri to triangulate the polygons, this way I can easily handle Buildings with courtyards etc (holes).

The whole 3d stuff was basically written in a week and builds on my client-side rendering infrastructure (which was written in a week last summer). WikiMiniAtlas gives priority to displaying Wikipedia articles on the map, OSM data gives context, but no OSM text labels are shown on the map (at high zoom levels you can click on roads, buildings etc. and the OSM name pops up).

The biggest problem I currently foresee with 3D building data is the unfortunate policy of tagging the main building=yes element with the maximum height of the building, while building:part=yes elements are tagged with actual (smaller) heights. If this is followed then every building will just render as a box, and the detail given by building:parts is covered up. Detecting if a building element contains building parts is not trivial (actually almost impossible if you only have the mapnik rendering database to work with). This is a bit frustrating.

11

The relation type=building is to make easy to detect buildings parts. All building with parts should have this relation. So if in this relation there is at least one element with building:part you should render only building parts and skip building outlines (building=yes).

12

the

is necessarily for the level of detail concept:
When You wish generalize easily the 3D objects to the cuboids without roofs, You need only one parameter, which is height. Every other techniquel like analyse of building geometry and calculation of total height is more complicated.

13

Level of detail is fine, but the current implementation is not. The height key is used inconsistently for both the maximum height and the minimum/true height. It would be simpler to introduce a key such as max_height. Also I wonder if this level of detail concept is used at all in any application that processes or visualizes OSM data. The current concept places undue burden on applications that want to extract building detail. The relation concept is flawed for two reasons: 1) relations are not consistently used in building tagging and the complicate the tagging process, 2) visualization applications may not have access to the full relation data, which is lost when using a mapnik rendering database. Accessing the original OSM data requires costly preprocessing to get something useful on the screen.

Executive summary: if you care about application developers and getting 3D buildings to the end-user then change the inconsistent usage policy of the height key!

14

Where is height being used for something else than the maximum height? On building parts, it refers to the distance from the lowest point of the building above the ground up to the highest point of the roof. Clearly this is the “maximum height”? I fail to see any inconsistency here.

15

Ok, Tordanik, I guess you can make a case for height being max height in every case. But my other point still stands:

  • it is unreasonably hard to detect if building element has parts associated

Is the level of detail concept leveraged at all in any application? If so let’s look at the application scenario. Is it performance critical, like displaying tons of 3D buildings on a map?

This thread started because a dev was looking for a data source for 3D buildings. Well, we have a simple one the osm2pgsql rendering database. It can be optimized for fast access with cheap queries, that is unless this level of detail policy is enforced. Then at best we get expensive queries involving relations, and in the worst case have to check for geometric containment which is even more expensive.

The 3D buildings standard is still new. It can still be fixed and made more application friendly.

16

Moin

Hmm, the building outline with building=yes is available, if you got no free ressources, use that. IF you got ressources, search for tags and apply them. Each building:part has a building=yes associated and has a closed polygon line. See each building:part as a single building.
And here the realation comes into play: it group these parts together into one building.
If you got the OSM data, you got the relation data, to. I do not know any service which has the OSM data but not the relations.
You wrote about map services - they do not have the tags with height, building parts,… also. The map services usual do only have building=yes outline and nothing more.

So for performance: use building=yes. For good results, use the relation.

But what is your idea about tagging buildings as they are (with all different parts) which will be application friendlier?

Amiga4000

17

I have access to a osm2pgsql mapnik rendering database. It does contain building and building:part data. I have a working 3D building display. But it currently only works because building elements are not tagged with max height. The rendering database does contain some relation data, but it is hardly usable (multipolygons are collapsed into new geometry objects and I’m not sure all the way number data is preserved) and would increase query complexity tremendously if it would work at all.

My idea is not using the height tag on the building elements as a low level of detail max height but use a separate key for that. This way it would be a straight forward process to paint all building and building:part elements in 3D and get the desired result. Buildings without parts would display fine, buildings with parts would not need the complicate suppression of the low level of detail volume, which would otherwise cover up the high detail from building:part elements.

If you want to get beautiful 3D buidings visualized for the users you’ll have to make it easy for application developers. And my tagging scheme would not complicate matters much for the taggers either.

18

Moin

So only the height on building=yes is a problem if building:part are available?
Currently I do not map any height on building=yes if building:part are available, as the building:part should cover all the building:yes area and should have all the tags, also.
In this case: drop all 3D information from building=yes and use only the building:part information, job done.
But others do tag some information on building=yes, as they keep the same for the whole building. One simple pre-process would be to copy over information from building=yes to the building:part in the same relation and not copy over the information present in building:part, afterward ignore building=yes. Looks quite simple and fast to me. Just instead one building, you result in 2,3,4 or even far more (a church could have 20 building:part).

About way to get a result: 2 ways, one is keep it simple for the DB, keep it simple for the user, keep it in the way of OSM. The other is: make it easy for the apps.
Currently the OSM way is the first one, as a lot of information could be gathered automatic by the apps, with a bit more work.
At least thats what I do see.

Amiga

19

That is perfectly fine and will automatically render fine.

Like roof:shape? This didn’t even occur to me yet! I was mainly concerned with the height tag. I’ve only just started implementing visualizing the roof shapes.

Now it starts to get more complicated. In the rendering database I use I cannot do that. We synchronize to a master copy via replication. I could only add a new table for augmented buildings, which i would somehow have to synchronize to the rest of the db. And I don;t think I have all the necessary data (relations).

The frustrating thing is that everything could work so easily and right now, without any postprocessing and qomplex db acrobatics, if only one tiny change would be made

*not tagging the main building element with height if parts are present
*or treating building just like building:part and code the actual height rather than the min height (unless parts are guaranteed to cover the entire area of the building! Are they?)

20

Moin

Ok, so you do not only limit it to the height, but to all tags which are set to building=yes AND to building:part=yes ?
Usual it is defined: if set on building=yes, it is for the whole building and should not be changed in building:part again. Which is the treat building like building:part (for that specific feature).
Why only limit it to the height?
Btw, if height is set in building:yes, it is the max height of the whole building (and all building:part) of the building, just some parts could have a min_height, aka free space under them.

Amiga4000