The Next Generation of OpenStreetMap — in Python!

General talk
, , ,
21

cgimap was deployed for calls other than map as part of my GSoC a number of years ago. It makes a huge difference.

My understanding of the issues with the rails map and other API calls is that their slowness is a combination of poor queries from the ORM and how objects are created/serialized in the code. Keep in mind the rails map call was written 16-17 years ago, and no one has bothered to optimize it because we have cgimap, and rails has also changed as a language since then.

Code that no one uses doesn’t get worked on.

My understanding is that most of the time was spent creating and serializing objects, but I don’t have details for the same reason - we use cgimap.

I don’t know how the performance of an optimized rails implementation would compare to an optimized python implementation or the current optimized C++ implementation. I expect all would be faster than the unoptimized rails implementation by as much as an order of magnitude.

9 Likes
22

I can understand that you wanted to have something to show since without anything to show it might come over like many posts that say something like “we need to do x” but without intending to any of the work. But by making something you show that you are serious about this.

But I do still think that your now you have made decisions that are weakly founded, and that may or may not have any consensus, for example the reasons I saw of why you picked python where:

  • It is easy.
  • I like it.

Not that they are bad reasons, but I should start with technical requirements and then look at non-technical and subjective stuff. I still think that a lot of other languages like go, Kotin, C# or even typescript would be better suited to a project like this.

23

Indeed, though I think regarding specifically the language choice it’s actually fine: I think looking at it from a purely technical perspective it would make the most sense to use a language such as Go or Rust, as they tend to work well for APIs such as OSM and when well written produce very efficient and lean executables, which could reduce the hardware requirements/cost which likely would be very interesting for a project such as OSM. With languages such as Kotlin/Scala or C# coming in second, mostly because their runtime requirements are a bit heavier.

However, when it comes to open source there is actually one requirement that weighs a lot heavier, even heavier than it does in a commercial setting (where one can always just throw money at the issue): The availability of (willing) developers.

We have here a developer willing to produce results in Python, so unless someone else comes along and credibly offers to do the same in another language that might be technically better, or to put in the work required in Ruby to improve the current code, then a discussion about using another language is pretty much a non-starter.

10 Likes
24

You do know that that is constantly happening and claiming the opposite is just hyperbole?

3 Likes
25

Sorry if it sounded like this is not the case, I guess a better expression might have been “…improve the current code at the same rate”. I’ve been careful not to downplay the importance of the current codebase, including repeatedly pointing out the possibility of improving it instead of replacing it (as in that quote).

That said, I think replacing it might be the best way forward (with a huge asterisk as I’m not that knowledgeable about it) as it gives the opportunity to clean up technical debt (which is natural in a project of this age, and in no way should be taken as criticism!) and improve the architecture.

This is also why I’ve tried to convince @NorthCrab to work with the existing database schema and to break the project into smaller components; then it would be possible to run the current code and the new code in parallell and let time tell which project has the most potential.

I’d be very interested to hear from the thoughts from the current maintainer(s); do they think that the current codebase has the potential for improvements, or is this mostly a case of keeping it on life support? Would they be interested to work with NorthCrab on this new project, bringing their expertise to it? Does anyone know if they are active on this forum?

10 Likes
26

This technique may not work effectively, as OSM often discusses matters without much progress. The approach that makes sense to me is to define a clear path forward and then make adjustments. While what you say is correct, the time has shown that it doesn’t lead us anywhere.

My goal is to bootstrap the project, so we have concrete elements to discuss, publish it, and then make adjustments based on community feedback. I believe the first step is a crucial factor in achieving something meaningful.

Nobody’s forced to participate in the discussion. Feel free to do whatever you want :slightly_smiling_face:.

Yes, when writing that, I had primarily considered experiences on off-forum platforms and personal discussions. The forum talk has been excellent, and I will prioritize it.

I fully understand that, in theory, discussing first may seem more optimal. However, in practice, the change we’re discussing will take no more than 2 days, and with concrete code to discuss, there’s a greater chance of success as something is actually being done.

First and foremost, I aim for the project to be community-maintained, making it easy for everyone involved to contribute. I intend to break away from the monolithic nature of the OSM project.

3 Likes
27

I’m going to take the contrarian view here and say that I absolutely welcome @NorthCrab’s project here. There is no downside to exploring the art of the possible of what a future OSM tech stack could look like. Even if @NorthCrab utterly fails, it costs us nothing and future attempts will be able to look back at what went wrong. And, if he makes any breakthroughs, even partial ones, they may turn out to be helpful. Open source is full of false starts and failures along the path towards successful projects and we shouldn’t poo-poo technology explorations just because they’re audacious.

I must confess that I didn’t make it through the full one-hour announcement video. However, I take it that you’ve thought a lot about what needs to be done and have programming skills. That will get you a long way. I launched OSM Americana with barely any relevant coding skills (just ask any of the other maintainers), so you’re already way ahead of where I was on day zero.

In my experience, writing code is actually the easy part in these community projects. The hard part, and also the part that makes you the most successful is the people. Nobody knows everything (though @pnorman and @jleedev and a few others come damn close). If you’re going to be successful you need the humility to network with folks and learn what they know and build trust. And if you’re going to be a community project it also means learning where and when to cede control to others. I talk about some of these topics in an OSM Mappy Hour talk I gave earlier in the year.

Let’s talk marketing. You’ve certainly figured out how to write an upbeat launch announcement in a cheery style that would make ChatGPT blush. However, it’s just the beginning. As you make technical progress, consider those to be additional opportunities to promote the project. Give talks at conferences, in person and virtual; write diary entries and blog posts; and create live, working demonstrations that people can interact with. Show how your building blocks are pushing forward the technology one advance at a time. This will build excitement, get people talking about it, and hopefully attract contributors. It also creates linkable content that you can use when discussing various aspects of the OSM tech stack in various community forums.

Good luck!

28 Likes
28

I’m reminded of how some of my former colleagues at Mapbox once developed a skunkworks project called Hecate that reimagined the Rails port as a “geospatial feature storage API” written in Rust: more modern and queryable than the Rails port, but lighterweight than an ArcGIS server. This was back in the days when Mapbox engineers had a bit of leeway to explore innovative ideas and present them to local meetups. They envisioned it as a competitor to the Rails port, but I don’t know how serious they were about that. As far as I know, there wasn’t the sort of engagement and dialogue with the OSM community and core developers that @NorthCrab is now seeking. Oh what could’ve been!

I’ve lost track of how many times I’ve heard from people who have some need for collaborating on geodata but feel intimidated by the Rails port. Not everyone is willing to spend the effort of an OpenGeofiction or an OpenHistoricalMap in configuring and adapting the Rails port to their needs. Those niche use cases might be an opportunity to prove out the new codebase and replace the faith that @NorthCrab has asked for with confidence.

6 Likes
29

Exactly my thoughts, there is always an upside to anything you do!

3 Likes
30

Sure, and that seems very understandable. And if your end goal is a demonstration project, I think that’s awesome and look forward to seeing what you build - I agree with all the points that @ZeLonewolf made. The question in my last post was genuine though - is this a demonstration project, an alternative implementation for additional projects, or intended to be deployed live on openstreetmap.org? if it’s one of the first two, then it’s your project, and I recognize you’re opening it up for discussion for us on that front, but you don’t need community consensus there. But if it’s the last option, I think you’ll need a more robust community feedback cycle than you mention in the quoted text.

Sorry, my comment you quoted there wasn’t well fleshed out. I was just trying to explain the types of responses you’re getting. I was guessing that many people are confused, as I am, about what your end goal is and it would change how we interact.

And I genuinely don’t want to be or appear negative about your project - I just wanted to share some cautions and recommendations if your end goal is a replacement of existing code rather than a separate implementation.

3 Likes
31

Hi - I’m one of the maintainers (and I only speak for myself here, of course). I’m not normally active on this forum but it might be helpful for me to join in on this topic.

I’m very happy to see more interest in this part of the OSM dev community! I think there’s a ton of improvements that we can make for the website and the API, and I’ve been working for years on some of these and making some of the other things possible. A lot of that work has been refactoring and simplifying the existing codebase, so that we can more easily work on new features and new developers can get started more easily, and we’re really starting to see that pay off now. It’s definitely not a case of being on life-support or anything like that, it’s under sustained active development and I encourage everyone who is interested to join in. I’m also giving a talk at SOTM-EU on Sunday if anyone wants to know more.

I have to applaud @NorthCrab for the JFDI approach, I’ve done that before on a different OSM project, but a key difference was that the original was completely abandoned, and the alternative tech stack had been widely discussed and agreed apon. So there’s definitely risks with the current approach, as discussed by other people above, that the effort might just be duplicative work and the new code might not get deployed anywhere.

I want to be very clear - everyone is welcome to work on whatever they want to! Of course!

However I think there actually is a cost here, specifically an opportunity cost. I think we need lots more people working together on this topic, as one big team. We have very few developers who are willing and able to work on these things, and so if it’s at all possible, it would be best for us to work together, and make sure none of our efforts are wasted. We don’t have the spare developer capacity to make it risk-free.

So while @NorthCrab can work on whatever he wants, I would encourage him to first get more familiar with the existing openstreetmap-website project, and see what can be done to help there. At the very least, a better understanding of the existing project, along with experience gained from contributing to it for a few months, would be useful experience and informative for any future projects.

This is the bit that surprises me most about this project! The openstreetmap-website project is moderately complex (nowhere near as complex as any editor, IMHO, and I say that having been a Potlatch2 developer back in the day) but the complexity is nothing to do with the language. All the complexity comes from the environment around the project - all the users, the browsers, the API clients, the security, the changing HTML standards, the frameworks, the list of community expectations and implicit requirements etc etc etc. The ruby code is very straightforward, and I can’t see how it would be substantially less complex if the same features were re-written in other language, particularly such a similar language like python.

All the other topics mentioned, like API 0.7, or anti-vandalism, or optimized performance, could just as easily be implemented in the existing codebase, as incremental updates. And I would encourage @NorthCrab to do that - any incremental improvements will get merged, and definitely benefit the OSM mappers immediately, rather than potentially benefit them maybe at some unknown point in the future.

I’m happy to discuss any part of the openstreetmap-website project (others can vouch that I’m happy to go on about it at length whenever I’m given the chance :slight_smile: ) and I can do that either here, or one-on-one with @NorthCrab if you want to deep-dive on any aspects. Or if anyone else has any questions then I’ll do my best to help.

24 Likes
32

For start I want to really thank you for all your work on osm website (and elsewhere) and also thank others working on it.

I am more than aware that few other things are stuck and waiting for me. I do not want to frame as criticism of people active in reviewing PRs. If something is bottlenecked then blaming people working on it for not working more is highly silly, unless they are somehow obligated to work and they do less than mandatory.

And it is not like I spend noticeably productive effort on reviewing PRs there.

I guess that main thing that discourages people is how long you need to wait for review and rejection/merger. See for example

that seem stuck.

And there is add icon for OSMF membership to user page by datendelphin · Pull Request #1914 · openstreetmap/openstreetmap-website · GitHub from 2018 and Add "BBBike Extracts" to the list of export services in the left sidebar by wosch · Pull Request #1383 · openstreetmap/openstreetmap-website · GitHub from 2016 (though I would rather reject this features if I would be sole decision maker, second probably would be replaceable with link to wiki page about downloading data).

Not sure would it be useful/welcome to make comment on that in PR.

(I would love to somehow solve that problem but sadly it is not so easy to solve, there are some ongoing attempts, I also tried to help personally but not sure what was net benefit of that…)

At least for me Ruby-on-Rails is bigger part of the problem, though I have no idea at all whether alternative used here would be more readable and understandable for me or others.

Ruby in general has more magic and weird behaviors and way too many ways to achieve the same, compared to Python. (that is why I moved to Python for my scripting, though not convinced that difference is big enough to reimplement existing stuff).

6 Likes
33

I can’t +1 this enough, even when I actively used Ruby on Rails I eventually got tired of there being to much magic. I guess this comes from the “convention over configuration” approach of Ruby, which results in many cases of “this happens by convention, but unless you know about it it’s hard to find out which convention”. Python tends to be much more explicit, usually making it possible to follow the code all the way through even without documentation. And though not the biggest reason (that being time), it being Ruby has also been a non-negligible reason for why I have yet to contribute.

But since we’re not talking about completely greenfield development here, it makes a lot of sense to consider what already is, i.e. the Ruby code. From what Andy has said I’m even less sure that it makes sense to replace it entirely, though the Python port can be a good way for NorthCrab to test his ideas before porting them to the current code base.

And I mean, there are projects such as PostgREST that have almost a hundred contributors, yet are written in Haskell, arguably the least contributor friendly language out there (please don’t make this a discussion about languages though, just making a point that the language isn’t the only thing that determines if there’ll be new contributors).

6 Likes
34

Isn’t that exactly what you have been doing in every single posting of yours in this thread?

It is IMHO the aspect (classical junior dev language fanboiing) that completely disqualified the whole project from the beginning, and I am -really- not known for my love of Rails.

1 Like
35

Sorry wait what?

That specific quote was about Haskell being an unfriendly language, which other than my anecdote is completely irrelevant for this topic which is why I didn’t want anyone starting to debate that statement.

I’ve mentioned languages a few times yes, since that was one of the primary features of this project according to OP. But in all cases that I can find it has been about adding nuance to the discussion (like pointing out that Python isn’t the only valid choice, and does indeed have its drawbacks for a project such as this). Also weird to call it “language fanboiing” when most of my mentions of languages have been about how little the actual choice of language matters.

4 Likes
36

This hits close to home – you’ve just about described OSM tagging. :sweat_smile:

6 Likes
37

this looks super cool! thank you so much for the work you’re putting in :smile:

38

I’m confident this is 100% true and that any reasonably motivated developer should be able to learn Ruby on Rails in order to contribute to the project (not that I have, but maybe one day). I’m never in favor of rewriting a working system in a different language unless there is some massive problem that will be solved by doing so. Unfortunately, I do believe there is a perception among many (especially younger) developers that Ruby and Rails are outdated technologies that are not worth their time investment to learn. If the project were written in Python, more developers might be interested in contributing, but in five or ten years would that still be the case? Python could easily be unpopular with the next generation of developers. Impossible to know.

10 Likes
39

The fun (or sad?) part of this is that it’s not actually that long ago that Rails was “the hot stuff”…

8 Likes
40

If @Minh_Nguyen is right and people do feel intimidated by the Rails port, then having @NorthCrab and others work on next generation software is not an opportunity cost, but a bridge that allows other devs to work on OSM infrastructure that they feel more comfortable with.

To turn this around, maybe the continued use of the Rails port is itself an opportunity cost and a sunken cost, because by using it we are missing out on the contributions from Python devs.

We don’t want to end up with 15 competing standards, but maybe with NorthCrab’s work Python could actually be the next generation of OSM.

10 Likes